Help with high CPU usage

Hi all

I wrote a program that continuously extracting specific format files to proper records for post processing, but the usage of CPU is keeping high, usually 99%.
So the extracting process was slowly and slowly, when I use sleep inside a loop, the CPU is down, but the extract performance wasn't improved because of sleeping.

I wonder what caused CPU keeping so high? Does my extract algorithm bad?
I use gprof to profile the program, nothing really helps. I was struggled what could I do, Please someone help me out.


Thanks

Black Thought

Any program that has no blocking operations or no sleep() command will probably use 100% cpu (or 50%). Also why would you think the performance of the extract algorithm would IMPROVE when you add a sleep command? It would definately slow down.

Hi 39ster.

In your words, no matter how good the algorithm is, a non-blocking program will exhaust the entire CPU, right?

Try this program:

Code:

---

int main(int argc, char* argv[])
{
    while(1){}
    return 0;
}

---

So for a GUI program it would use "GetMessage" in the main loop, which pauses until it receives a windows message. A server program might use "select" which pauses until at least one socket has data to receive. A game might limit the FPS using Sleep(). If you don't want it to use 99% cpu than you need to use a blocking operation in the main loop.

Quote:

---

Originally Posted by 51dunk

Hi 39ster.

In your words, no matter how good the algorithm is, a non-blocking program will exhaust the entire CPU, right?

---

Yes, basically. Exhausting the CPU is not by itself a bad thing. Exhausting the CPU and accomplishing nothing is.

A better algorithm will finish its work quicker. I'd rather have 100% cpu for 2 seconds and the program be complete than 50% cpu for 2 minutes for the same result.

Now, if you need to wait on, say, more data to become available, don't have the program enter a while(1) loop - waiting for some condition to occur is when a program should sleep. (And then, wake when the condition is met.)

Your code does what is called Busy waiting.

Thanks everybody for replying.

My program is an extracting program, it will extract one or more 100MB+ files to 1,000,000+ records .
So it isn't a one-two seconds operation, and I've read "Busy Waiting", yes, I have a lot of "while" loops, but I think everything is necessary.

I can't find anything helpful except sleep() which will slow down my program the other way.

You may want to use some profiling on your application and see where it spends time. It sounds like you do a bit of file-IO and some string handing [I'm just guessing]. It is quite often possible to find more efficient methods. However, that's only of some importance if 2 seconds of CPU usage is considered too much. Otherwise, just leave it as it is.

--
Mats

Hello mats

Yes, I've use GProf to profile my program, there are some functions that was called million times, so I said before I wonder if it might be problem of bad algorithm. I believe that 2 seconds of CPU usage is not critical, but my program always run a couple of minutes or hours, so high CPU usage will kill the system.

Black Thought

Just because a function is called a million times doesn't make it bad in itself.

What are those functions doing, and can they be done in a different way more efficiently?

--
Mats

Sounds like an infinite loop to me.....just sayin. But if that isnt it and you arent busy waiting (which pretty much is an infinite loop) the only thing that you can do is try and optimize your algorithm

Quote:

---

Originally Posted by chacham15

Sounds like an infinite loop to me.....just sayin. But if that isnt it and you arent busy waiting (which pretty much is an infinite loop) the only thing that you can do is try and optimize your algorithm

---

You're absolutely right. It may not actually be busy waiting here. The initial description said it was "continually" doing something, but it may be that what it really does is a fixed number of iterations, taking a long time overall. That is not busy waiting, just plain old high CPU usage (bottleneck).

In general decompression algorithms are normally very fast, and the bottleneck is on the disk IO. If the bottleneck in your decompression is CPU usage rather than IO, then that suggests that there is some horrible inefficiency in your decompressor. To help with that we'd need to know the algorithm used, or even have access to portions of the source code.