Trying to make this run faster
First, I'd like to say that I'm new here and I hope to find the help I need.
I wrote a c program that reads a GIGANTIC file (~16GB) line by line using fscanf in a while loop as show below
I'm supposed to output a result every million lines; the problem that this thing takes A LONG LONG TIME. It's been weeks so far and it's not done (running on a quad-core opteron).
while(fscanf(file, "%d %d %d", &inst_type, &in_delay, &sp_type ) != EOF)
// here I do simple processing with the variables read from the file
Anyways, if I don't read the file and instead generate those variables randomly, the program runs very quickly (few minutes to finish).
I tried to output something(say result y) every 100 thousand lines and see how fast it's going and I noticed the following.
The first 10 y's came out quickly (they followed one another in an almost fixed interval of a one second or two), but then the y's started to come out slower and slower. It was like becoming exponentially SLOWER.
I don't know what the problem is. Is it the way I'm reading the file? Is there a buffer somewhere that's getting huge and making things dog slow? I really don't know.
Does anyone have any idea about this?
I also tried to profile (using gprof ) the program, but did not have luck getting it to work. There was all kids of errors that gprof generated and I'm no mood getting busy with those.