help with my mIRC log file stat generator.
For about the last month I've been working on a stat generator for mIRC log files, similar to mIRC Stats, but of course free.
I have it working well, but it is very very slow. A <1 meg log file takes nearly a minute to complete the scan. A log file of 9 megs takes about 10 minutes or so, and the 17 meg file has never completed (I didn't try running that one, a freind did).
Right now I'm reading the file in, line by line, using fgets. I then break that line into word by word, and then reconstruct the line from the re-built words (rebuilding does stuff such as stripping +%@ symbols). I can then either compare the file using the entire line, or individual words from that line.
This is where it goes slow... It has to re-scan and re-build the log file everytime it makes a new analysis. If you are testing 10 different things, then it must read the entire log 10 seperate times, re-building it word for word everytime. Because of the way I have it set up, which is very accurate, I cannot run all 10 stats in one pass of the log file without the source code getting so confusing that I would get lost.
So, is there a quicker, easier way of doing things? I was thinking of reading the entire log file into a linked list. The "meat" of the list would be the entire re-built line, then each link after that would be like a new line of the log file. This would essentially build up the entire log file inside of memory, making it much quicker to make the multiple passes.
But if I read the entire 17 meg log file into memory, wouldn't it take up 17+ megs of memory? And 17 megs is only half a month worth of logs, it could be easily looking at 35+ megs at the end of the month. Even with the high-end systems of today I can't afford to waste that much memory (and I have 512 megs).
So, is there a better way to do this? I know it's possible, mIRC Stats can scan the log file, and do it quite quickly, but I don't have the slightest clue how they did it.
(If you want to see the source, let me know. It's over 20 pages so I didn't want to post it here, and I don't think unregistered people can upload files to here)