fopen vs. _open (for BIG image files)

Hello all,

My program reads a large sequence of image files from the hard drive and processes them... it's working but after a certain amount of image reads the hard disk starts chugging away and it starts to crawl... the image files are 8+ megs each so i'm guessing that its overloading the cache? but is it the cache on the disk or is this the OS(windows 2k) deciding to alter some other cache? i'm assuming its a cache issue since i can restart it and it will breeze thru files its already read... i've gone back and forth from fopen to _open to windows api memory mapped files and always having the same problem... of course this issue disappears when i read a smaller sized image files... but i really need to have a sustained consistent read of these big files from the hd.. please help if you can but if not thanks for atleast lookin!

-Roger

Why not use CreateFile() ?

Originally Posted by http://msdn2.microsoft.com/en-us/library/aa363858.aspx

FILE_FLAG_NO_BUFFERING The system opens a file with no system caching. This flag does not affect hard disk caching. When combined with FILE_FLAG_OVERLAPPED, the flag gives maximum asynchronous performance, because the I/O does not rely on the synchronous operations of the memory manager. However, some I/O operations take more time, because data is not being held in the cache. Also, the file metadata may still be cached. To flush the metadata to disk, use the FlushFileBuffers function.

That should get you consistant results.

fopen and _open are no doubt implemented in the same way with the same underlying code. You would probably get much the same performance with them.

Also see this page: http://www.delorie.com/gnu/docs/glibc/libc_226.html

Run task manager.
Select the "Processes" tab.
Then use the "View->Select Columns" menu, and tick Memory Usage and Virtual Memory Size.

Now while your program runs, watch what resources it uses.
If you see the memory used by your program jump every time it reads a new file, then you have a resource leak.