Thread: fopen vs. _open (for BIG image files)

  1. #1
    Registered User
    Join Date
    Mar 2007
    Posts
    6

    Question fopen vs. _open (for BIG image files)

    Hello all,

    My program reads a large sequence of image files from the hard drive and processes them... it's working but after a certain amount of image reads the hard disk starts chugging away and it starts to crawl... the image files are 8+ megs each so i'm guessing that its overloading the cache? but is it the cache on the disk or is this the OS(windows 2k) deciding to alter some other cache? i'm assuming its a cache issue since i can restart it and it will breeze thru files its already read... i've gone back and forth from fopen to _open to windows api memory mapped files and always having the same problem... of course this issue disappears when i read a smaller sized image files... but i really need to have a sustained consistent read of these big files from the hd.. please help if you can but if not thanks for atleast lookin!

    -Roger

  2. #2
    Woof, woof! zacs7's Avatar
    Join Date
    Mar 2007
    Location
    Australia
    Posts
    3,459
    Why not use CreateFile() ?

    Quote Originally Posted by http://msdn2.microsoft.com/en-us/library/aa363858.aspx
    FILE_FLAG_NO_BUFFERING The system opens a file with no system caching. This flag does not affect hard disk caching. When combined with FILE_FLAG_OVERLAPPED, the flag gives maximum asynchronous performance, because the I/O does not rely on the synchronous operations of the memory manager. However, some I/O operations take more time, because data is not being held in the cache. Also, the file metadata may still be cached. To flush the metadata to disk, use the FlushFileBuffers function.
    That should get you consistant results.
    Last edited by zacs7; 03-31-2007 at 05:28 PM.

  3. #3
    Frequently Quite Prolix dwks's Avatar
    Join Date
    Apr 2005
    Location
    Canada
    Posts
    8,057
    fopen and _open are no doubt implemented in the same way with the same underlying code. You would probably get much the same performance with them.

    Also see this page: http://www.delorie.com/gnu/docs/glibc/libc_226.html
    dwk

    Seek and ye shall find. quaere et invenies.

    "Simplicity does not precede complexity, but follows it." -- Alan Perlis
    "Testing can only prove the presence of bugs, not their absence." -- Edsger Dijkstra
    "The only real mistake is the one from which we learn nothing." -- John Powell


    Other boards: DaniWeb, TPS
    Unofficial Wiki FAQ: cpwiki.sf.net

    My website: http://dwks.theprogrammingsite.com/
    Projects: codeform, xuni, atlantis, nort, etc.

  4. #4
    and the hat of int overfl Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    39,660
    Run task manager.
    Select the "Processes" tab.
    Then use the "View->Select Columns" menu, and tick Memory Usage and Virtual Memory Size.

    Now while your program runs, watch what resources it uses.
    If you see the memory used by your program jump every time it reads a new file, then you have a resource leak.
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Replies: 12
    Last Post: 10-17-2005, 06:49 AM
  2. Help! (serial number, image files)
    By param in forum C Programming
    Replies: 1
    Last Post: 01-18-2003, 03:54 PM
  3. Changing a treeview item's image
    By SMurf in forum Windows Programming
    Replies: 0
    Last Post: 01-14-2003, 07:08 PM
  4. How to change a mouse cursor in console mode
    By GaPe in forum Windows Programming
    Replies: 10
    Last Post: 07-03-2002, 07:42 AM
  5. *question* Reading Image Files
    By madsmile in forum C Programming
    Replies: 5
    Last Post: 03-05-2002, 12:57 PM