File map performance

This is a discussion on File map performance within the C++ Programming forums, part of the General Programming Boards category; Hello everyone, I have several physical file and I want to use file map (MapViewOfFileEx) to map the file into ...

  1. #1
    Registered User
    Join Date
    May 2006
    Posts
    1,579

    File map performance

    Hello everyone,


    I have several physical file and I want to use file map (MapViewOfFileEx) to map the file into memory to improve performance. Each file is about several hundred M bytes. All the memory mapped files are kept open during my application.

    The mapping is successful, but the strange thing is,

    1. the performance to access the files which are opened at first is very fast;
    2. the performance to access the files which are opened later is slower and slower (the performance to access the 10th file is very bad).

    Any ideas to improve performance?


    thanks in advance,
    George

  2. #2
    and the hat of wrongness Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    32,558
    I'd say that either
    - you're running out of virtual memory space to map all those files in their entirety.
    - your mapped files are ending up in the swap file (not a good idea)

    What characteristic of your file access profile suggested that mapping files was a good idea?
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.
    I support http://www.ukip.org/ as the first necessary step to a free Europe.

  3. #3
    Captain Crash brewbuck's Avatar
    Join Date
    Mar 2007
    Location
    Portland, OR
    Posts
    7,243
    There is nothing at all strange going on. The problem is that your expectations don't match reality, as usual.

  4. #4
    Registered User
    Join Date
    May 2006
    Posts
    1,579
    Hi brewbuck,


    Do you have any good resources/tutorials about best practices of file map on Windows? I think I need to learn some experience and my current implementation is too straightforward.

    Quote Originally Posted by brewbuck View Post
    There is nothing at all strange going on. The problem is that your expectations don't match reality, as usual.

    regards,
    George

  5. #5
    Registered User
    Join Date
    May 2006
    Posts
    1,579
    Hi Salem,


    What do you mean *your file access profile*? You mean sequential access? Random access?

    If you mean such context, my access pattern is sequential in my application initialization phase, and random access in running phase. Any good ideas to improve performance?

    Quote Originally Posted by Salem View Post
    I'd say that either
    - you're running out of virtual memory space to map all those files in their entirety.
    - your mapped files are ending up in the swap file (not a good idea)

    What characteristic of your file access profile suggested that mapping files was a good idea?

    regards,
    George

  6. #6
    and the hat of wrongness Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    32,558
    Well you "might" gain something from mapping the file whilst in sequential access mode, but I really don't see the point of it in random access mode.

    How many random accesses per second are you looking at?
    Are they read, write or update accesses?
    Is that random across all the files you have open (say read record 10 of file 2, then write record 14 of file 7)?

    Remember disks are horribly slow devices compared to say the speed of RAM. Mapping the file might make it seem like it's in memory, but you're not escaping the underlying reality.

    Plus (if what I suspect is true), you also involve the swap file, then you have twice as much disk activity and as a result, things are much much worse.
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.
    I support http://www.ukip.org/ as the first necessary step to a free Europe.

  7. #7
    Kernel hacker
    Join Date
    Jul 2007
    Location
    Farncombe, Surrey, England
    Posts
    15,677
    Mapping files into the virtual space is only beneficial if the size of all the files are noticeably smaller than the amount of RAM in the machine [or at least the amount of accessed areas in the files are]. And the real benefit comes when the same area of the same file is accessed multiple times and relatively often. If you don't even write to the same place, you may just as well use fopen/fread/fwrite.

    --
    Mats
    Compilers can produce warnings - make the compiler programmers happy: Use them!
    Please don't PM me for help - and no, I don't do help over instant messengers.

  8. #8
    Registered User
    Join Date
    May 2006
    Posts
    1,579
    Hi Salem,


    I am using the file to implement a hash table and it is only read access to quickly lookup something, no write/update.

    For my situation, do you have any advice whether file map is faster? If not faster, what do you suggest the best way (performance efficient)?

    Quote Originally Posted by Salem View Post
    Well you "might" gain something from mapping the file whilst in sequential access mode, but I really don't see the point of it in random access mode.

    How many random accesses per second are you looking at?
    Are they read, write or update accesses?
    Is that random across all the files you have open (say read record 10 of file 2, then write record 14 of file 7)?

    Remember disks are horribly slow devices compared to say the speed of RAM. Mapping the file might make it seem like it's in memory, but you're not escaping the underlying reality.

    Plus (if what I suspect is true), you also involve the swap file, then you have twice as much disk activity and as a result, things are much much worse.

    regards,
    George

  9. #9
    Registered User
    Join Date
    May 2006
    Posts
    1,579
    Thanks Mats,


    I have provided the file access pattern in post #8, do you have any advice?

    Quote Originally Posted by matsp View Post
    Mapping files into the virtual space is only beneficial if the size of all the files are noticeably smaller than the amount of RAM in the machine [or at least the amount of accessed areas in the files are]. And the real benefit comes when the same area of the same file is accessed multiple times and relatively often. If you don't even write to the same place, you may just as well use fopen/fread/fwrite.

    --
    Mats

    regards,
    George

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. sequential file program
    By needhelpbad in forum C Programming
    Replies: 80
    Last Post: 06-08-2008, 01:04 PM
  2. help with text input
    By Alphawaves in forum C Programming
    Replies: 8
    Last Post: 04-08-2007, 04:54 PM
  3. Replies: 3
    Last Post: 03-04-2005, 01:46 PM
  4. System
    By drdroid in forum C++ Programming
    Replies: 3
    Last Post: 06-28-2002, 10:12 PM
  5. Need a suggestion on a school project..
    By Screwz Luse in forum C Programming
    Replies: 5
    Last Post: 11-27-2001, 01:58 AM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21