File map performance

Hello everyone,


I have several physical file and I want to use file map (MapViewOfFileEx) to map the file into memory to improve performance. Each file is about several hundred M bytes. All the memory mapped files are kept open during my application.

The mapping is successful, but the strange thing is,

1. the performance to access the files which are opened at first is very fast;
2. the performance to access the files which are opened later is slower and slower (the performance to access the 10th file is very bad).

Any ideas to improve performance?


thanks in advance,
George

I'd say that either
- you're running out of virtual memory space to map all those files in their entirety.
- your mapped files are ending up in the swap file (not a good idea)

What characteristic of your file access profile suggested that mapping files was a good idea?

There is nothing at all strange going on. The problem is that your expectations don't match reality, as usual.

Hi brewbuck,


Do you have any good resources/tutorials about best practices of file map on Windows? I think I need to learn some experience and my current implementation is too straightforward.

Quote:

---

Originally Posted by brewbuck

There is nothing at all strange going on. The problem is that your expectations don't match reality, as usual.

---

regards,
George

Hi Salem,


What do you mean *your file access profile*? You mean sequential access? Random access?

If you mean such context, my access pattern is sequential in my application initialization phase, and random access in running phase. Any good ideas to improve performance?

Quote:

---

Originally Posted by Salem

I'd say that either
- you're running out of virtual memory space to map all those files in their entirety.
- your mapped files are ending up in the swap file (not a good idea)

What characteristic of your file access profile suggested that mapping files was a good idea?

---

regards,
George

Well you "might" gain something from mapping the file whilst in sequential access mode, but I really don't see the point of it in random access mode.

How many random accesses per second are you looking at?
Are they read, write or update accesses?
Is that random across all the files you have open (say read record 10 of file 2, then write record 14 of file 7)?

Remember disks are horribly slow devices compared to say the speed of RAM. Mapping the file might make it seem like it's in memory, but you're not escaping the underlying reality.

Plus (if what I suspect is true), you also involve the swap file, then you have twice as much disk activity and as a result, things are much much worse.

Mapping files into the virtual space is only beneficial if the size of all the files are noticeably smaller than the amount of RAM in the machine [or at least the amount of accessed areas in the files are]. And the real benefit comes when the same area of the same file is accessed multiple times and relatively often. If you don't even write to the same place, you may just as well use fopen/fread/fwrite.

--
Mats

Hi Salem,


I am using the file to implement a hash table and it is only read access to quickly lookup something, no write/update.

For my situation, do you have any advice whether file map is faster? If not faster, what do you suggest the best way (performance efficient)?

Quote:

---

Originally Posted by Salem

Well you "might" gain something from mapping the file whilst in sequential access mode, but I really don't see the point of it in random access mode.

How many random accesses per second are you looking at?
Are they read, write or update accesses?
Is that random across all the files you have open (say read record 10 of file 2, then write record 14 of file 7)?

Remember disks are horribly slow devices compared to say the speed of RAM. Mapping the file might make it seem like it's in memory, but you're not escaping the underlying reality.

Plus (if what I suspect is true), you also involve the swap file, then you have twice as much disk activity and as a result, things are much much worse.

---

regards,
George

Thanks Mats,


I have provided the file access pattern in post #8, do you have any advice?

Quote:

---

Originally Posted by matsp

Mapping files into the virtual space is only beneficial if the size of all the files are noticeably smaller than the amount of RAM in the machine [or at least the amount of accessed areas in the files are]. And the real benefit comes when the same area of the same file is accessed multiple times and relatively often. If you don't even write to the same place, you may just as well use fopen/fread/fwrite.

--
Mats

---

regards,
George