Thread: Creating a memory mapped file

Having the file memory mapped may make it easier to access the data efficiently, but memory mapping doesn't necessarily mean that the whole file will be in memory -- it will just ACT like it is. Depending on how much physical RAM is available you may have paging.

Anyway, the general method is to open() the file and then call mmap() on the resulting file descriptor. The manual pages for these two functions should be enough to get started.

Or you can use Boost.Interprocess, which makes this very easy.

I'd like to also point out that mapping a 1GB file into memory will use up 1GB of virtual address space. This is not a problem, but if your file(s) grow to more than 3GB [1] on a 32-bit system, you will not be able to map the entire file into RAM. If you have a 64-bit version of Linux, this is not a problem - the limit is around 47 bits of memory address - which is 32768 times 4GB - so you need a few hundred 500GB hard disks to hold such a file.... (32768 / 125, whatever that is).

Using such a memory mapped file is faster if there are many reads and/or writes to the same area of the file, and particularly so if there are multiple processes that are using the same file (but this also requires careful locking of the data accesses).

In previous posts, it's been shown that a traditional copy of a mmap'd file is slower than reading and writing the file using fread/fwrite. This is of course not the same as a sorting operation, where a particular element of the file may be read multiple times. Of course, searching of unsorted data should not read more than once (and searching sorted data of fixed size can be done without reading much of the data at all), nor should "writing it to disk when you are done" be any effect in this. So it seems like sorting is the only case where any of the file-content is being read more than once.

It may be better for general use to load the file content (on demand - that is, when needed) into ram using the regular file read and write operations in C or C++.

[1] Actually, 3GB minus a couple of hundred megabytes or so, as Linux reserves the first 128MB to catch null pointers, and the very top of the memory is used as stack-space, and some other areas of the user memory may also be taken by various other components. So the actual size that you could possibly map is somewhat less than 3GB, but higher than 2GB. Exactly where in between those two numbers you'd end up depends on many factors, and it would be difficult to enumerate ALL of those, and under which conditions they are relevant.

--
Mats

I'd like to also point out that mapping a 1GB file into memory will use up 1GB of virtual address space.

Note that you can partially map files if your use cases let you get away with it.

Originally Posted by CornedBee

Note that you can partially map files if your use cases let you get away with it.

Yes, of course. But in this case, it appears that the user actually WANTS to use ALL of the file at once, which means that it will use the size of the file's worth of virtual space.

--
Mats