mmap-ing files > 4GB?

zxcv · 06-09-2008, 12:32 AM

Hello

Odd question, is it possible to safely and sanely mmap a file (with a 2.6 kernel, if it matters) that is greater than 4 gigs, on a 32bit machine?

I'm toying around with some whole partition encryption and I think this would simplify things. While the file size should be arbitrary, my 'execution unit' is fixed. The number of 'execution units' that I will tackle at once is also arbitrary.

Im hoping this would simplify I/O by saving me from managing these execution units and having the reads/writes be optimized by the Virtual Memory System directly.

Thanks

matsp · 06-09-2008, 02:01 AM

You can mmap() sections of a file up to about 2.8GB (your user-mode application is allowed to access 3GB of memory, and some of that is already taken by your application, the stack and the "NULL-dereference catching area" (the area between 0 and beginning of your code) . But you can supply a 64-bit offset to anywhere within that file. Look up the man page for mmap.

However, using mmap if you are only reading/writing once is probably not giving the best performance. Read/Write are much more optimized for this pattern, whilst mmap() is best for man read/write within the same section of a file. There is a thread a few weeks old about copying in Linux, and the performance of mmap() vs. read/write gives the read/write a win with a fair margin (about 50% or so, I think). I'll see if I can find the thread and link it here.

--
Mats

Elysia · 06-09-2008, 03:03 AM

Mapping files bigger than ~2.8 GB won't work since the OS cannot address that much memory. 32-bit limitations.

matsp · 06-09-2008, 03:09 AM

Quote:

Originally Posted by Elysia

Mapping files bigger than ~2.8 GB won't work since the OS cannot address that much memory. 32-bit limitations.

You can mmap into files that are bigger than 2.8GB, but as I described above, the size of the section mmap'd will be restricted to about 2.8GB - and yes, that's a 32-bit limitation, since the 32-bit address space is split between user-mode and kernel mode, with the kernel getting 1GB of address space, and 4GB-1GB -> 3GB user-mode address range.

The only way around that would be to use 64-bit OS, which would allow you to map up to 48 bits worth of memory, which is about 1TB if memory serves. If you have partitions or files that are bigger than that, then they still need to use an offset.

Man-page:
http://www.hmug.org/man/2/MMAP.php

--
Mats

zacs7 · 06-09-2008, 03:14 AM

> The only way around that would be to use 64-bit OS, which would allow you to map up to 48 bits worth of memory, which is about 1TB if memory serves.
Isn't it 256TB?

2^48 B / 2^40 B = 2^8 TB?

matsp · 06-09-2008, 03:15 AM

And here is the thread I was talking about:
using mmap for copying large files

--
Mats

brewbuck · 06-09-2008, 09:34 PM

Quote:

Originally Posted by Elysia

Mapping files bigger than ~2.8 GB won't work since the OS cannot address that much memory. 32-bit limitations.

Right, and because my eyes can only see the side of the room which lies directly in front of me, the rest of the room must not exist.

matsp · 06-10-2008, 01:57 AM

Quote:

Originally Posted by zacs7

> The only way around that would be to use 64-bit OS, which would allow you to map up to 48 bits worth of memory, which is about 1TB if memory serves.
Isn't it 256TB?

2^48 B / 2^40 B = 2^8 TB?

Sorry, I missed this comment yesterday.

Yes, of course it is - I got myself confused with the 2^40 bytes of physical memroy that the older Athlon64 processors would accommodate - newer ones have a 2^48 physical address bus, so can (very theoretical at present [1]) accommodate 256TB of virtual and physical memory too.

[1] Since the Athlon 64 memory controller is the limitation, and it is limited to a maximum of 8 sticks of memory, and the biggest machine configs are 8 sockets, you'd be hard pushed to get beyond some hundreds of megabytes - but it's certainly a good thing to prepare for the > 1TB memory range that will certainly be used at some point in the not too distant future.

--
Mats

Elysia · 06-10-2008, 04:05 AM

Quote:

Originally Posted by brewbuck

Right, and because my eyes can only see the side of the room which lies directly in front of me, the rest of the room must not exist.

Right, right. You cannot map more than than 2.8 GB of the file in memory at once, due to 32-bit limitation.
Of course, it doesn't stop you from dissecting the file and working with several 2.8 GB parts.

06-09-2008, 12:32 AM	#1
zxcv Registered User Join Date: May 2006 Posts: 52	mmap-ing files > 4GB? Hello Odd question, is it possible to safely and sanely mmap a file (with a 2.6 kernel, if it matters) that is greater than 4 gigs, on a 32bit machine? I'm toying around with some whole partition encryption and I think this would simplify things. While the file size should be arbitrary, my 'execution unit' is fixed. The number of 'execution units' that I will tackle at once is also arbitrary. Im hoping this would simplify I/O by saving me from managing these execution units and having the reads/writes be optimized by the Virtual Memory System directly. Thanks
zxcv is offline

06-09-2008, 02:01 AM	#2
matsp Kernel hacker Join Date: Jul 2007 Location: Farncombe, Surrey, England Posts: 15,686	You can mmap() sections of a file up to about 2.8GB (your user-mode application is allowed to access 3GB of memory, and some of that is already taken by your application, the stack and the "NULL-dereference catching area" (the area between 0 and beginning of your code) . But you can supply a 64-bit offset to anywhere within that file. Look up the man page for mmap. However, using mmap if you are only reading/writing once is probably not giving the best performance. Read/Write are much more optimized for this pattern, whilst mmap() is best for man read/write within the same section of a file. There is a thread a few weeks old about copying in Linux, and the performance of mmap() vs. read/write gives the read/write a win with a fair margin (about 50% or so, I think). I'll see if I can find the thread and link it here. -- Mats __________________ Compilers can produce warnings - make the compiler programmers happy: Use them! Please don't PM me for help - and no, I don't do help over instant messengers.
matsp is offline

06-09-2008, 03:14 AM	#5
zacs7 Woof, woof! Join Date: Mar 2007 Location: Australia Posts: 3,288	> The only way around that would be to use 64-bit OS, which would allow you to map up to 48 bits worth of memory, which is about 1TB if memory serves. Isn't it 256TB? 2^48 B / 2^40 B = 2^8 TB? __________________ "I.T. gets the chicky-babes" - M. Kelly bakefile \| vim
zacs7 is offline

06-09-2008, 03:15 AM	#6
matsp Kernel hacker Join Date: Jul 2007 Location: Farncombe, Surrey, England Posts: 15,686	And here is the thread I was talking about: using mmap for copying large files -- Mats __________________ Compilers can produce warnings - make the compiler programmers happy: Use them! Please don't PM me for help - and no, I don't do help over instant messengers.
matsp is offline

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
I Need To Know Some Things That I Can Put Into A Batch File	TheRealNapster	A Brief History of Cprogramming.com	1	10-20-2003 08:12 PM
Dos commands hehe	Carp	A Brief History of Cprogramming.com	2	01-17-2003 02:51 PM
Using c++ standards	subdene	C++ Programming	4	06-06-2002 09:15 AM
reinserting htm files into chm help files	verb	Windows Programming	0	02-15-2002 09:35 AM
displaying text files, wierd thing :(	Gades	C Programming	2	11-20-2001 05:18 PM