Thread: mmap-ing files > 4GB?

  1. #1
    Registered User
    Join Date
    May 2006
    Posts
    57

    mmap-ing files > 4GB?

    Hello

    Odd question, is it possible to safely and sanely mmap a file (with a 2.6 kernel, if it matters) that is greater than 4 gigs, on a 32bit machine?

    I'm toying around with some whole partition encryption and I think this would simplify things. While the file size should be arbitrary, my 'execution unit' is fixed. The number of 'execution units' that I will tackle at once is also arbitrary.

    Im hoping this would simplify I/O by saving me from managing these execution units and having the reads/writes be optimized by the Virtual Memory System directly.

    Thanks

  2. #2
    Kernel hacker
    Join Date
    Jul 2007
    Location
    Farncombe, Surrey, England
    Posts
    15,677
    You can mmap() sections of a file up to about 2.8GB (your user-mode application is allowed to access 3GB of memory, and some of that is already taken by your application, the stack and the "NULL-dereference catching area" (the area between 0 and beginning of your code) . But you can supply a 64-bit offset to anywhere within that file. Look up the man page for mmap.

    However, using mmap if you are only reading/writing once is probably not giving the best performance. Read/Write are much more optimized for this pattern, whilst mmap() is best for man read/write within the same section of a file. There is a thread a few weeks old about copying in Linux, and the performance of mmap() vs. read/write gives the read/write a win with a fair margin (about 50% or so, I think). I'll see if I can find the thread and link it here.

    --
    Mats
    Compilers can produce warnings - make the compiler programmers happy: Use them!
    Please don't PM me for help - and no, I don't do help over instant messengers.

  3. #3
    C++まいる!Cをこわせ!
    Join Date
    Oct 2007
    Location
    Inside my computer
    Posts
    24,654
    Mapping files bigger than ~2.8 GB won't work since the OS cannot address that much memory. 32-bit limitations.
    Quote Originally Posted by Adak View Post
    io.h certainly IS included in some modern compilers. It is no longer part of the standard for C, but it is nevertheless, included in the very latest Pelles C versions.
    Quote Originally Posted by Salem View Post
    You mean it's included as a crutch to help ancient programmers limp along without them having to relearn too much.

    Outside of your DOS world, your header file is meaningless.

  4. #4
    Kernel hacker
    Join Date
    Jul 2007
    Location
    Farncombe, Surrey, England
    Posts
    15,677
    Quote Originally Posted by Elysia View Post
    Mapping files bigger than ~2.8 GB won't work since the OS cannot address that much memory. 32-bit limitations.
    You can mmap into files that are bigger than 2.8GB, but as I described above, the size of the section mmap'd will be restricted to about 2.8GB - and yes, that's a 32-bit limitation, since the 32-bit address space is split between user-mode and kernel mode, with the kernel getting 1GB of address space, and 4GB-1GB -> 3GB user-mode address range.

    The only way around that would be to use 64-bit OS, which would allow you to map up to 48 bits worth of memory, which is about 1TB if memory serves. If you have partitions or files that are bigger than that, then they still need to use an offset.

    Man-page:
    http://www.hmug.org/man/2/MMAP.php

    --
    Mats
    Compilers can produce warnings - make the compiler programmers happy: Use them!
    Please don't PM me for help - and no, I don't do help over instant messengers.

  5. #5
    Woof, woof! zacs7's Avatar
    Join Date
    Mar 2007
    Location
    Australia
    Posts
    3,459
    > The only way around that would be to use 64-bit OS, which would allow you to map up to 48 bits worth of memory, which is about 1TB if memory serves.
    Isn't it 256TB?

    2^48 B / 2^40 B = 2^8 TB?

  6. #6
    Kernel hacker
    Join Date
    Jul 2007
    Location
    Farncombe, Surrey, England
    Posts
    15,677
    And here is the thread I was talking about:
    http://cboard.cprogramming.com/showthread.php?t=102991

    --
    Mats
    Compilers can produce warnings - make the compiler programmers happy: Use them!
    Please don't PM me for help - and no, I don't do help over instant messengers.

  7. #7
    Officially An Architect brewbuck's Avatar
    Join Date
    Mar 2007
    Location
    Portland, OR
    Posts
    7,396
    Quote Originally Posted by Elysia View Post
    Mapping files bigger than ~2.8 GB won't work since the OS cannot address that much memory. 32-bit limitations.
    Right, and because my eyes can only see the side of the room which lies directly in front of me, the rest of the room must not exist.

  8. #8
    Kernel hacker
    Join Date
    Jul 2007
    Location
    Farncombe, Surrey, England
    Posts
    15,677
    Quote Originally Posted by zacs7 View Post
    > The only way around that would be to use 64-bit OS, which would allow you to map up to 48 bits worth of memory, which is about 1TB if memory serves.
    Isn't it 256TB?

    2^48 B / 2^40 B = 2^8 TB?
    Sorry, I missed this comment yesterday.

    Yes, of course it is - I got myself confused with the 2^40 bytes of physical memroy that the older Athlon64 processors would accommodate - newer ones have a 2^48 physical address bus, so can (very theoretical at present [1]) accommodate 256TB of virtual and physical memory too.

    [1] Since the Athlon 64 memory controller is the limitation, and it is limited to a maximum of 8 sticks of memory, and the biggest machine configs are 8 sockets, you'd be hard pushed to get beyond some hundreds of megabytes - but it's certainly a good thing to prepare for the > 1TB memory range that will certainly be used at some point in the not too distant future.

    --
    Mats
    Compilers can produce warnings - make the compiler programmers happy: Use them!
    Please don't PM me for help - and no, I don't do help over instant messengers.

  9. #9
    C++まいる!Cをこわせ!
    Join Date
    Oct 2007
    Location
    Inside my computer
    Posts
    24,654
    Quote Originally Posted by brewbuck View Post
    Right, and because my eyes can only see the side of the room which lies directly in front of me, the rest of the room must not exist.
    Right, right. You cannot map more than than 2.8 GB of the file in memory at once, due to 32-bit limitation.
    Of course, it doesn't stop you from dissecting the file and working with several 2.8 GB parts.
    Quote Originally Posted by Adak View Post
    io.h certainly IS included in some modern compilers. It is no longer part of the standard for C, but it is nevertheless, included in the very latest Pelles C versions.
    Quote Originally Posted by Salem View Post
    You mean it's included as a crutch to help ancient programmers limp along without them having to relearn too much.

    Outside of your DOS world, your header file is meaningless.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. I Need To Know Some Things That I Can Put Into A Batch File
    By TheRealNapster in forum A Brief History of Cprogramming.com
    Replies: 1
    Last Post: 10-20-2003, 08:12 PM
  2. Dos commands hehe
    By Carp in forum A Brief History of Cprogramming.com
    Replies: 2
    Last Post: 01-17-2003, 02:51 PM
  3. Using c++ standards
    By subdene in forum C++ Programming
    Replies: 4
    Last Post: 06-06-2002, 09:15 AM
  4. reinserting htm files into chm help files
    By verb in forum Windows Programming
    Replies: 0
    Last Post: 02-15-2002, 09:35 AM
  5. displaying text files, wierd thing :(
    By Gades in forum C Programming
    Replies: 2
    Last Post: 11-20-2001, 05:18 PM

Tags for this Thread