Thread: how to read in files that are bigger than 3gigabyte?

  1. #16
    Registered User
    Join Date
    Dec 2004
    Posts
    163
    Yes, the C and C++ IO libraries for your platform seems to choke on files larger than 4GB for 32 bit CPU because there is at most 2^32 memory address and > 4GB exceeded that.

    I also tried CreateFile, ReadFile, and WriteFile (Windows API functions), doesn't work for files larger than 4GB too.

    Thank you guys for your advices, I will go try iostreams and Boost.IOStreams.

    Another option is to buy a 64bit CPU

  2. #17
    Registered User C_ntua's Avatar
    Join Date
    Jun 2008
    Posts
    1,853
    So, fopen64(), fread64() doesn't work?

  3. #18
    Registered User Codeplug's Avatar
    Join Date
    Mar 2003
    Posts
    4,981
    >> I also tried CreateFile, ReadFile, and WriteFile (Windows API functions), doesn't work for files larger than 4GB too.
    They work for files of any size.

    >> So, fopen64(), fread64() doesn't work?
    Not part of the MS CRT. Those are in GNU libc.

    gg

  4. #19
    Registered User
    Join Date
    Dec 2004
    Posts
    163
    Quote Originally Posted by Codeplug View Post
    >> I also tried CreateFile, ReadFile, and WriteFile (Windows API functions), doesn't work for files larger than 4GB too.
    They work for files of any size.

    >> So, fopen64(), fread64() doesn't work?
    Not part of the MS CRT. Those are in GNU libc.

    gg
    I tried createfile and readfile, it doesnt work for > 4gb files
    Code:
    Char c;
    DWORD wmWritten;
    HANDLE a =CreateFile(result_file.c_str(), GENERIC_READ|GENERIC_WRITE,FILE_SHARE_READ,NULL,OPEN_ALWAYS,FILE_ATTRIBUTE_NORMAL,NULL); 
    ReadFile(a, (char *)(&c), sizeof(int), &wmWritten, NULL);
    CloseHandle(a);

  5. #20
    Kernel hacker
    Join Date
    Jul 2007
    Location
    Farncombe, Surrey, England
    Posts
    15,677
    That seems extremely unlikely, since that would be the API that created the file in the first place [unless the unlikely scenario is that you have a file created by a different OS on a removable media of some sort].

    Of course, what you are doing in ReadFile doesn't seem quite right, since I expect that Char is some sort of "char" type, and you are asking to read sizeof(int) bytes, which would be, in most systems, 4 times larger than Char.

    --
    Mats
    Compilers can produce warnings - make the compiler programmers happy: Use them!
    Please don't PM me for help - and no, I don't do help over instant messengers.

  6. #21
    Registered User
    Join Date
    Dec 2004
    Posts
    163
    Quote Originally Posted by matsp View Post

    Of course, what you are doing in ReadFile doesn't seem quite right, since I expect that Char is some sort of "char" type, and you are asking to read sizeof(int) bytes, which would be, in most systems, 4 times larger than Char.

    --
    Mats
    because the input file is in binary format and it stores integer. I am reading in 4 bytes (size int) each time and store it in var c.

    the code works for me when the file is less than 4GB

  7. #22
    Cat without Hat CornedBee's Avatar
    Join Date
    Apr 2003
    Posts
    8,895
    I am reading in 4 bytes (size int) each time and store it in var c.
    But unless c is 4 bytes large (and the type name Char suggests it isn't) you're overwriting memory you shouldn't.
    All the buzzt!
    CornedBee

    "There is not now, nor has there ever been, nor will there ever be, any programming language in which it is the least bit difficult to write bad code."
    - Flon's Law

  8. #23
    Kernel hacker
    Join Date
    Jul 2007
    Location
    Farncombe, Surrey, England
    Posts
    15,677
    Quote Originally Posted by franziss View Post
    because the input file is in binary format and it stores integer. I am reading in 4 bytes (size int) each time and store it in var c.

    the code works for me when the file is less than 4GB
    That may be pure coincidence. If Char is a char, then it can only hold ONE BYTE, and that is not enough to store 4 bytes (quite obviously).

    I wrote a piece of code that generates an 8GB file, and then opens that file again. Works fine for me. I only have about 10GB free on my hard-disk, so I can't really do a much bigger file, but seeing as the failure is either at 4GB or some MUCH larger number, I don't think it matters. [Yes, I do need to clean up my disk or rebuild my computer]

    --
    Mats
    Compilers can produce warnings - make the compiler programmers happy: Use them!
    Please don't PM me for help - and no, I don't do help over instant messengers.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. How to read files into a struct
    By verbity in forum C Programming
    Replies: 16
    Last Post: 11-24-2006, 03:39 PM
  2. Read INDEX.DAT files?
    By Queatrix in forum Windows Programming
    Replies: 3
    Last Post: 09-08-2006, 08:03 AM
  3. using threads to write & read from files
    By kishorepalle in forum C Programming
    Replies: 4
    Last Post: 10-19-2004, 05:19 PM
  4. Unknown Memory Leak in Init() Function
    By CodeHacker in forum Windows Programming
    Replies: 3
    Last Post: 07-09-2004, 09:54 AM
  5. Replies: 1
    Last Post: 07-24-2002, 06:33 AM