Thread: Reading and Writing file chunks

  1. #1
    Registered User
    Join Date
    Mar 2011
    Posts
    216

    Reading and Writing file chunks

    Hi, I need some help with reading in 1 mb of binary data, and writing it to a different file. I want to continue this until the whole file has been read and written to the new one. I've searched around and I have found nothing useful. Thank you for taking the time to look.

    Code:
    #include <stdio.h>
    
    int main(int argc, char *argv[])
    {
          FILE *fp;
          FILE *out;
    
          size_t = result;
    
          int size = 1024 * 1024; // 1 mb
          int i;
    
          char *buffer;
    
          out = fopen("file.bin", "wb");
    
          for(i = 0; i < argc; i++)
          {
                fp = fopen(argv[i + 1], "rb");
                
                while(!feof(fp))
                {
                    result = fread(buffer, size, 1, fp);
                    if(result != size)  Error("Could not load into memory\n");
                    
                    if( fwrite(buffer, 1, size, out) != size)  Error("Error writing file to bin\n");
                    
                    free(buffer);
                 }
                 fclose(fp);
           }
    
           getchar();
           return 0;
    }

  2. #2
    Programming Wraith GReaper's Avatar
    Join Date
    Apr 2009
    Location
    Greece
    Posts
    2,738
    Devoted my life to programming...

  3. #3
    C lover
    Join Date
    Oct 2007
    Location
    Virginia
    Posts
    266
    Code:
    {
        result = fread(buffer, size, 1, fp);
        if(result != size)  Error("Could not load into memory\n");
                     
        if( fwrite(buffer, 1, size, out) != size)  Error("Error writing file to bin\n");
                     
        free(buffer);
    }
    I would say that you should write how much you read in this case.

    Here's what I mean:
    Code:
    {
        result = fread(buffer, size, 1, fp);
        if(result != size)  Error("Could not load into memory\n");
                     
        if( fwrite(buffer, 1, result, out) != size)  Error("Error writing file to bin\n");
                     
        free(buffer);
    }
    Last edited by Syscal; 04-29-2012 at 06:23 PM.

  4. #4
    Registered User
    Join Date
    Mar 2011
    Posts
    216
    Size is declared as 1 mb, but what if it reads the end of the file which may be 500 kb, then won't it exit with error()?

    What do you suggest I use instead of feof?
    Last edited by binks; 04-29-2012 at 06:49 PM.

  5. #5
    - - - - - - - - oogabooga's Avatar
    Join Date
    Jan 2008
    Posts
    2,808
    You can write a file in chunks something like this:
    Code:
        // FILE *fin;    //input file
        // FILE *fout;   // output file
        char buffer[4096]; // 4k buffer
        do {
            size = fread(buffer, 1, sizeof buffer, fin);
            if (size <= 0) break;
            my_fwrite(buffer, 1, size, fout);
        } while (size == sizeof buffer); // if it was a full buffer, loop again
    And you don't need to read/write a MB at a time. A few K (4 or 8) is probably as efficient as you can get.
    BTW, you didn't allocate any space for your buffer! In the code here, it needs to be allocated as an actual array so sizeof will work on it.
    The cost of software maintenance increases with the square of the programmer's creativity. - Robert D. Bliss

  6. #6
    TEIAM - problem solved
    Join Date
    Apr 2012
    Location
    Melbourne Australia
    Posts
    1,907
    size_t= result;

    Should be...

    size_t result;

    Buffer is never actually defined (It doesn't point to anything)

    free(buffer); // Doesn't do what you think it does...

  7. #7
    Registered User
    Join Date
    Jan 2009
    Posts
    1,485
    Code:
    if(result != size)  Error("Could not load into memory\n");
    This test will break your program if your file is not an exact multiple of "size", since the last fread will read less than "size".

  8. #8
    Registered User
    Join Date
    Mar 2011
    Posts
    216
    So, I shouldn't have a 1 mb char, but a 4 kb one? Won't that take forever then?

    What if the entire file has been read, with the last buffer being a full one. Won't it loop again, and then read nothing?

  9. #9
    - - - - - - - - oogabooga's Avatar
    Join Date
    Jan 2008
    Posts
    2,808
    Quote Originally Posted by binks View Post
    So, I shouldn't have a 1 mb char, but a 4 kb one? Won't that take forever then?
    Why? File operations are buffered anyway, and the buffer is unlikely to be more than 4k. It's probably only BUFSIZ (defined in stdio.h) which may only be 512 bytes! At any rate, you can always test with different buffer sizes. Try 4k and 1MB and time them (wall-clock time, not cpu time).

    Quote Originally Posted by binks View Post
    What if the entire file has been read, with the last buffer being a full one. Won't it loop again, and then read nothing?
    Yes, it will loop again, it will read nothing (detecting EOF) and the test in the middle of the loop will cause the loop to end. That's what that test is for.
    The cost of software maintenance increases with the square of the programmer's creativity. - Robert D. Bliss

  10. #10
    - - - - - - - - oogabooga's Avatar
    Join Date
    Jan 2008
    Posts
    2,808
    I just noticed that my code in post#5 uses a function my_fwrite (not a great name; shame on me). It looks like this:
    Code:
    void my_fwrite(void *data, size_t size, size_t count, FILE *fp) {
        if (fwrite(data, size, count, fp) != count)
            die("fwrite failed\n");
    }
    die prints the error message and exits.
    The cost of software maintenance increases with the square of the programmer's creativity. - Robert D. Bliss

  11. #11
    Registered User
    Join Date
    Mar 2011
    Posts
    216
    Great! It's working nicely.

    Is there a problem though if a funtion parameter is an int, but you supply a size_t? They are both integral so won't it be ok?

  12. #12
    Registered User
    Join Date
    Sep 2008
    Location
    Toronto, Canada
    Posts
    1,834
    size_t might be long long though, depending on the file system allowing greater than 4 GB files. That's why the library functions use 'size_t' so that it's flexible independent of size of long (32 bits) or long long (64 bits). And who knows how large int is - it's usually some nice machine-friendly size depending on whether the hardware and/or compiler is fundamentally 32 bit or 64 bit.

    So to answer your question - if you are passing 'int' you may not have enough size, or too much size, relative to the file system. You end up casting, but you may truncate bits.

  13. #13
    Registered User
    Join Date
    Mar 2011
    Posts
    216
    So if possible (which it is) I should change to a size_t?

  14. #14
    Registered User
    Join Date
    Sep 2008
    Location
    Toronto, Canada
    Posts
    1,834
    If you are writing a wrapper function for the file I/O library functions, then yes. That way the user is forced, outside of calling your function, to understand the correct integer size that's expected. Any casting and maybe truncation he encounters will be his doing. It won't be your problem. You have done all you can to make your wrapper function “future proof” if Microsoft decides to change the definition of size_t again. In other words, your wrapper function is always in sync with the library function.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. reading/writing to file
    By JonathanS in forum C Programming
    Replies: 2
    Last Post: 01-25-2012, 12:46 AM
  2. Reading PNG chunks.
    By Desolation in forum C++ Programming
    Replies: 8
    Last Post: 03-13-2008, 07:48 PM
  3. Reading a file in 1-kilobyte chunks...
    By Crilston in forum C Programming
    Replies: 1
    Last Post: 06-30-2005, 04:33 PM
  4. How to read a file in chunks
    By caduardo21 in forum C Programming
    Replies: 4
    Last Post: 06-22-2005, 08:09 PM
  5. Reading chunks from a file
    By Skarr in forum Linux Programming
    Replies: 1
    Last Post: 09-07-2002, 04:53 AM