Read/Write large arrays from/to a file (efficient plz)

This is a discussion on Read/Write large arrays from/to a file (efficient plz) within the C Programming forums, part of the General Programming Boards category; Hi, as final result I want to read/write per random access in binary files to retrieve a block/part of a ...

  1. #1
    Registered User
    Join Date
    Jan 2005
    Posts
    4

    Read/Write large arrays from/to a file (efficient plz)

    Hi,

    as final result I want to read/write per random access in binary files to retrieve a block/part of a really large array.

    So how could one write such an array as fast as possible to disk?
    I know about using fwrite(myArray, sizeof(int), arraySize, myFile); but there is such a thing as setvbuf.

    So how could one determine the best buffer size and what does it look like in detail as setvbuf call? I found different use of setvbuf for example NULL as buffer.

    Any help, ideas would be most appreciated.

    Mat.

  2. #2
    and the hat of wrongness Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    32,592
    > So how could one write such an array as fast as possible to disk?
    Well your fwrite() call seems pretty good to me.

    > I found different use of setvbuf for example NULL as buffer.
    Which is the same as using BUFSIZ as the size, which is almost certainly the size it has by default anyway.

    If the arrays are really as big as you seem to imply, then performance is going to be dominated by physical disk performance, not processor-memory performance.
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.
    I support http://www.ukip.org/ as the first necessary step to a free Europe.

  3. #3
    Registered User
    Join Date
    Jan 2005
    Posts
    4
    Okay,

    so basically you say adjusting the buffer size for fwrite/fread doesn't do any good? I was still under the impression an optimal size has yet to be found ... Thank you for this one.

    I was thinking on reading the array in two ways:

    1st
    read it in one fread(myArray, sizeof(int), arraySize, file)

    2nd
    read it in BUFSIZ chunks:
    for (i = 0; i < arraySize; i += BUFSIZ)
    fread(myArray + i, sizeof(int), BUFSIZ, file);

    So normally I thought the 1st one was the one to get but surprinsingly then second one runs faster (as done in multiple loops to determine the time run).

    I'm still confused about the optimizing I/O thing in C =(

  4. #4
    and the hat of wrongness Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    32,592
    BUFSIZ is chosen to be an averagely optimal size for your system.

    > but surprinsingly then second one runs faster
    Well if you ran both tests in sequence one after the other, it's probably because the file was already cached, and thus far quicker to read in the second case.
    Try swapping the tests over.

    You might need to average out over many tests
    Code:
    for ( i = 0 ; i < 10 ; i++ ) {
        test1();
        test2();
    }
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.
    I support http://www.ukip.org/ as the first necessary step to a free Europe.

  5. #5
    Registered User
    Join Date
    Jan 2005
    Posts
    4
    Thank you, this solved the riddle =)

    So I learned that reading in BUFSIZ chunks is the most efficient (and therefor fastest) way of doing multiple I/Os.
    Also one doesn't need to read in these chunks by oneself because by reading/writing arrays of multiple BUFSIZ the system will take care of it.

    Thanks a lot =)

    Mat.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. File Writing Problem
    By polskash in forum C Programming
    Replies: 3
    Last Post: 02-13-2009, 09:47 AM
  2. Formatting a text file...
    By dagorsul in forum C Programming
    Replies: 12
    Last Post: 05-02-2008, 03:53 AM
  3. C++ std routines
    By siavoshkc in forum C++ Programming
    Replies: 33
    Last Post: 07-28-2006, 12:13 AM
  4. Read/Write linked lists from/to file
    By XorpiZ in forum C Programming
    Replies: 35
    Last Post: 01-29-2005, 07:13 PM
  5. Need a suggestion on a school project..
    By Screwz Luse in forum C Programming
    Replies: 5
    Last Post: 11-27-2001, 01:58 AM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21