Read/Write large arrays from/to a file (efficient plz)

Hi,

as final result I want to read/write per random access in binary files to retrieve a block/part of a really large array.

So how could one write such an array as fast as possible to disk?
I know about using fwrite(myArray, sizeof(int), arraySize, myFile); but there is such a thing as setvbuf.

So how could one determine the best buffer size and what does it look like in detail as setvbuf call? I found different use of setvbuf for example NULL as buffer.

Any help, ideas would be most appreciated.

Mat.

> So how could one write such an array as fast as possible to disk?
Well your fwrite() call seems pretty good to me.

> I found different use of setvbuf for example NULL as buffer.
Which is the same as using BUFSIZ as the size, which is almost certainly the size it has by default anyway.

If the arrays are really as big as you seem to imply, then performance is going to be dominated by physical disk performance, not processor-memory performance.

Okay,

so basically you say adjusting the buffer size for fwrite/fread doesn't do any good? I was still under the impression an optimal size has yet to be found ... Thank you for this one.

I was thinking on reading the array in two ways:

1st
read it in one fread(myArray, sizeof(int), arraySize, file)

2nd
read it in BUFSIZ chunks:
for (i = 0; i < arraySize; i += BUFSIZ)
fread(myArray + i, sizeof(int), BUFSIZ, file);

So normally I thought the 1st one was the one to get but surprinsingly then second one runs faster (as done in multiple loops to determine the time run).

I'm still confused about the optimizing I/O thing in C =(

BUFSIZ is chosen to be an averagely optimal size for your system.

> but surprinsingly then second one runs faster
Well if you ran both tests in sequence one after the other, it's probably because the file was already cached, and thus far quicker to read in the second case.
Try swapping the tests over.

You might need to average out over many tests

Code:

---

for ( i = 0 ; i < 10 ; i++ ) {
    test1();
    test2();
}

---

Thank you, this solved the riddle =)

So I learned that reading in BUFSIZ chunks is the most efficient (and therefor fastest) way of doing multiple I/Os.
Also one doesn't need to read in these chunks by oneself because by reading/writing arrays of multiple BUFSIZ the system will take care of it.

Thanks a lot =)

Mat.