Thread: Read/Write large arrays from/to a file (efficient plz)

Hi,

as final result I want to read/write per random access in binary files to retrieve a block/part of a really large array.

So how could one write such an array as fast as possible to disk?
I know about using fwrite(myArray, sizeof(int), arraySize, myFile); but there is such a thing as setvbuf.

So how could one determine the best buffer size and what does it look like in detail as setvbuf call? I found different use of setvbuf for example NULL as buffer.

Any help, ideas would be most appreciated.

Mat.

Okay,

so basically you say adjusting the buffer size for fwrite/fread doesn't do any good? I was still under the impression an optimal size has yet to be found ... Thank you for this one.

I was thinking on reading the array in two ways:

1st
read it in one fread(myArray, sizeof(int), arraySize, file)

2nd
read it in BUFSIZ chunks:
for (i = 0; i < arraySize; i += BUFSIZ)
fread(myArray + i, sizeof(int), BUFSIZ, file);

So normally I thought the 1st one was the one to get but surprinsingly then second one runs faster (as done in multiple loops to determine the time run).

I'm still confused about the optimizing I/O thing in C =(

Thank you, this solved the riddle =)

So I learned that reading in BUFSIZ chunks is the most efficient (and therefor fastest) way of doing multiple I/Os.
Also one doesn't need to read in these chunks by oneself because by reading/writing arrays of multiple BUFSIZ the system will take care of it.

Thanks a lot =)

Mat.