Thread: Handling f-write

Originally Posted by kodax

From my observations, speed is reduced with large 8mb buffers, as compared to 256kb buffers..
Without a doubt, large buffers decrease speed significantly.

Exactly what is your program doing with the buffer? Are you reading from a file into a large buffer then writing it to another file? Many things could be going on that can cause a large buffer to slow you down vs a smaller buffer, depending on exactly what you're doing. Besides that, there's definitely a case of diminishing returns with buffer size for any given application.

But again, what exactly is your question or issue? Your spewing of random dribbles of statements and theories and irrelevant links to forum posts or books makes it hard to see what your point is.

Mostly reading and writing..

in that case,
all the files from the user's directory.

Why not FAFO, and test it yourself? This is my laptop's HDD performance for reading a 56MB file with different block sizes:

Code:

0.5k  3.3 MB/s
1k    6.8 MB/s
4k    44.5 MB/s
16k   157 MB/s
32k   286 MB/s
64k   502 MB/s
128k  836 MB/s
256k  581 MB/s
512k  795 MB/s
1024k 866 MB/s

Originally Posted by flp1969

This contradicts the assumption.
Best speeds in green.

$ i=4096; while [ $i -le $(bc <<< '2^25') ]; do echo -ne "\e[1;33m$i\e[m: "; dd if=/dev/zero of=./file.bin bs=$i count=100 2>&1 | sed -nE '/B\/s/p'; i=$((i*2)); done
4096: 409600 bytes (410 kB, 400 KiB) copied, 0,000410758 s, 997 MB/s
8192: 819200 bytes (819 kB, 800 KiB) copied, 0,000500906 s, 1,6 GB/s
16384: 1638400 bytes (1,6 MB, 1,6 MiB) copied, 0,000894711 s, 1,8 GB/s
32768: 3276800 bytes (3,3 MB, 3,1 MiB) copied, 0,00154947 s, 2,1 GB/s
65536: 6553600 bytes (6,6 MB, 6,2 MiB) copied, 0,002985 s, 2,2 GB/s
131072: 13107200 bytes (13 MB, 12 MiB) copied, 0,0414716 s, 316 MB/s
262144: 26214400 bytes (26 MB, 25 MiB) copied, 0,0672004 s, 390 MB/s
524288: 52428800 bytes (52 MB, 50 MiB) copied, 0,131374 s, 399 MB/s
1048576: 104857600 bytes (105 MB, 100 MiB) copied, 0,224383 s, 467 MB/s
2097152: 209715200 bytes (210 MB, 200 MiB) copied, 0,480652 s, 436 MB/s
4194304: 419430400 bytes (419 MB, 400 MiB) copied, 0,959415 s, 437 MB/s
8388608: 838860800 bytes (839 MB, 800 MiB) copied, 2,00964 s, 417 MB/s
16777216: 1677721600 bytes (1,7 GB, 1,6 GiB) copied, 9,57332 s, 175 MB/s
33554432: 3355443200 bytes (3,4 GB, 3,1 GiB) copied, 23,2783 s, 144 MB/s

Measuring anything over 0,000500906s seems to be a bit optimistic.

It does also highlight that the performance of reads and writes is largely dependent on the what you are reading/writing from.