Thread: file IO speed over network

You could also cache the files locally in memory or on disk. Check the last modified timestamp of the file(s) on the network share at some interval, and if newer update the cache.

Originally Posted by MK27

An megabit is about 1 kilobyte.

1 MBit is 125 or 128 KByte depending on if you count Kilo as 1000 or 1024

Originally Posted by _Mike

1 MBit is 125 or 128 KByte depending on if you could Kilo as 1000 or 1024

Actually we're both wrong, lol. A megabit (100000 bits) is 12500 bytes, which is 12.2 kB (using a 1024 byte kB).

Originally Posted by jimblumberg

Reading a file character by character is slower than reading a large section of the file at a time.

And that difference will probably be compounded on a network, presuming each read requires some kind of in-protocol packet exchange as overhead, which is going to be a number of bytes.

Ie, if you read one byte at a time and there are 9 bytes of packet overhead per request, your 100 MBits/sec connection will be 10 Mbits effectively. If you are reading a line at a time, the lines are ~40 bytes, and the overhead is 8 bytes, you will get at most 80% of the network maximum.

If the files are not huge, stat() them and read the entire thing in at once. Do not use getline().

Originally Posted by _Mike

You're missing a 0 in mega

Yeah, more than once since I did that on a calculator too. I need a less mathematically impaired brain. :/

Originally Posted by rogster001

This is what i was thinking, could you outline any alternative? I can get filesize, read into a buffer, but how to extract the lines after that? as i need to provide a rowcount as part of the reporting

Stringstreams also have a getline() function. Read into string buffer, apply stringstream, and all your current parsing code can work with that.

To be honest though, I doubt that will make much difference in terms of getting the network speed to better match the speed of a hard drive -- but it is probably a more polite use of the network & equipment. Imagine if you had three computers reading three different files off the same hard drive at the same time, one line at a time.