Fastest read from stdin

**diws** · 11-10-2013

Folks ,
I have to optimize a code for below scenario. Please help

I am reading stdin (a file redirected to stdin) character by character. How many chars are going to come is not known. After every few chars there is a seaparator. e.g $ as below

rhhrkkj$hghjhdf$ddfkrjt

While reading, if the separator arrives I'm processing the string stored before that separator and then continue reading stdin in same fashion, till EOF. I am using getc(stdin) to read chars.
Using gprof I can see most of the program time is spent inside main() , for this reading logic. Rest of the program is just some insert and search operations. I am getting time of 0.01 secs at the moment, want to reduce further.

**tabstop** · 11-10-2013

Are there natural "chunks" in your data? If your data is in lines, for instance, you can read in a line at a time and then process all the chunks in that line.

**c99tutorial** · 11-10-2013

Have you tried setvbuf? This lets you control the buffering done. Normally line buffering is the default, but in theory full buffering with the size of the buffer the same as the disk block size should be the most efficient.

Code:

int main(void)
{
    char mybuf[4096];
    setvbuf(stdin, mybuf, _IOFBF, 4096);

    int c;
    while ((c = fgetc(stdin)) != EOF) {
        putc(c, stdout);
    }
    return 0;
}

Also when you do timings you need to have a large test case to eliminate noise. You say your test is at 0.01 secs, well, that's probably mostly noise and meaningless. Why not try a test data with about 10-20Gigabytes for example and see how long it takes with full-buffering vs. the default.

**Adak** · 11-11-2013

Setting your screen buffer as mentioned above is very important for such an optimization. I had a similar problem some time back (it was a programming challenge), and i thought the algorithm was the key thing -- wrong! (It required finding several thousands of prime numbers, from the 50,000 requests it was sending).

My program was failing consistently, because the time (very short), had expired by the time I received all the requests. Made a huge difference when it was set. Passed immediately.

**diws** · 11-11-2013

I have a similar problem statement where I am reading thousands of words from stdin (redirection from a file), then searching some thousands of other words in those reference words. But i tried above and not getting a difference

The server I am uploading my code on , still says time as 0.01 sec. I tried buffer sizes 4096, 16192 ... tried 1024 as well which I found from fstat on stdin.

**rcgldr** · 11-11-2013

How much ram is available and how large is the file? You could try reading larger chunks of data into an allocated (malloc()) buffer, say 1 MB at a time, then parsing the buffer to get the data.

**c99tutorial** · 11-11-2013

Originally Posted by diws

The server I am uploading my code on , still says time as 0.01 sec. I tried buffer sizes 4096, 16192 ... tried 1024 as well

I said before this test case is simply too small. 0.01 sec is too small to tell if any difference is taking place. Try your program on an input file large enough to make it last 10-20 seconds. It means you need a file about 1000 times larger than what you've got right now.

Thread: Fastest read from stdin

Thread Tools

Search Thread

Display

Fastest read from stdin

Similar Threads

How to read input from stdin

Multiplexing read socket and stdin

using fread to read from stdin

How do you catch a read error from stdin?

fastest way to read / output strings ??

Tags for this Thread