Thread: Buffering??

I've been away for months from C and i came back to it lately with lots of misconceptions so bear with me.

correct me if im wrong :
- input functions like scanf() buffers whatever you type then once you hit <enter> it saves it to the variable\array you've chosen.

now my questions :

1. output functions buffer output before displaying it right? when and why?

2. How is it possible that a function leaves some characters in the buffer? and do different functions\streams use the same buffer?

3. why does 'fgets()' terminate when 'scanf()' precede it , because 'scanf()' leaves '\n' in the stream or the buffer?

4. how do 'fgets()' know that the '\n' is in the buffer\stream? does it like search it before you have the chance to enter anything or what?

I'm 90% certain on these things...

Your console can either be in "cooked" versus "raw" mode. The benefits of cooked mode are that special keys are interpreted for you instead of by the application so you don't have to worry about erasing a character from the screen when someone hits BACKSPACE for instance. The effect of this is that the input must be buffered. Once the user presses ENTER the operating system shoots the buffer off to the application.

Hopefully that kind of answered #1...

2) There is 1 stream for stdout and 1 stream for stdin. All functions that use this stdout or stdin use the same stream. The stream utilizes an internal buffer when you're in cooked mode.

3) I'm not sure what you mean here, but if you do scanf("%d", &num); scanf("%d", num); then the first scanf() leaves the \n in the stream's buffer so the second scanf() just returns right away. fgets() pulls the the \n off the buffer. When someone says stream and buffer they generally mean the same thing. To confuse matters, programmers usually call the memory "buffer " that they pass to a function like fgets() (e.g. fgets(buffer, sizeof(buffer), stdin);).

4) fgets() searches through the stream's buffer to find the \n. Then it stores everything up to and including the \n into the buffer that you pass to it.

So if you're in cooked mode there's usually two buffers: one that you pass to functions for the function to store data in, and one used internally by the operating system.

regarding my question #3 , i ment where does it leave the '\n'? in the buffer or the stream?

and question #4. I ment how will it know about the '\n' when a preceding 'scanf()' leave it there?

itsme86 : sorry but i didn't understand your reply..
brain : you didn't really answer my questions , but thanks anyway

The buffer and the stream are the same thing. I think that's where you're getting confused.

fgets() looks through the buffer for the \n. I'm not sure what you don't understand about that. After a scanf() call the \n will be right at the front of the buffer usually.

>The stream utilizes an internal buffer when you're in cooked mode.
All of the information on cooked and raw input is incorrect in the context of C, sorry. Standard C input is either text or binary. For the purposes of stdin and stdout we'll assume there's a distinction and that they use text. Text streams are line based, that is, "an ordered sequence of characters composed into lines"[1]. Until a newline is encountered, the input function doesn't get any part of the input. That's why even if you are only using getchar to read a single character, you can still type as much as you want and nothing happens until you press return.

C implementations will typically use buffering for performance purposes. The point is to avoid accessing a device for input as much as possible because devices are slow. So a C implementation will call a system function (such as read) to read a large block of input and place that block in a buffer for quick access by the running program. All of this is still moot because C's text input is line based, so even though the implementation may buffer large portions of input, it still cannot legally pass it on to the program until a newline is reached. This is guaranteed by the standard and must be adhered to for any standard implementation.

The standard library gives you functions such as setbuf for manipulating this buffer, but keep in mind that it only affects performance. You can disable buffering for a stream, but you still cannot get a character without introducing a newline into the stream. It's important to understand what this buffer is and what it is used for; a lot of people have misconceptions about it.

>So if you're in cooked mode there's usually two buffers
How the operating system handles these things shouldn't matter unless you need precise control such as reading a character without pressing return. For the most part, any input in C can be described in terms of the stream, and buffers can be left out of the discussion except for internal memory used by your program (ie. the array you write the input to for later use).

>now my questions :
Now my answers:

>1. output functions buffer output before displaying it right? when and why?
The short answer is yes, always, and because writing to a device is very slow. The long answer is maybe, maybe never, and it's up to the implementation. A C implementation is not required to buffer input or output by default, though you can change this behavior programmatically by supplying your own buffer:

Code:

#include <stdio.h>

static char buffer[BUFSIZ];

int main ( void )
{
  setbuf ( stdout, buffer );

  /* The rest of your program */

  return 0;
}

>2. How is it possible that a function leaves some characters in the buffer? and do different functions\streams use the same buffer?
You'll be much happier if you think in terms of the stream and forget about buffers (they're just there for performance reasons anyway). A function will leave characters in the stream if it doesn't read them. For example:

Code:

#include <stdio.h>

int main ( void )
{
  int ch;

  printf ( "Enter a character: " );
  fflush ( stdout );
  
  while ( ( ch = getchar() ) != EOF )
    printf ( "%c\n", ch );

  return 0;
}

getchar only asks for a single character from the stream, so if you type "This is a test", getchar will give you 'T' and leave "his is a test" in the stream. It's a very simple queue setup: the first characters enqueued are the first dequeued and any that are not dequeued are left in the queue for later.

As for using the same buffer, assuming that an implementation uses a buffer, yes, every stream has its own buffer. If buffers were shared (as is possible, but very awkward) then input and output from different sources would get all jumbled up and that's just not fun.

>3. why does 'fgets()' terminate when 'scanf()' precede it , because 'scanf()' leaves '\n' in the stream or the buffer?
fgets is defined to read a line (ie. stopping at a newline, or when the limiting number of characters are read). If the first character in the stream is a newline then fgets will terminate successfully, having read a line. Since we're not thinking about buffers because it's confusing, scanf leaves a newline in the stream for fgets to terminate successfully on. It's simple as that.

>4. how do 'fgets()' know that the '\n' is in the buffer\stream?
Remember the queue. fgets will dequeue characters from the stream until it finds a newline or it dequeues from the stream N - 1 times. This can be done in C by repeatedly calling fgetc to dequeue a single character:

Code:

#include <stdio.h>

char *my_fgets ( char *s, int n, FILE *stream )
{
  int c = EOF;
  int i = 0;

  while ( --n > 0 ) {
    /* Dequeue from the stream */
    c = fgetc ( stream );

    if ( c == EOF )
      break;

    /* Save the character */
    if ( ( s[i++] = (char)c ) == '\n' )
      break;
  }

  if ( ( c == EOF && i == 0 ) || ferror ( stream ) )
    return NULL;

  s[i] = '\0';

  return s;
}

int main ( void )
{
  char string[BUFSIZ];

  printf ( "Enter a string: " );
  fflush ( stdout );
  
  if ( my_fgets ( string, sizeof string, stdin ) != NULL )
    fputs ( string, stdout );

  return 0;
}

>does it like search it before you have the chance to enter anything or what?
No, it waits for you to enter something if a terminating condition is not met.

[1] ISO/IEC 9899:1999 7.19.2

Geez, it doesn't seem like I even met my 90% expectation. Thanks for straightening us out, Prelude

ok i think i found what really confuses me.

When you say "stream" and talk about your own stream dealing with files , you mean the files (on HDD) that represent a "stream" of bits ...etc

But what do you mean by "stream" when you talk about 'stdin' and 'stdout'? like where does those "streams" reside? and how do characters get saved or left there? i just can't get a clear conception of the standard streams..


here are more Q's if you don't mind :

- When dealing with scanf() , does it ONLY leave the '\n' and nothing else?

- Why doesn't 'fgets()' terminates when another 'fgets()' precede it although 'fgets()' leaves a '\n' in the string? (sorry but i didn't understand your answer to Q#4 )

- Why are you flushing 'stdout' after each printf()?


sorry if i sound stupid but this new concept (for me) confuses me a bit.

basically input and output with devices is treated as you would a file. A stream is just a concept of a series of bytes that you send to or read from a device or file. You have functions that operate on this stream of information to extract the data you need. Depending on the funcitons specificiations different tasks can be accomplished. If you want a visualization try imaging a river with barges floating down to mouth. Each barge carries a cargo and each barge waits in line until some action is taken to remove it. If you created a canal to hold several of the barges you could consider that to be a buffer of sorts. fgetc() would take the first barge out of the stream and put it on a truck to be hauled away. it no longer waits in the stream fputc() would take that barge back off the truck and put it back into the stream. As far as buffering the operating system can create buffers . the default buffer is 4k (as far as i know) . Any file opened with stream routines is buffered by default. As far as the behavior of the functions you ask about.

Scanf() reads formatted input from stdin if it encounters data that doesn't match its format it terminates leaving that data in the stream as if it had never been read.

fgets() reads a string from stdin. basically with our analogy it takes however many barges you specify out of the stream and adds a null (an anchor if you will) to the end of the string.

fgets doesn't termintate because it will continue to read data from the stream until it gets to the end of the stream, a newline \n, or however many characters you specified -1.

sorry but , that doesn't clearly answer my questions...

Thanks for replying anyway man of steel

>But what do you mean by "stream" when you talk about 'stdin' and 'stdout'?
A stream is a stream. Under the hood you'll find that stdin and stdout are macros hiding a FILE *. It would look something like this:

Code:

extern FILE _io[FOPEN_MAX]; /* Details not important */

#define stdin  &_io[0]
#define stdout &_io[1]
#define stderr &_io[2]

When I say "stream" I mean however the FILE type implements a uniform handling of input and output from various devices. A stream isn't anything physical, it's a logical representation, a wrapper if you will, around the myriad ways that devices send and receive data.

Because the standard I/O streams are FILE *'s, you can often treat them just as you do files that you open yourself. The only difference is that the implementation opens and closes them for you. Oh, and seeking may or may not work, but that's beyond the scope of your question.

>like where does those "streams" reside?
The "streams" are basically just arrays that are filled with data and that data is doled out to your program by the standard I/O functions according to their defined behavior.

>and how do characters get saved or left there?
A low level system call is made to get data from the device or write data to the device depending on whether the stream is input or output. It's also possible to poll the device directly and control it manually, but that's not worth the effort when the operating system supports and API that does what you want.

>i just can't get a clear conception of the standard streams..
They're restricted files that happen to be attached with great regularity to devices such as the keyboard and the monitor. You can treat them like files, but whether some operations that work with files work with the standard streams is platform dependent and usually implementation dependent as well.

>When dealing with scanf() , does it ONLY leave the '\n' and nothing else?
It leaves whatever it cannot read whether through error, end-of-file, or design. If you tell scanf to read an integer and then type "1234abcd\n", "abcd\n" will be left in the stream.

>Why doesn't 'fgets()' terminates when another 'fgets()' precede it although 'fgets()' leaves a '\n' in the string?
fgets extracts a newline from the stream and writes it to the string. The next call to fgets doesn't terminate because the newline was already removed. Notice the difference between the stream being read from and the string being written to.

>Why are you flushing 'stdout' after each printf()?
Streams are flushed when their buffer is filled. Remember that a buffer is used to improve performance by reducing the number of system calls to read from or write to a device. In the case of output (or update, such as mode "r+" in fopen) streams, you can flush the stream by writing a newline to it or by filling the buffer. In some cases you may want to write an interactive prompt for input without writing a newline for formatting reasons. For example:

Code:

printf ( "Enter a number\n" );

Doesn't look as good to the user as:

Code:

printf ( "Enter a number: " );

The problem is that the buffer may not be filled when you place that message in it. Because there's no newline, you also cannot be sure that the stream will be flushed and it's possible for the pending request for input to be called before the prompt is written. That causes all kinds of trouble because the user doesn't know what to do yet the program is waiting for input.

The solution is to manually flush the stream with fflush:

Code:

printf ( "Enter a number: " );
fflush ( stdout );

Now you know for sure that the stream will always be flushed and the prompt will be written before any request for input is made. Windows users are often confused by this necessity because the problem doesn't exist AFAIK, but portable code must take it into account because other systems aren't as tolerant.

>sorry if i sound stupid but this new concept (for me) confuses me a bit.
Not at all, it's a subtle topic, and one that really isn't well understood until you've actually tried to implement I/O in C from scratch.

It leaves whatever it cannot read whether through error, end-of-file, or design. If you tell scanf to read an integer and then type "1234abcd\n", "abcd\n" will be left in the stream.

THATS the explanation i've been waiting for !!! i made this example to understand the concept and it went as expected :

Code:

#include <stdio.h>

int main()
{
 int x;
 char str[BUFSIZ];

 scanf("%d", &x);
 fgets(str , sizeof(str) , stdin);

 printf("%d , %s", x , str);

 return 0;
}

when you input '444abc\n' for scanf() it'll leave 'abc\n' in the buffer thus fgets() will read the rest untill it encountres '\n' and terminates ... sweet

BUT i noticed something wierd with another example. See this :

Code:

#include <stdio.h>

int main()
{
 char x[BUFSIZ];
 char str[BUFSIZ];

 scanf("%s", &x);
 fgets(str , sizeof(str), stdin);

 printf("%s , %s", x , str);

 system("PAUSE");

 return 0;
}

compile\run it and enter any text. fgets() now will display the same text again !? does it mean that scanf() left the whole string in stdin? it should not leave what it can read , i told it to read a string , so why did it leave the whole string in the stream?? is this an exception or what?

one last question , why doesn't scanf() care if there was a '\n' left in stdin or not? it works in both cases..

i didn't clearly understand the stdout part but thats fine. ill study and research\expirement more about it.

Ill go with your advise and forget about buffers. Now i imagine stdin a stream of bytes going from the keyboard to whatever function uses it. Functions like fgets() takes all the characters from the stream and save them to the memory under their chosen handles , and functions like scanf() might leave some characters in it that might affect other functions using the same stream ... right?


your help is much appreciated

Originally Posted by Brain Cell

BUT i noticed something wierd with another example. See this :

Code:

#include <stdio.h>

int main()
{
 char x[BUFSIZ];
 char str[BUFSIZ];

 scanf("%s", &x);
 fgets(str , sizeof(str), stdin);

 printf("%s , %s", x , str);

 system("PAUSE");

 return 0;
}

compile\run it and enter any text. fgets() now will display the same text again !? does it mean that scanf() left the whole string in stdin? it should not leave what it can read , i told it to read a string , so why did it leave the whole string in the stream?? is this an exception or what?

No it doesn't. You can clarify your output, by changing this line:

Code:

 printf("%s , %s", x , str);

To this:

Code:

printf("x is \"%s\", and str is \"%s\", x, str );

What's happening is:
1) You enter foo\n. Your input is echoed to the screen as you type it. This is the first instance of foo you see.
2) x contains color=sienna]foo[/color], so it is displayed. This is the second instance of foo you see.
3) This is followed by a space and a comma, and nothing else, because str contains nothing.

Also, since x is an array, you don't need the address of operator to read into it in scanf.

Quzah.

oh yea it doesn't leave anything but the '\n'. The compiler must have acted wierd by the time i experimented that

just one question left "why doesn't scanf() care if there was a '\n' left in stdin or not? it works in both cases.."

thanks for replying quzah

>why doesn't scanf() care if there was a '\n' left in stdin or not? it works in both cases..
Because scanf ignores whitespace unless you use the %c, %[...] or %n format flags.