Thread: Linux file redirection buffering issue !!!!

  1. #1
    Registered User
    Join Date
    Mar 2009
    Posts
    3

    Unhappy Linux file redirection buffering issue !!!!

    Hello,
    I am facing a problem since several days that i can't solve by myself.
    Let me explain you my problem.
    I am working on a TVbox configured with a small linux kernel.
    The default GUI of this box is not efficient, so I will to improve it.
    I found the binary launched ('dmaosd') on the box to handle the GUI.
    I have developped my own code in C and php to make some tuning of the GUI.
    To go further I have to parse some information coming from the running 'dmaosd' binary.
    When i lauch the 'dmaosd' binary in a telnet console, it shows its output in a line buffered mode to the console screen.
    So I have just make a bash redirection in a log file ('dmaosd > /tmp/GUI.log') to collect all its output in a file in order to be able to parse it with my code.
    But the pb I am facing is that the redirection make the 'dmaosd' to fill its output in a block buffer mode in the log file, and not anymore in a line buffered mode.

    And i try to find a way to make the kernel to have the same bahaviour in term of buffering between console outputs and log file redirection, but unsucessfully.

    I have test several things with the bash redirection, making some C codes to launch the 'dmaosd' through my own C code to force flush and no buffer settings, I have tried to find in the kernel settings where I could change its default buffer behaviours for file IO, but I don't fins someting till now to solve my pb.

    is there anybody that could help me on this issue ???

    thanks a lot per advance .

    AirSeb

  2. #2
    Kernel hacker
    Join Date
    Jul 2007
    Location
    Farncombe, Surrey, England
    Posts
    15,677
    If the code is written in C (as opposed to C++), you could probably use sertvbuf(stdout, buffer, _IONBF, size);
    where buffer is a global or malloc'd array of, say, 80 characters.

    _IONBF will tell it to be non-buffered, so it will output data more or less directly.

    This is how stderr is normally done.

    --
    Mats
    Compilers can produce warnings - make the compiler programmers happy: Use them!
    Please don't PM me for help - and no, I don't do help over instant messengers.

  3. #3
    Registered User
    Join Date
    Mar 2009
    Posts
    3

    Unhappy Linux buffering !

    Yes I have made tests with my own C code to setup this information, But I think it has an impact only for my code and not for the behaviour of the 'dmaosd' binary. for this one I can't control its code.

    for example with my code :
    Code:
    # include <stdio.h>
    
    int main () {
     char c;
     FILE *file;
    
     file=popen("run_all","r");
     setvbuf(file,NULL,_IONBF,0);
     if (file==NULL){
      printf("Error: can t open file.\n");
      return 1;
     }
     else {
      printf("[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[file opened sucessfully. Contents: \n\n");
    
      while (1) { /* ---------------*/
       fflush(file);
       c=fgetc(file);
       if (c!=EOF){
        printf("%c",c);
        if (c=='\n') {
         printf("[[[[[");
        }
       }
       else {
       break;
       }
     }/* -------------------------*/
    
    printf("\n\n Now closing file ...\n");
    fclose(file);
    return 0;
    }
    }
    as you can see I used setvbuf and a fflush, but it doesn't change the behaviour of the external 'dmaosd' binary, that delivers its output to my C code with its ............ buffer.
    can you help ?

    thanks

    AirSeb

  4. #4
    Kernel hacker
    Join Date
    Jul 2007
    Location
    Farncombe, Surrey, England
    Posts
    15,677
    Ah, sorry, I didn't realize that dmaosd is not your own source - if that's the case, there isn't much you can actually do about it - if dmaosd is distributed as source, you could just add a bit of code to do that in the start if main() [or some such].

    --
    Mats
    Compilers can produce warnings - make the compiler programmers happy: Use them!
    Please don't PM me for help - and no, I don't do help over instant messengers.

  5. #5
    Registered User
    Join Date
    Mar 2009
    Posts
    3

    Unhappy linux buffering

    so you think I don't have to loose time to find a solution for this issue.
    because I can't change anything in the 'dmaosd' binary.
    So my last seraches were in the linux kernel parameters to find a parameter that I can change to modify its default bahaviour with its block buffering for file writing ...
    but i don't find anything till now.

    AirSeb

  6. #6
    Kernel hacker
    Join Date
    Jul 2007
    Location
    Farncombe, Surrey, England
    Posts
    15,677
    Quote Originally Posted by AirSeb View Post
    so you think I don't have to loose time to find a solution for this issue.
    because I can't change anything in the 'dmaosd' binary.
    So my last seraches were in the linux kernel parameters to find a parameter that I can change to modify its default bahaviour with its block buffering for file writing ...
    but i don't find anything till now.

    AirSeb
    Linux itself doesn't affect this - the buffering is done in the C library (libc.so). What you possibly could do is change the C library functionality so that it does something along the lines I described (change how stdout is created to match stderr, for example).

    --
    Mats
    Compilers can produce warnings - make the compiler programmers happy: Use them!
    Please don't PM me for help - and no, I don't do help over instant messengers.

  7. #7
    Registered User
    Join Date
    Oct 2007
    Posts
    32
    Quote Originally Posted by AirSeb View Post
    so you think I don't have to loose time to find a solution for this issue.
    because I can't change anything in the 'dmaosd' binary.
    So my last seraches were in the linux kernel parameters to find a parameter that I can change to modify its default bahaviour with its block buffering for file writing ...
    but i don't find anything till now.

    AirSeb
    I think you have two options to achieve unbuffered outout from unchanged dmaosd binary:
    1) Set unbuffered stdout as was suggested above in your executable and then execve dmaosd.
    2) Run dmaosd with LD_PRELOADed library, that will set unbuffered stdout

    Valery

  8. #8
    Officially An Architect brewbuck's Avatar
    Join Date
    Mar 2007
    Location
    Portland, OR
    Posts
    7,396
    Quote Originally Posted by Valery Reznic View Post
    I think you have two options to achieve unbuffered outout from unchanged dmaosd binary:
    1) Set unbuffered stdout as was suggested above in your executable and then execve dmaosd.
    2) Run dmaosd with LD_PRELOADed library, that will set unbuffered stdout

    Valery
    Preloading might work. If it doesn't, the only other method I can think of (that Valery didn't already mention) would be direct code injection into the dmaosd process. Injection on Linux is nowhere near as simple as on Windows, but it's possible.

    To try a preload, maybe something like this:

    Code:
    /* setbuf.c */
    void _init()
    {
        setvbuf(stdout, NULL, _IOLBF, 0);
    }
    
    $ gcc -fPIC -c setbuf.c
    $ ld -shared -o setbuf.so setbuf.o -lc
    $ LD_PRELOAD=./setbuf.so dmaosd
    I haven't fully tested this though. I did place an "int 3" breakpoint in _init() and verified that the code does execute during load. But who knows whether the C library just sets it back again later on... Try it and see.

    EDIT: It's also possible that dmaosd itself is deliberately setting full buffering if it detects that its stdout is not a terminal device. There are many programs that do that. Not a very smart practice for a logging mechanism though... If that's the case, you could write a pseudo-tty forwarding utility that looks like a terminal, tricking dmaosd into doing line buffering.
    Last edited by brewbuck; 03-14-2009 at 12:27 PM.
    Code:
    //try
    //{
    	if (a) do { f( b); } while(1);
    	else   do { f(!b); } while(1);
    //}

  9. #9
    Kernel hacker
    Join Date
    Jul 2007
    Location
    Farncombe, Surrey, England
    Posts
    15,677
    Would execve not re-initialize the C file-pointer internal structure for stdout - in which case, execve will not work. I'm 99% sure that stdout is re-initialized if you start a new program through for exec*().

    --
    Mats
    Compilers can produce warnings - make the compiler programmers happy: Use them!
    Please don't PM me for help - and no, I don't do help over instant messengers.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. File transfer- the file sometimes not full transferred
    By shu_fei86 in forum C# Programming
    Replies: 13
    Last Post: 03-13-2009, 12:44 PM
  2. Formatting a text file...
    By dagorsul in forum C Programming
    Replies: 12
    Last Post: 05-02-2008, 03:53 AM
  3. gcc link external library
    By spank in forum C Programming
    Replies: 6
    Last Post: 08-08-2007, 03:44 PM
  4. help with text input
    By Alphawaves in forum C Programming
    Replies: 8
    Last Post: 04-08-2007, 04:54 PM
  5. simulate Grep command in Unix using C
    By laxmi in forum C Programming
    Replies: 6
    Last Post: 05-10-2002, 04:10 PM