When is a pipe writer done?

This is a discussion on When is a pipe writer done? within the Linux Programming forums, part of the Platform Specific Boards category; I'm trying to figure out the best way for the writer on a pipe to let the reader know that ...

  1. #1
    Registered User
    Join Date
    Aug 2010
    Posts
    1

    When is a pipe writer done?

    I'm trying to figure out the best way for the writer on a pipe to let the reader know that its done writing.

    Background:
    I have a process running that reads commands on STDIN, interprets the commands, and writes results to STDOUT. This process never exits. Furthermore, the process could take hours to execute a command. Also, the process may send zero or more lines to STDOUT when it's finished with a command. I want to send a command to this process, wait for a response, and then send another command, wait for a response, and so on. The problem is I don't know how long to wait for a response. I can use select() with a timeout value, but if I give it a timeout value that's long enough to accommodate a long-running command, select() will waste time waiting on a short-running command.

    Here are some ways I've thought of to solve the problem:
    1) Look for a key in the result that got written to STDOUT which indicates that the process is finished writing. Problem: the result contains arbitrary data. If the result happens to contain the key in its data, I would have to escape it somehow.

    2) Send the number of bytes that will be transmitted over STDOUT before sending the result. Problem: The process I'm talking about is an existing program (it's been around since 1984, actually) and customers don't want to see byte counts in the standard output stream.

    3) Send SIGUSR1 or SIGUSR2 when the process is done writing a result to STDOUT. Problem: If the process that invoked (i.e. a shell) the program in question doesn't handle SIGUSR1 or SIGUSR2, it'll get killed.

    4) Use Sun RPC instead of STDOUT to send results. This could work, though I don't really like working with Sun RPC (it's difficult to integrate it into an existing C++ app).

    5) Use a C++ XML-RPC implementation for IPC. Problem: While this is easier than Sun RPC, our legal department has to qualify any open source shared library we use (it takes months for them to do this). In addition, they have to re-qualify it for every new version we use (another several months).

    6) Make a flow control socket that sends a message when a result is finished. So far this is the best thing I can think of.

    7) Use some other for of IPC such as queues, shared memory, etc. I haven't explored this yet.

    Do you guys have any advice for the best way to solve my problem?

    Sometimes problems are easier to explain in code. Here's an example program that illustrates what I'm dealing with. There's a child that writes a random number of messages (with random delays in between them) to a pipe, and the parent somehow must determine when the child is done writing. Presently, it fails miserably because the select() timeout is too short.

    Code:
    #include <iostream>
    #include <string>
    #include <cstdlib>
    #include <cassert>
    #include <sys/wait.h>
    
    
    // The maximum number of seconds between the child's messages
    #define MAX_WAIT 2
    
    using namespace std;
    
    // Easy way to get a random number
    unsigned int rand_num(unsigned int limit) {
       assert(limit);
       time_t t = time(NULL);
       unsigned int seed = t;
       unsigned int div = RAND_MAX / limit;
       return (rand_r(&seed) / (div == 0 ? 1 : div));
    }
    
    int main() {
       int pfds[2];
       pipe(pfds);
    
       // Fork off the child
       pid_t pid = fork();
       if (pid == 0)
       {
          // Setup the pipe
          dup2(pfds[1], STDOUT_FILENO);
    
          // The child prints a random number of messages with a random wait period between messages
          cerr << "Child starting to send messages" << endl;
          for (unsigned int i = 0; i < rand_num(8); ++i) {
             sleep(rand_num(8)); // This is sometimes larger than MAX_WAIT!!
             cout << "Here's line " << i << endl;
             cerr << "The child sent line " << i << endl;
          }
          cerr << "Child stopped sending messages" << endl;
    
          // Wait for another 10 seconds to simulate a long-running server process
          sleep(10);
          exit(0);
       }
    
       // Read the output
       char buf[2048];
       struct timeval tv;
       fd_set read_set;
       ssize_t bytes_read;
       int retval;
       do
       {
          tv.tv_sec = MAX_WAIT;
          tv.tv_usec = 0;
    
          FD_ZERO(&read_set);
          FD_SET(pfds[0], &read_set);
    
          retval = select(pfds[0] + 1, &read_set, NULL, NULL, &tv);
          if (retval <= 0)
             break;
    
          if (FD_ISSET(pfds[0], &read_set))
          {
             // Read and print what the child sent
             bytes_read = read(pfds[0], buf, sizeof(buf) - 1);
             buf[bytes_read] = '\0';
             cerr << buf;
          }
       } while (retval > 0);
    
       // Wait for the child to exit
       waitpid(pid, NULL, 0);
    
       return 0;
    }

  2. #2
    and the hat of wrongness Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    32,767
    > Also, the process may send zero or more lines to STDOUT when it's finished
    So how would you know that it has finished, if there is no output.

    Is the child doing anything that can be observed, say
    - reading and writing files (have the parent periodically examine the output of the 'lsof' command)
    - using large amounts of CPU time (have the parent periodically examine the output of the 'top' command)

    > Problem: The process I'm talking about is an existing program (it's been around since 1984, actually) and
    > customers don't want to see byte counts in the standard output stream.
    Well you could always add a "--batch" option which makes the program a bit more verbose (in a usefully predictable way) for when you're running it in the context that you're running it.
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.
    I support http://www.ukip.org/ as the first necessary step to a free Europe.

  3. #3
    Registered User
    Join Date
    Oct 2008
    Location
    TX
    Posts
    2,050
    Quote Originally Posted by n8_sl8er View Post
    I'm trying to figure out the best way for the writer on a pipe to let the reader know that its done writing.
    ...
    I can use select() with a timeout value, but if I give it a timeout value that's long enough to accommodate a long-running command, select() will waste time waiting on a short-running command.
    Why would select() waste time for a shorter process ie it won't wait until the timeout expires but returns as soon as the data is available.
    Quote Originally Posted by n8_sl8er View Post
    Sometimes problems are easier to explain in code. Here's an example program that illustrates what I'm dealing with. There's a child that writes a random number of messages (with random delays in between them) to a pipe, and the parent somehow must determine when the child is done writing. Presently, it fails miserably because the select() timeout is too short.
    The best bet is to use read() in its default mode ie with the O_NONBLOCK'ing flag cleared.
    This way the reader waits for as long as the data doesn't show up on the pipe for reading.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Replies: 4
    Last Post: 10-14-2009, 05:44 PM
  2. cont of IPC using PIPE
    By BMathis in forum C Programming
    Replies: 1
    Last Post: 03-15-2009, 06:16 PM
  3. Pipe class.
    By eXeCuTeR in forum Linux Programming
    Replies: 8
    Last Post: 08-21-2008, 04:44 AM
  4. Pipe(): Interprocess or Intraprocess comm?
    By @nthony in forum C Programming
    Replies: 2
    Last Post: 03-28-2007, 08:27 PM
  5. Named Pipe Problems.
    By Mastadex in forum Windows Programming
    Replies: 2
    Last Post: 06-16-2006, 09:35 AM

Tags for this Thread


1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21