pipe() read(), write() - more then 128Kb of data

This is a discussion on pipe() read(), write() - more then 128Kb of data within the C Programming forums, part of the General Programming Boards category; Hello all, Thanking in advance for the answers again! I have a program reads from a file then pipes it ...

  1. #1
    Registered User
    Join Date
    Sep 2011
    Posts
    16

    pipe() read(), write() - more then 128Kb of data

    Hello all,

    Thanking in advance for the answers again!

    I have a program reads from a file then pipes it to a Unix filter program, this last program adds lines to the original file, then prints it on the screen.

    I tested my program with a 116K file, it worked, but then I tried a larger file (400+ Kb) and it halted. The program froze, didn't do anything, as if it's blocking.I found out that a 127-128 KB file will still work, but a larger file will not.

    What is causing this?? Is it because I somehow have to manage the reads and writes?

    My program so far:

    Code:
    
    /*      Chihwah Li
     *      elke textregel in een bestand voorzien van regelnummer.
     */
    
    #include <stdio.h>
    #include <sys/types.h> //
    #include <unistd.h>
    #include <stdlib.h>
    #include <errno.h>
    #include <fcntl.h>
    #include <sys/wait.h>
    #include <string.h>
    #include <netinet/in.h>
    #include <sys/time.h>
    
    struct timeval start, finish ;
    int msec;
    
    
    int main(int argc, char** argv) {
        int X[2], Y[2];
    
        void Parent(int *X, int *Y, int fp);
        void Child(int *X, int *Y, int noLines);
    
        pipe(X);
        pipe(Y); /* Create Communication Channels */
    
        char *inFile;       //
        int fp, noLines;    //
    
        if (argc != 2 && argc != 3) {
            fprintf(stderr, "usage: %s [-n] <filename>\n", argv[0]);
            return (EXIT_FAILURE);
        }
    
        // dubbele check of de argumenten goed staan
        inFile = (argc == 2) ? argv[1] : argv[2]; 
        noLines = (argc == 3) ? 1 : 0; 
        if ((noLines == 1) && (strcmp(argv[1],"-n") < 0)) {
            fprintf(stderr,"usage: %s [-n] <filename>\n",argv[0]);
            return (EXIT_FAILURE);
        }
        if ((fp = open( inFile, O_RDONLY)) == -1) {
            perror("");
            return (EXIT_FAILURE);
        }
        
        gettimeofday (&start, NULL); // Tijdmeting starten
    
        switch (fork()) {
            case 0: Child(X, Y, noLines);
                break;
            case -1: perror("Can\'t create child");
                exit(1);
            default: Parent(X, Y, fp);
        }
    
        if (close(fp) == -1) {
            perror("close filename: ");
            return (EXIT_FAILURE);
        }
    
        exit(0);
    }
    
    void Parent(int *X, int *Y, int fp) {
        char c;
        close(X[0]);
        close(Y[1]); 
    
        while (read(fp, &c, 1) > 0){
            write(X[1], &c, 1); /* read data from stdin */
        }
    
        close(X[1]);
    
        while (read(Y[0], &c, 1) > 0){
            write(1, &c, 1); /* write data to stdout */
        }
        
        
        // checken y en x of dat juist is
        gettimeofday (&finish, NULL);
        msec = finish.tv_sec * 1000 + finish.tv_usec / 1000;
        msec -= start.tv_sec * 1000 + start.tv_usec / 1000;
    
        fprintf(stderr, "Time: %d milliseconds\nPress [Enter] to see the uniq, sorted list: ", msec);
    
        
        
        close(Y[0]);
    
        wait(0);
    }
    
    void Child(int *X, int *Y, int noLines) {
        close(X[1]);
        close(Y[0]);
        close(0);
        dup(X[0]);
        close(X[0]); 
        close(1);
        dup(Y[1]);
        close(Y[1]); 
        if (noLines == 0){
            execlp("nl", "nl", NULL);                       // add line numbers
        } else {
            execlp("sed", "sed","'N;s/\n/\t/'", NULL);    // remove line numbers
        }
    
        perror("fault by exec: ");
        exit(1);
    }

  2. #2
    cas
    cas is offline
    Registered User
    Join Date
    Sep 2007
    Posts
    983
    The problem is that, in Child(), you are writing a lot of data to Y[1], but you are not attempting to read from the other end of the pipe (in Parent()) until after the parent has sent all data. However, pipes have a capacity, and the output from Child() is filling up the pipe to its capacity. Once it hits capacity, write() in the child will block. Parent(), however, is still trying to write() to a pipe owned by the child. Since the child is blocking on its write, it will not be able to issue a read() on that pipe, causing Parent() to hit the capacity and block on its write(). You wind up with both parent and child blocking in their respective write() calls because the other side is not issuing any read()s.

  3. #3
    Registered User
    Join Date
    Sep 2011
    Posts
    16
    Thank you Cas, That means I have to implement a read function to my child process. Then pass the data to execlp.
    Last edited by chihwahli; 10-31-2011 at 06:19 AM.

  4. #4
    Registered User
    Join Date
    Sep 2011
    Posts
    16
    I was wondering if this is a possible way to solve it:

    - set pipes to non_block
    Reason: Parent can writes and the child read at the same time

    - Child stores the read data into an array

    - Exec() reads the variable as an argument. (normally it would look for the file on the harddisk.)

    On top of this: I have to open my files in Binary mode. Reason: Portable code. Linux and windows use different END OF FILE characters. This will cause a read to stop as it ancounters a EOF character. In Binary it would just treat it as a normal character.

    Would like to hear your comments and tips.
    Last edited by chihwahli; 11-01-2011 at 04:45 AM.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Replies: 1
    Last Post: 04-14-2011, 10:43 AM
  2. Replies: 7
    Last Post: 06-03-2010, 07:21 PM
  3. Replies: 2
    Last Post: 12-06-2008, 01:17 PM
  4. write() with pipe()
    By und3rdog in forum C Programming
    Replies: 11
    Last Post: 05-03-2003, 10:57 PM
  5. pipe stream data I/O
    By vector7 in forum Linux Programming
    Replies: 1
    Last Post: 03-11-2003, 05:46 PM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21