pipe() read(), write() - more then 128Kb of data

Hello all,

Thanking in advance for the answers again!

I have a program reads from a file then pipes it to a Unix filter program, this last program adds lines to the original file, then prints it on the screen.

I tested my program with a 116K file, it worked, but then I tried a larger file (400+ Kb) and it halted. The program froze, didn't do anything, as if it's blocking.I found out that a 127-128 KB file will still work, but a larger file will not.

What is causing this?? Is it because I somehow have to manage the reads and writes?

My program so far:

Code:

/*      Chihwah Li
 *      elke textregel in een bestand voorzien van regelnummer.
 */

#include <stdio.h>
#include <sys/types.h> //
#include <unistd.h>
#include <stdlib.h>
#include <errno.h>
#include <fcntl.h>
#include <sys/wait.h>
#include <string.h>
#include <netinet/in.h>
#include <sys/time.h>

struct timeval start, finish ;
int msec;


int main(int argc, char** argv) {
    int X[2], Y[2];

    void Parent(int *X, int *Y, int fp);
    void Child(int *X, int *Y, int noLines);

    pipe(X);
    pipe(Y); /* Create Communication Channels */

    char *inFile;       //
    int fp, noLines;    //

    if (argc != 2 && argc != 3) {
        fprintf(stderr, "usage: %s [-n] <filename>\n", argv[0]);
        return (EXIT_FAILURE);
    }

    // dubbele check of de argumenten goed staan
    inFile = (argc == 2) ? argv[1] : argv[2]; 
    noLines = (argc == 3) ? 1 : 0; 
    if ((noLines == 1) && (strcmp(argv[1],"-n") < 0)) {
        fprintf(stderr,"usage: %s [-n] <filename>\n",argv[0]);
        return (EXIT_FAILURE);
    }
    if ((fp = open( inFile, O_RDONLY)) == -1) {
        perror("");
        return (EXIT_FAILURE);
    }
    
    gettimeofday (&start, NULL); // Tijdmeting starten

    switch (fork()) {
        case 0: Child(X, Y, noLines);
            break;
        case -1: perror("Can\'t create child");
            exit(1);
        default: Parent(X, Y, fp);
    }

    if (close(fp) == -1) {
        perror("close filename: ");
        return (EXIT_FAILURE);
    }

    exit(0);
}

void Parent(int *X, int *Y, int fp) {
    char c;
    close(X[0]);
    close(Y[1]); 

    while (read(fp, &c, 1) > 0){
        write(X[1], &c, 1); /* read data from stdin */
    }

    close(X[1]);

    while (read(Y[0], &c, 1) > 0){
        write(1, &c, 1); /* write data to stdout */
    }
    
    
    // checken y en x of dat juist is
    gettimeofday (&finish, NULL);
    msec = finish.tv_sec * 1000 + finish.tv_usec / 1000;
    msec -= start.tv_sec * 1000 + start.tv_usec / 1000;

    fprintf(stderr, "Time: %d milliseconds\nPress [Enter] to see the uniq, sorted list: ", msec);

    
    
    close(Y[0]);

    wait(0);
}

void Child(int *X, int *Y, int noLines) {
    close(X[1]);
    close(Y[0]);
    close(0);
    dup(X[0]);
    close(X[0]); 
    close(1);
    dup(Y[1]);
    close(Y[1]); 
    if (noLines == 0){
        execlp("nl", "nl", NULL);                       // add line numbers
    } else {
        execlp("sed", "sed","'N;s/\n/\t/'", NULL);    // remove line numbers
    }

    perror("fault by exec: ");
    exit(1);
}

The problem is that, in Child(), you are writing a lot of data to Y[1], but you are not attempting to read from the other end of the pipe (in Parent()) until after the parent has sent all data. However, pipes have a capacity, and the output from Child() is filling up the pipe to its capacity. Once it hits capacity, write() in the child will block. Parent(), however, is still trying to write() to a pipe owned by the child. Since the child is blocking on its write, it will not be able to issue a read() on that pipe, causing Parent() to hit the capacity and block on its write(). You wind up with both parent and child blocking in their respective write() calls because the other side is not issuing any read()s.

Thank you Cas, That means I have to implement a read function to my child process. Then pass the data to execlp.

I was wondering if this is a possible way to solve it:

- set pipes to non_block
Reason: Parent can writes and the child read at the same time

- Child stores the read data into an array

- Exec() reads the variable as an argument. (normally it would look for the file on the harddisk.)

On top of this: I have to open my files in Binary mode. Reason: Portable code. Linux and windows use different END OF FILE characters. This will cause a read to stop as it ancounters a EOF character. In Binary it would just treat it as a normal character.

Would like to hear your comments and tips.