parallel download

[Erik Anderson]

Hi
I need a little help with a parallel download. My code is working as serial, I cannot find the way to fork as parallel...please help...Thank you.

while (!feof (file)) {

memset (line,'\0',1000);

char *urlPtr = fgets (line,LINE_MAX, file);

++numberOfChildren;

int lineLen = strlen(urlPtr);
urlPtr[lineLen-1] = '\0';
pid = fork();

    if (pid == 0) {  /* child process */
        execlp("/usr/bin/wget", "wget", urlPtr, NULL);
        }
....

[christop]

What is the actual problem, and what does your code currently do wrong? Does it crash? I see a few issues with the code that may cause crashes (such as using feof as a loop condition and not checking the return value of fgets).

Frankly I wouldn't use C for a simple task like this. A one-line shell script would be more appropriate, IMHO:

while read url; do wget "$url" & done <file

(Yes, I know this is a C forum, but unless you have to do this in C, it's a good idea to look at other languages that make the task easy.)

[Adak]

Isn't wget just a system script, anyway?

[christop]

Isn't wget just a system script, anyway?

It's a binary on my system and every other system I've used. I think it's written in C.

[Erik Anderson]

I have to do it in C

[cfanatic]

Here is the faq about why it's bad to use feof() in while() loop:

FAQ > Why it's bad to use feof() to control a loop - Cprogramming.com

[Erik Anderson]

did some changes, but still not working as parallel.

#include <sys/wait.h>
#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <string.h>
FILE *file; /*declare the file pointer*/
#define LINE_MAX 1000
char line [LINE_MAX];
//Parent process
int main()
{
pid_t  pid;
file= fopen ("urls.txt", "rt"); /*open file and read it*/
int numberOfChildren = 0;

while (!feof (file)) {

memset (line,'\0',1000);

char *urlPtr; 
while (fgets (line,sizeof(line),file)!=0 {
urlPtr= fgets (line,LINE_MAX, file);
int lineLen = strlen(urlPtr);
urlPtr[lineLen-1] = '\0';
pid = fork();

++numberOfChildren;


    if (pid == 0) {  /* child process */
        execlp("/usr/bin/wget", "wget", urlPtr, NULL);
        }

    else if (pid < 0) { /* error occurred */
        fprintf(stderr, "Fork Failed");
        exit(-1);
        }

    while (numberOfChildren>0) { /* parent process */
    /* parent will wait for the child to complete */
        wait (NULL);
        --numberOfChildren;
        printf ("Child Complete");
        }
}
}    
fclose (file); /*close file command*/
return 0;
}

[christop]

Your program waits for each child process to exit before starting another child, so it won't start more than one at a time.