Like Tree1Likes
  • 1 Post By christop

parallel download

This is a discussion on parallel download within the C Programming forums, part of the General Programming Boards category; Hi I need a little help with a parallel download. My code is working as serial, I cannot find the ...

  1. #1
    Registered User
    Join Date
    Sep 2012
    Posts
    4

    parallel download

    Hi
    I need a little help with a parallel download. My code is working as serial, I cannot find the way to fork as parallel...please help...Thank you.

    Code:
    while (!feof (file)) {
    
    memset (line,'\0',1000);
    
    char *urlPtr = fgets (line,LINE_MAX, file);
    
    ++numberOfChildren;
    
    int lineLen = strlen(urlPtr);
    urlPtr[lineLen-1] = '\0';
    pid = fork();
    
        if (pid == 0) {  /* child process */
            execlp("/usr/bin/wget", "wget", urlPtr, NULL);
            }
    ....

  2. #2
    Registered User
    Join Date
    May 2012
    Location
    Arizona, USA
    Posts
    383
    What is the actual problem, and what does your code currently do wrong? Does it crash? I see a few issues with the code that may cause crashes (such as using feof as a loop condition and not checking the return value of fgets).

    Frankly I wouldn't use C for a simple task like this. A one-line shell script would be more appropriate, IMHO:
    Code:
    while read url; do wget "$url" & done <file
    (Yes, I know this is a C forum, but unless you have to do this in C, it's a good idea to look at other languages that make the task easy.)

  3. #3
    Registered User
    Join Date
    Sep 2006
    Posts
    8,868
    Isn't wget just a system script, anyway?

  4. #4
    Registered User
    Join Date
    May 2012
    Location
    Arizona, USA
    Posts
    383
    Quote Originally Posted by Adak View Post
    Isn't wget just a system script, anyway?
    It's a binary on my system and every other system I've used. I think it's written in C.
    Adak likes this.

  5. #5
    Registered User
    Join Date
    Sep 2012
    Posts
    4
    I have to do it in C

  6. #6
    Registered User
    Join Date
    Jul 2012
    Location
    Australia
    Posts
    242
    Here is the faq about why it's bad to use feof() in while() loop:

    FAQ > Why it's bad to use feof() to control a loop - Cprogramming.com
    IDE: Code::Blocks | Compiler Suite for Windows: TDM-GCC (MingW, gdb)

  7. #7
    Registered User
    Join Date
    Sep 2012
    Posts
    4
    did some changes, but still not working as parallel.
    Code:
    #include <sys/wait.h>
    #include <stdio.h>
    #include <unistd.h>
    #include <stdlib.h>
    #include <string.h>
    FILE *file; /*declare the file pointer*/
    #define LINE_MAX 1000
    char line [LINE_MAX];
    //Parent process
    int main()
    {
    pid_t  pid;
    file= fopen ("urls.txt", "rt"); /*open file and read it*/
    int numberOfChildren = 0;
    
    while (!feof (file)) {
    
    memset (line,'\0',1000);
    
    char *urlPtr; 
    while (fgets (line,sizeof(line),file)!=0 {
    urlPtr= fgets (line,LINE_MAX, file);
    int lineLen = strlen(urlPtr);
    urlPtr[lineLen-1] = '\0';
    pid = fork();
    
    ++numberOfChildren;
    
    
        if (pid == 0) {  /* child process */
            execlp("/usr/bin/wget", "wget", urlPtr, NULL);
            }
    
        else if (pid < 0) { /* error occurred */
            fprintf(stderr, "Fork Failed");
            exit(-1);
            }
    
        while (numberOfChildren>0) { /* parent process */
        /* parent will wait for the child to complete */
            wait (NULL);
            --numberOfChildren;
            printf ("Child Complete");
            }
    }
    }    
    fclose (file); /*close file command*/
    return 0;
    }

  8. #8
    Registered User
    Join Date
    May 2012
    Location
    Arizona, USA
    Posts
    383
    Your program waits for each child process to exit before starting another child, so it won't start more than one at a time.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. parallel port
    By kmp in forum C++ Programming
    Replies: 0
    Last Post: 05-08-2011, 11:06 PM
  2. Parallel Port in XP
    By torqu3e in forum C++ Programming
    Replies: 6
    Last Post: 01-30-2011, 08:33 AM
  3. Serial-Parallel
    By Unregistered in forum A Brief History of Cprogramming.com
    Replies: 2
    Last Post: 04-19-2002, 04:38 AM
  4. Help with Parallel Port
    By Unregistered in forum C++ Programming
    Replies: 3
    Last Post: 04-16-2002, 10:20 PM

Tags for this Thread


1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21