Producer/Consumer - problems creating threads

This is a discussion on Producer/Consumer - problems creating threads within the C Programming forums, part of the General Programming Boards category; Originally Posted by tabstop Why not check whether the fopen succeeds? I probably should, but the thing is, if I ...

  1. #16
    Registered User
    Join Date
    May 2009
    Posts
    43
    Quote Originally Posted by tabstop View Post
    Why not check whether the fopen succeeds?
    I probably should, but the thing is, if I take out the while loops from the producer/consumer functions, AND I just use buffer[0] as my only slot... And change the pthread creating to what I had originally (20 producers and 20 consumers with joins), then I get the output I expect:
    IN PRODUCER
    aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
    IN CONSUMER
    AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
    IN PRODUCER
    bbbbbbbb
    IN CONSUMER
    BBBBBBBB
    IN PRODUCER
    ccccccc
    IN CONSUMER
    CCCCCCC
    IN PRODUCER
    ddddddd
    IN CONSUMER
    DDDDDDD
    IN PRODUCER
    eeeeeee
    IN CONSUMER
    EEEEEEE
    IN PRODUCER
    ffffffffffffffffffffffffffffffffffffffffffffffffff ffffffffffffffffffffffffffffffffffffffffffffffffff ffffffffffffffffffffffffffffffffffffffffffffffffff ffffffffffffffffffffffffffffffffffffffffffffffffff ffffffffffffffffffffffffffffffffffffffffffffffffff ffffffffffffffffffffffffffffffffffffffffffffffffff ffffffffffffffff
    IN CONSUMER
    FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF FFFFFFFFFFFFFFFFFFFFFFFFFFFFIN PRODUCER
    ggggggg
    IN CONSUMER
    GGGGGGG
    IN PRODUCER
    hhhhhhh
    IN CONSUMER
    HHHHHHH
    IN PRODUCER
    iiiiiii
    IN CONSUMER
    IIIIIII
    IN PRODUCER
    jjjjjjj
    IN CONSUMER
    JJJJJJJ
    IN PRODUCER
    kkkkkk
    IN CONSUMER
    KKKKKK
    IN PRODUCER
    llllllll
    IN CONSUMER
    LLLLLLLL
    IN PRODUCER
    mmmmmmm
    IN CONSUMER
    MMMMMMM
    IN PRODUCER
    nnnnnnn
    IN CONSUMER
    NNNNNNN
    IN PRODUCER
    ooooooo
    IN CONSUMER
    OOOOOOO
    IN PRODUCER
    ppppppp
    IN CONSUMER
    PPPPPPP
    IN PRODUCER
    qqqqqq
    IN CONSUMER
    QQQQQQ
    IN PRODUCER
    rrrrrrr
    IN CONSUMER
    RRRRRRR
    IN PRODUCER
    sssssssss
    IN CONSUMER
    SSSSSSSSS
    IN PRODUCER
    tttttt
    IN CONSUMER
    TTTTTT


    Here's the edited code that produces the above output:
    Code:
    #include<stdio.h>
    #include<stdlib.h>
    #include<ctype.h>
    #include<unistd.h>
    #include<pthread.h>
    #include<semaphore.h>
    
    #define FILESIZE 19
    #define BUFFER_SIZE 5
    #define PRODUCERS 20
    #define CONSUMERS 20
    
    struct fifo_struct
    {
            char fileName[1024];
            struct fifo_struct* next;
    };
    
    struct linked_list
    {
            struct fifo_struct* head;
            struct fifo_struct* tail;
    };
    
    pthread_mutex_t mutex;
    sem_t full, empty;
    char buffer[BUFFER_SIZE][128];
    int counter;
    
    void print_queue(const struct linked_list* ps)
    {
            struct fifo_struct* p = NULL;
    
            if(ps)
            {
                    for(p = ps->head; p; p = p->next)
                    {
                            if(p)
                                    printf("int = %s\n", p->fileName);
                            else
                                    printf("can't print NULL STRUCT\n");
                    }
            }
    }
    void *producer(void *q);
    void *consumer(void *q);
    
    int main()
    {
            pthread_t producerVar;//[PRODUCERS];
            pthread_t consumerVar;//[CONSUMERS];
            char line[1024];
            char str[200];
            int i = 0;
            counter = 0;
            sem_init(&full, 0, 0);
            sem_init(&empty, 0, BUFFER_SIZE);
            pthread_mutex_init(&mutex,NULL);
            struct fifo_struct* fifo;
    
    
            struct linked_list* s = malloc( 1 * sizeof(*s));
            if(s == NULL)
            {
                    fprintf(stderr, "LINE: %d, malloc() failed\n", __LINE__);
            }
            s->head = s->tail = NULL;
    
            for(i = 0; i < (FILESIZE + 1); i++)
            {
                    sprintf(str, "in%d.txt", i);
    
                    fifo = malloc(1 * sizeof(*fifo));
                    if(fifo == NULL)
                    {
                            fprintf(stderr, "IN %s, %s: malloc() failed\n", __FILE__, "list_add");
                    }
                    strcpy(fifo->fileName,str);
                    fifo->next = NULL;
    
                    if(s == NULL)
                    {
                            printf("Error: Queue has not been initialized\n");
                    }
                    else if(s->head == NULL && s->tail == NULL)
                    {
                            s->head = s->tail = fifo;
                    }
                    else if(s->head == NULL || s->tail == NULL)
                    {
                            printf("Error: Problem with code\n");
                            free(fifo);
                    }
                    else
                    {
                            s->tail->next = fifo;
                            s->tail = fifo;
                    }
            }
            //print_queue(s);
    
            for(i = 0; i < 20; i++)
            {
                    pthread_create(&producerVar, NULL, producer, s);
                    pthread_join(producerVar, NULL);
            //}
            //for(i = 0; i < 20; i++)
            //{
                    pthread_create(&consumerVar, NULL, consumer, s);
                    pthread_join(consumerVar, NULL);
            }
    
            return 0;
    }
    
    void *producer(void *arg)
    {
            int i = 0;
            char fileContent[1024];
            char line[1024];
            FILE * myfile;
            struct linked_list *q = arg;
            printf("IN PRODUCER\n");
            //print_queue(q);
            struct fifo_struct* tmp1 = NULL;
            struct fifo_struct* tmp2 = NULL;
    //while(1)
    //{
            sem_wait(&empty);
            pthread_mutex_lock(&mutex);
    
            if(q == NULL)
            {
                    printf("List is empty\n");
                    return(NULL);
            }
            else if(q->head == NULL && q->tail == NULL)
            {
                    printf("List is empty\n");
                    return(NULL);
            }
            else if(q->head == NULL || q->tail == NULL)
            {
                    printf("Error: Problem with code\n");
                    return(NULL);
            }
    
            myfile = fopen(q->head->fileName,"r+");
            while ((fgets(line, 1024, myfile)) != NULL)
            {
                    strcpy(fileContent, line);
                    printf("%s",fileContent);
            }
            fclose(myfile);
    
            while(fileContent[i] != '\n')
            {
                    fileContent[i] = toupper(fileContent[i]);
                    i++;
            }
            //printf("%s\n", fileContent);
            //if(counter < BUFFER_SIZE)
            //{
            //      strncpy(buffer[counter],fileContent,128);
            //      printf("buffer[%d] = %c\n", counter, buffer[counter]);
            //      counter++;
            //}
            strncpy(buffer[0],fileContent,128);
            //printf("%s\n", fileContent);
    
    
            tmp1 = q->head;
            tmp2 = tmp1->next;
            free(tmp1);
            q->head = tmp2;
    
            if(q->head == NULL)
                          q->tail = q->head;
    
            pthread_mutex_unlock(&mutex);
            sem_post(&full);
    //}
            //print_queue(q);
            return(NULL);
    }
    
    void *consumer(void *q)
    {
            int i = 0;
    
            printf("IN CONSUMER\n");
    //while(1)
    //{
            sem_wait(&full);
            pthread_mutex_lock(&mutex);
            printf("%s", buffer[0]);
            pthread_mutex_unlock(&mutex);
            sem_post(&empty);
    //}
            return(NULL);
    }

    So the files open fine, the problem lies somewhere else. But I should check if fopen actually works, thanks for pointing that out.

  2. #17
    Registered User jeffcobb's Avatar
    Join Date
    Dec 2009
    Location
    Henderson, NV
    Posts
    875
    I am working on your app while I am making dinner and watching some anime with the wench. You do know your printf() is fubared, right?
    Code:
    printf("buffer[%d] = %c\n", counter, buffer[counter]);
    Never printf a string into a char. the compiler doesn't like it. I also fixed all of the warnings and errors about your mallocs converting them into the various things you were malloc-ing. Also you do realize you are not leaving your consumers (haven't checked the producers) any way to complete the thread execution, right?

    More later probably..gotta mock up the text files..
    C/C++ Environment: GNU CC/Emacs
    Make system: CMake
    Debuggers: Valgrind/GDB

  3. #18
    Registered User jeffcobb's Avatar
    Join Date
    Dec 2009
    Location
    Henderson, NV
    Posts
    875
    Finally fixed everything that I could see, still need to generate data files but I took a look at your "optimal" output and a really hope you are not expecting something as neat and order in terms of order of execution...real life task switching doesn't work that way... you might have two producers arguing over a mutex, shutting out the consumers (and other producers), things like that so it will execute in a more or less random order, only marginally guided by the semaphores...
    C/C++ Environment: GNU CC/Emacs
    Make system: CMake
    Debuggers: Valgrind/GDB

  4. #19
    Registered User
    Join Date
    May 2009
    Posts
    43
    Quote Originally Posted by jeffcobb View Post
    Finally fixed everything that I could see, still need to generate data files but I took a look at your "optimal" output and a really hope you are not expecting something as neat and order in terms of order of execution...real life task switching doesn't work that way... you might have two producers arguing over a mutex, shutting out the consumers (and other producers), things like that so it will execute in a more or less random order, only marginally guided by the semaphores...
    Yeah the output order or aesthetically pleasing factor, really don't matter. The key is that each piece of text is read once and written once. So the output of my program should just show the content files (up to 128 characters) for each of the 20 files. Btw, I really appreciate the help so far, and sorry for taking a chunk of your life away.

  5. #20
    Registered User
    Join Date
    May 2009
    Posts
    43
    Quote Originally Posted by jeffcobb View Post
    I am working on your app while I am making dinner and watching some anime with the wench. You do know your printf() is fubared, right?
    Code:
    printf("buffer[%d] = %c\n", counter, buffer[counter]);
    Never printf a string into a char. the compiler doesn't like it. I also fixed all of the warnings and errors about your mallocs converting them into the various things you were malloc-ing. Also you do realize you are not leaving your consumers (haven't checked the producers) any way to complete the thread execution, right?

    More later probably..gotta mock up the text files..
    YES! thank you, that's why my buffer was ever printing correctly. That solves a small problem, but better then nothing. I still am not getting the output I'm hoping for, but it was baffling me why that line wasn't printing. +1 thanks.

    Had to change it to %s, obviously.

  6. #21
    Registered User jeffcobb's Avatar
    Join Date
    Dec 2009
    Location
    Henderson, NV
    Posts
    875
    I am seeing a segfault in the code where:
    Code:
            while(fileContent[i] != '\n')
    I will look at it in a bit; have some honey-dos to take care of...
    C/C++ Environment: GNU CC/Emacs
    Make system: CMake
    Debuggers: Valgrind/GDB

  7. #22
    Registered User jeffcobb's Avatar
    Join Date
    Dec 2009
    Location
    Henderson, NV
    Posts
    875
    OK I need to shut down for the night, gotta go into the office at 5-something in the morning and I am beat. I fixed a few issues I have seen but you are getting a segfault in your print queue method; it seems the linked list is being damaged, something I was afraid of. I am trying to resist the urge to just rip out what is here and rewrite it defensively but then it would not be your design anymore so I am trying to fix what I have. Two key things I have done to help clean up the output and help with visualizing which thread is doing what, I created a function for writing stuff to the screen called writeToStdin(const char *ptrOwner, int nOwner, const char *ptrText which is essentially what I posted earlier to keep multiple threads all jumping though the IO layer at once but also now all output is prefaced by either CONSUMER[xx] or PRODUCER[xx] (or MAINTHRD or PRINTQ) and I rejiggered the for loops that created the threads to pass in an instance of a structure like this:
    Code:
    struct TDataPacket
    {
       struct linked_list* ps;
       int nOwner;
    
    };
    So all of the output is like PRODUCER[01]: some-text and so on. Anyhow, like I said, GDB reports your linked list is getting stepped on; p in the while loop is 0x1230, not a good address. Anyhow I tarred up what I have so far; if you want to build this project, install cmake first, then cd to the main folder for this, run cmake . and then make clean && make.

    The link is: http://jbcobb.net/bin/threadfu.tar.gz
    C/C++ Environment: GNU CC/Emacs
    Make system: CMake
    Debuggers: Valgrind/GDB

  8. #23
    Registered User
    Join Date
    May 2009
    Posts
    43
    Quote Originally Posted by jeffcobb View Post
    OK I need to shut down for the night, gotta go into the office at 5-something in the morning and I am beat. I fixed a few issues I have seen but you are getting a segfault in your print queue method; it seems the linked list is being damaged, something I was afraid of. I am trying to resist the urge to just rip out what is here and rewrite it defensively but then it would not be your design anymore so I am trying to fix what I have. Two key things I have done to help clean up the output and help with visualizing which thread is doing what, I created a function for writing stuff to the screen called writeToStdin(const char *ptrOwner, int nOwner, const char *ptrText which is essentially what I posted earlier to keep multiple threads all jumping though the IO layer at once but also now all output is prefaced by either CONSUMER[xx] or PRODUCER[xx] (or MAINTHRD or PRINTQ) and I rejiggered the for loops that created the threads to pass in an instance of a structure like this:
    Code:
    struct TDataPacket
    {
       struct linked_list* ps;
       int nOwner;
    
    };
    So all of the output is like PRODUCER[01]: some-text and so on. Anyhow, like I said, GDB reports your linked list is getting stepped on; p in the while loop is 0x1230, not a good address. Anyhow I tarred up what I have so far; if you want to build this project, install cmake first, then cd to the main folder for this, run cmake . and then make clean && make.

    The link is: http://jbcobb.net/bin/threadfu.tar.gz
    That's weird that you are getting seg faults, my code compiles and runs perfectly fine without any noticeable errors. But I'll check out what you have in the morning, thanks again!

  9. #24
    Registered User
    Join Date
    May 2009
    Posts
    43
    Quote Originally Posted by jeffcobb View Post
    I am seeing a segfault in the code where:
    Code:
            while(fileContent[i] != '\n')
    I will look at it in a bit; have some honey-dos to take care of...

    Just realized that you don't have my input files, so that could be why you're having a problem..


    How do I attach my files? I have them in a zip..

  10. #25
    Registered User jeffcobb's Avatar
    Join Date
    Dec 2009
    Location
    Henderson, NV
    Posts
    875
    As you can see I made my own. The issue was I just wrote straight data and closed the file; you probably typed yours and had a \n at the end of each line of input. Changing the condition to watch for the end of the string (NULL) was safer anyways. I just need to sort out what the producers are doing to the linked list...
    C/C++ Environment: GNU CC/Emacs
    Make system: CMake
    Debuggers: Valgrind/GDB

  11. #26
    Registered User
    Join Date
    May 2009
    Posts
    43
    Quote Originally Posted by jeffcobb View Post
    As you can see I made my own. The issue was I just wrote straight data and closed the file; you probably typed yours and had a \n at the end of each line of input. Changing the condition to watch for the end of the string (NULL) was safer anyways. I just need to sort out what the producers are doing to the linked list...
    Maybe I'm not understanding how creating pthreads work, but when I use pthread_join and print the file name queue list inside each producer, it prints 19 file names, then 18 file names, then 17 file names, then 16 file names,.... , then 1 file names, then ends. Which is ideal output. However, when I comment out the pthread_join inside my loop:

    for(i = 0; i < PRODUCERS; i++)
    {
    pthread_create(&producerVar[i], NULL, producer, s);
    // pthread_join(producerVar[i], NULL);

    }

    then my list doesn't get dequeued properly, and things are a mess. Why is it when I create 20 threads without pthread_join, the queue list isn't working?

    (NOTE, this is with commenting out everything except the producer and my main. The consumer and buffer are basically taken out of the equation right now. I just want my queue list to get dequeued properly inside my producer after creating 20 threads.)

  12. #27
    Registered User jeffcobb's Avatar
    Join Date
    Dec 2009
    Location
    Henderson, NV
    Posts
    875
    Because when you hit the join(), you are telling that thread to stop until the last thread you started finished. This is the basic flaw with doing one producer thread per input. I am (slowly, have other stuff going on) working on one that works more normal that you may even like and the linked list is kept cleaner.
    C/C++ Environment: GNU CC/Emacs
    Make system: CMake
    Debuggers: Valgrind/GDB

  13. #28
    Registered User
    Join Date
    May 2009
    Posts
    43
    Quote Originally Posted by jeffcobb View Post
    Because when you hit the join(), you are telling that thread to stop until the last thread you started finished. This is the basic flaw with doing one producer thread per input. I am (slowly, have other stuff going on) working on one that works more normal that you may even like and the linked list is kept cleaner.
    OK, I fully understand that. But when I run the code without the pthread-join, I still don't get why the output only prints the producer twice like so:
    IN PRODUCER
    int = in1.txt
    int = in2.txt
    int = in3.txt
    int = in4.txt
    int = in5.txt
    int = in6.txt
    int = in7.txt
    int = in8.txt
    int = in9.txt
    int = in10.txt
    int = in11.txt
    int = in12.txt
    int = in13.txt
    int = in14.txt
    int = in15.txt
    int = in16.txt
    int = in17.txt
    int = in18.txt
    int = in19.txt

    IN PRODUCER
    int = in2.txt
    int = in3.txt
    int = in4.txt
    int = in5.txt
    int = in6.txt
    int = in7.txt
    int = in8.txt
    int = in9.txt
    int = in10.txt
    int = in11.txt
    int = in12.txt

    Whereas the output with the join call prints out like so:
    IN PRODUCER
    int = in1.txt
    int = in2.txt
    int = in3.txt
    int = in4.txt
    int = in5.txt
    int = in6.txt
    int = in7.txt
    int = in8.txt
    int = in9.txt
    int = in10.txt
    int = in11.txt
    int = in12.txt
    int = in13.txt
    int = in14.txt
    int = in15.txt
    int = in16.txt
    int = in17.txt
    int = in18.txt
    int = in19.txt

    IN PRODUCER
    int = in2.txt
    int = in3.txt
    int = in4.txt
    int = in5.txt
    int = in6.txt
    int = in7.txt
    int = in8.txt
    int = in9.txt
    int = in10.txt
    int = in11.txt
    int = in12.txt
    int = in13.txt
    int = in14.txt
    int = in15.txt
    int = in16.txt
    int = in17.txt
    int = in18.txt
    int = in19.txt

    IN PRODUCER
    int = in3.txt
    int = in4.txt
    int = in5.txt
    int = in6.txt
    int = in7.txt
    int = in8.txt
    int = in9.txt
    int = in10.txt
    int = in11.txt
    int = in12.txt
    int = in13.txt
    int = in14.txt
    int = in15.txt
    int = in16.txt
    int = in17.txt
    int = in18.txt
    int = in19.txt

    IN PRODUCER
    int = in4.txt
    int = in5.txt
    int = in6.txt
    int = in7.txt
    int = in8.txt
    int = in9.txt
    int = in10.txt
    int = in11.txt
    int = in12.txt
    int = in13.txt
    int = in14.txt
    int = in15.txt
    int = in16.txt
    int = in17.txt
    int = in18.txt
    int = in19.txt

    IN PRODUCER
    int = in5.txt
    int = in6.txt
    int = in7.txt
    int = in8.txt
    int = in9.txt
    int = in10.txt
    int = in11.txt
    int = in12.txt
    int = in13.txt
    int = in14.txt
    int = in15.txt
    int = in16.txt
    int = in17.txt
    int = in18.txt
    int = in19.txt

    IN PRODUCER
    int = in6.txt
    int = in7.txt
    int = in8.txt
    int = in9.txt
    int = in10.txt
    int = in11.txt
    int = in12.txt
    int = in13.txt
    int = in14.txt
    int = in15.txt
    int = in16.txt
    int = in17.txt
    int = in18.txt
    int = in19.txt

    IN PRODUCER
    int = in7.txt
    int = in8.txt
    int = in9.txt
    int = in10.txt
    int = in11.txt
    int = in12.txt
    int = in13.txt
    int = in14.txt
    int = in15.txt
    int = in16.txt
    int = in17.txt
    int = in18.txt
    int = in19.txt

    IN PRODUCER
    int = in8.txt
    int = in9.txt
    int = in10.txt
    int = in11.txt
    int = in12.txt
    int = in13.txt
    int = in14.txt
    int = in15.txt
    int = in16.txt
    int = in17.txt
    int = in18.txt
    int = in19.txt

    IN PRODUCER
    int = in9.txt
    int = in10.txt
    int = in11.txt
    int = in12.txt
    int = in13.txt
    int = in14.txt
    int = in15.txt
    int = in16.txt
    int = in17.txt
    int = in18.txt
    int = in19.txt

    IN PRODUCER
    int = in10.txt
    int = in11.txt
    int = in12.txt
    int = in13.txt
    int = in14.txt
    int = in15.txt
    int = in16.txt
    int = in17.txt
    int = in18.txt
    int = in19.txt

    IN PRODUCER
    int = in11.txt
    int = in12.txt
    int = in13.txt
    int = in14.txt
    int = in15.txt
    int = in16.txt
    int = in17.txt
    int = in18.txt
    int = in19.txt

    IN PRODUCER
    int = in12.txt
    int = in13.txt
    int = in14.txt
    int = in15.txt
    int = in16.txt
    int = in17.txt
    int = in18.txt
    int = in19.txt

    IN PRODUCER
    int = in13.txt
    int = in14.txt
    int = in15.txt
    int = in16.txt
    int = in17.txt
    int = in18.txt
    int = in19.txt

    IN PRODUCER
    int = in14.txt
    int = in15.txt
    int = in16.txt
    int = in17.txt
    int = in18.txt
    int = in19.txt

    IN PRODUCER
    int = in15.txt
    int = in16.txt
    int = in17.txt
    int = in18.txt
    int = in19.txt

    IN PRODUCER
    int = in16.txt
    int = in17.txt
    int = in18.txt
    int = in19.txt

    IN PRODUCER
    int = in17.txt
    int = in18.txt
    int = in19.txt

    IN PRODUCER
    int = in18.txt
    int = in19.txt

    IN PRODUCER
    int = in19.txt

    IN PRODUCER




    Well this is a school project, so I do prefer to do my own code (Not just because of moral and academic integrity reasons, but also to learn!), but I greatly appreciate your help. With that being said, send me what you have when you're done haha. I've put 2 weeks of anywhere from 2-5 hours a day on this project, so I just want to get this done. Thanks.

  14. #29
    Registered User
    Join Date
    May 2009
    Posts
    43
    YES I finally got it to work for just one buffer slot. Everything is working fine. The TA gave me some tips, apparently my linked list s, should be declared GLOBALLY.

    That and a few other things, and things are looking good, woo. Now I just need to clean up my code and get it to run with 5 buffer slots.
    Last edited by hansel13; 05-26-2010 at 06:31 PM.

  15. #30
    Registered User jeffcobb's Avatar
    Join Date
    Dec 2009
    Location
    Henderson, NV
    Posts
    875
    Thursday as in tomorrow? OK.

    Well I will do what I can tonight to give some guidance but honestly for a BFO into what is *actually* happening, assign each producer/consumer an id number and have them print it in your output. What you are going to find is something like this:
    1. You need to read 20 files and ultimately uppercase them (the "work" and then print them to the screen.
    2. You want the thing with the most work to have the most threads.
    3. The way your producers are set up is they (all 20) will fire up, fight over a mutex, pull off a filename, read that file, stick it in the data store, set the semaphore for the readers and the *loop* to see if there is more work.
    4. The readers fight over the same mutex, 1 (one) of them wins, pulls off a value and prints it to the screen.

    Now of the work presented, the way you have distributed it, the producers might as well print it to the screen and be done with it if getting the job done is the point; the readers are superfluous.

    *however*

    What actually happens on a dual-core box is:
    The first 1 or 2 producers will do 90% of the work. This is what will be revealed if you assign each one a number as I had done some time ago. That means that 90-95% of your producers will spin on the while(1) loop and do nothing but eat clock cycles. If they all try to lock the same mutex, most won't even finish. This will demonstrate the lunacy of 20 producers: they won't get the job done 20x faster and most won't do anything at all.

    The readers on the other hand really don't have much to do , just fight over a mutex and print a string to the screen.

    In a real producer/consumer model, the producers (usually just a handful if even more than 1 or 2) will simply describe the job which could look like this:

    Code:
    struct TJob
    {
         // data
         char achBuffer[MAX_FILENAME_SIZE];
         bool bProcessed; // has this job been done yet?
    };
    
    // you know you only have 20 jobs so...
    struct TJob jobQueue[20];
    Then you keep a job count (how many are in your jobQueue), the currentAddPosition (init to -1) and the current job being done. Init this to -1;

    Spin your threads as follows:
    1-2 (only one is necessary) producers that feed filenames into the queue

    4-5 readers.

    Make a mutex-protected function to add to the queue which would involve:
    increment the totalJobs variablle.
    if( currentAddPosition == -1)
    Its the first job so set it to 0
    setCurrentJob to 0
    else
    increment the currentAddPosition
    copy job to jobQueue[currentAddPosition]
    incrementTotalJobs

    then you need a reader function (something to dole out the "jobs" to readers" so
    that function would:
    read the jobQueue[currentJob]
    decrement totalJobs
    increment currentJob

    The producers all the top function, the readers call the bottom one (protected by mutexes but ONLY as long as it takes to push data into it or pull data from jobQueue. The rest of the time let the threads work in solitude. The readers producers could read the files and push the contents into the jobQueue, readers (and long as each gets a unique index into the data) multiply access the data store at once, getting the data, upper-casing it and printing to the screen. Be prepared for a bottle-neck at the IO layer (screen)..
    C/C++ Environment: GNU CC/Emacs
    Make system: CMake
    Debuggers: Valgrind/GDB

Page 2 of 4 FirstFirst 1234 LastLast
Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Need Help: Multi-threading and Synchronization
    By Anom in forum C Programming
    Replies: 7
    Last Post: 12-08-2009, 04:34 PM
  2. CFiledialog threads termination
    By ks_lohith in forum C++ Programming
    Replies: 0
    Last Post: 09-11-2009, 12:57 AM
  3. problems creating a linked list
    By jamjar in forum C Programming
    Replies: 5
    Last Post: 10-23-2002, 05:50 AM
  4. Block and wake up certain threads
    By Spark in forum C Programming
    Replies: 9
    Last Post: 06-01-2002, 03:39 AM
  5. problems using threads.....?
    By stumpert in forum C++ Programming
    Replies: 2
    Last Post: 04-27-2002, 09:30 PM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21