pthreads Seg Fault woes

OK, I'm feeling really dumb right now on a C program I'm working on. I'm getting a segfault that I assume is due to an int array I am creating. I say "i assume" because when I comment out that line of code, the segfaults go away.

The code is multi-threaded using pthreads, and I change the number of threads by a #define directive. A low number of threads never causes the segfault. I am manually assigning the stack size before creating the threads, and I am including the overhead of the array.

I've narrowed down the potential area where it occurs and I've gone through and commented out ALL array indexing operations to get eliminate the possibility of bad indexing. It's to the point where I am literally just indexing through for loops.

here's how I am allocating the thread stack size:

Code:

---

pthread_attr_t attr;
pthread_attr_init(&attr);
pthread_attr_getstacksize (&attr, &s);
s = 7 * sizeof(int) + 1 * sizeof(long) +
#ifndef SKIPSTEAL
          sizeof(int[NUM_THREADS - 1]) +
#endif
              sizeof(struct1 *) + sizeof(struct2 *) + 100;
if (s < PTHREAD_STACK_MIN)
    s = PTHREAD_STACK_MIN;
pthread_attr_setstacksize (&attr, s);

---

all of the threads are successfully created before the segfault, and if I run out of memory, the program catches it and aborts.

I can't get the same place to show up in GDB for the segfault.

WTF am I missing here?

Why are you messing about trying to set the stack sizes anyway?

I have to wonder, because your calculation seems to add up to 140 bytes, plus whatever sizeof(int[NUM_THREADS - 1]) evaluates to.
On my system, PTHREAD_STACK_MIN is 16K.

One thing you are missing is any error checking on the functions you're calling.
pthread_attr_setstacksize

> I can't get the same place to show up in GDB for the segfault.
But do you get a segfault in gdb?
Posting the stack trace might help.

I'm messing with the stack size because I want to make a large number of threads, sometimes greater than 1000, for the purpose of testing for deadlocks and such. The standard stack size limits me, on my machine, to about 387 threads.

And, yes, I definitely get the segfault on gdb. Sometimes it occurs in my code, sometimes it occurs in the "compiler code". And it moves around, depending on what code I have enabled. Here's my thread function, and yes, it's frickin nasty with all of the #if's at this point.

Code:

---

void *DoStuff(void* theData) {
        int loop = 0;
        int valueToWait = 0;
        int i, j;
        int myCount;
        int val;
        thread_data * data = (thread_data *) theData;
        my_deque * deque = data->deque;
        int victim;

#ifdef DOSTEAL
        unsigned int t;
        long oldAge;

#ifdef POPTOPHERE
        unsigned int localBot;
        long newAge;
#endif /* POPTOPHERE */
#endif /* DOSTEAL */
#ifndef NOARRAY
        int threads[NUM_THREADS - 1];
#endif

        while (1) {
                #ifdef DEBUG
                if (DEBUG >= 1)
                        printf("thread %5i approaching barrier in loop %i.  numArrived %i\n", data->id, loop, numArrived);
                #endif

                // barrier
                valueToWait = 1 - valueToWait;
                pthread_mutex_lock(&mut);
                numArrived++;
                myCount = numArrived;

                // wait for the signal, man
                while ((allDie == 0) && (canGo != valueToWait)) {
                        pthread_cond_wait(&cond, &mut);
                }

                pthread_mutex_unlock(&mut);

                // is it time to go?
                if (allDie != 0) {
                        #ifdef DEBUG
                        if (DEBUG >= 1)
                                printf("thread %5i dying in loop %i\n", data->id, loop);
                        #endif
                        pthread_exit(NULL);
                        break;
                }

                // do some work
                data->writes = 0;
                data->misses = 0;
                data->stealAttempts = 0;
                data->steals = 0;
#ifndef SKIPNORMAL
                while (1) {
                        j = (int)(deque->back);
                        if (j <= 0)
                                break;

                        val = (int)(my_deque_pop_bottom(deque));

//                        printf("j: %i  val: %i  datum->age:  %u  data[j]: %i\n", j, val, datum->age, (int)(datum->data[j]));

                        if (val == 0) {
                                deque->data[j-1] = data->id;
                                data->writes++;
                        }
                        else if (val == -1)
                                my_deque_push_bottom(deque, 0);
                        else if (val == (int)NIL) {
//                                printf("Thread %5i got a NIL for slot %5i in loop %i\n", datum->id, j, loop);
//                                break;
                        }
                        else
                                break;
                }
#endif /* SKIPNORMAL */

#ifndef SKIPSTEAL
                // now, try to steal, yo!
                {
                        // set up my list of threads I've used
#ifndef SKIPMAKEARRAY
                        for (j = 0; j < NUM_THREADS - 1; j++) {
                                if (j < data->id-1)
                                        threads[j] = j;
                                else
                                        threads[j] = j+1;
                        }

                        // print the array to screen
#ifdef PRINTTHREADARRAY
                        printf("Thread %5i's threads array:\n", data->id);
                        for (j = 0; j < NUM_THREADS - 1; j++)
                                printf("  threads[%6i]:  %6i\n", j, threads[j]);
#endif /* PRINTTHREADARRAY */
#endif /* SKIPMAKEARRAY */

                        // go through all the threads
                        for (j = 0; j < NUM_THREADS - 1; j++) {
                                // pick a victim (pictim!)
#ifdef RANDOM
                                i = (rand() % (NUM_THREADS - 1 - j));
                                victim = threads[i];
                                if (victim == INT_MIN) {
                                        printf("something's wrong, sherlock!\n");
                                        continue;
                                }

                                // swap victim with last number in list
                                threads[i] = threads[NUM_THREADS - 2 - j];                                                              /* changed line of code */
                                threads[NUM_THREADS - 2 - j] = INT_MIN;        // put a dummy value in so we know it's bad                    /* changed line of code */
#else
                                victim = j;
#endif /* RANDOM */

#ifdef DOSTEAL
                                // now, start from top and try to steal
                                while (1) {
                                        oldAge = threadData[victim].deque->age;
                                        t = top(oldAge);

                                        if (threadData[victim].deque->back > t) {
                                                data->stealAttempts++;

                                                // don't pop it if it's a sentinel
                                                if (threadData[victim].deque->data[t] != 0)
                                                        continue;
#ifdef POPTOPHERE
                                                /* this section of code does the popping right here */
                                                localBot = threadData[victim].deque->back;
                                                if (localBot <= t) {
                                                        val = NIL;
                                                        goto PopDone;
                                                }
                                                val = threadData[victim].deque->data[t];
                                                newAge = oldAge + 1;
                                                // can do a ++ here because we are only incrementing the top
                                                newAge = __sync_val_compare_and_swap(&(threadData[victim].deque->age), oldAge, newAge);
                                                if (oldAge != newAge)
                                                        val = NIL;
#else
                                                /* this section of code tries to use the pop function */
                                                // make sure age hasn't changed, just for kicks
                                                // this reduces possibility for a race, but doesn't eliminate it
                                                if (oldAge != threadData[victim].deque->age)
                                                        continue;

                                                val = (int)(my_deque_pop_top(threadData[victim].deque));
#endif /* POPTOPHERE */
                                        PopDone:
                                                if (val == 0) {
                                                        threadData[victim].deque->data[t] = data->id;
                                                        data->writes++;
                                                        data->steals++;
                                                }
// removed this because this caused threads to quit when they got rejected
//    which isn't always the right thing to do
//                                                else        // stop stealing from this guy
//                                                        break;
                                        }
                                        else        // this guy is already finished
                                                break;
                                }
#endif        /* DOSTEAL */
                        }
                }
#endif /* SKIPSTEAL */

                // calculate number of missed writes
                for (i = 0; i < MY_DEQUE_ARRAY_SIZE; i++) {
                        if (deque->data[i] == 0)
                                data->misses++;
                }

                // now, print out the results
#ifndef SKIPPRINT
                printf("Thread %5i done with loop %i.  Writes: %5i  Steals Attempts: %10li  Missed Writes: %5i\n", data->id, loop, data->writes, data->steals, data->misses);
#endif

                loop++;
        }
}

---

Sometimes it segfaults in the thread, sometimes it segfaults when trying to join the to the threads to end program execution. In case you are curious, yes, this code is for an assignment, and I hope that's kosher to ask on here. I'm more curious about the segfault than any particular way to solve the programming assignment.

I don't know how all your #defines are set up, but you allocate stack size for int[NUM_THREADS - 1] if SKIPSTEAL is not defined, but you actually define the local variable threads if NOARRAY is not defined. It seems that, if NOARRAY is not define, and DoStuff has int threads[NUM_THREADS - 1] on the stack, but SKIPSTEAL is defined, the pthread stack size will not account for threads.

A couple side notes:
1. You can use gcc -E to see the result of your C code after the preprocessor handles all your #includes, #ifdefs, etc. There will be a bunch of mess at the top from the standard include headers, but it can be quite helpful.
2. Negative boolean names (NOARRAY instead of ARRAY) can lead to confusing, double negative boolean statements, like ifndef NOARRAY, which would be logically equivalent to ifdef ARRAY. The only difference is that the latter is usually much easier to grok and keep track of. This can add up with all your conditional compilation code.

Yeah, I intentionally set it up so that I can keep the array in even if I skip the stealing section, because that array being defined initially seemed to be what causes the segfault. Thus, I wanted to be able to cause the memory to be allocated without actually using the array to see if the allocation, itself was causing the segfault. This didn't prove to be true, so that's just a hold over.

I've narrowed it down, right now, to the random section, but that didn't seem to matter before... Right now, as long as the random section doesn't run, then there's no segfault, and the other options don't seem to matter.

I'm going to edit the code above, if it will let me, with the current version of the thread function.


Oddly enough, I found something that SHOULD have caused a segfault, but doesn't seem to do so. I changed
NUM_THREADS - 1 - j
to
NUM_THREADS - 2 - j
in the part where I swap an entry in the threads array. In the other places, it seems to be correct
the original would have caused the threads array to be read one past the end of the array, which I would think would cause a segfault, but it didn't seem to do so. I fixed it in the code above

wow, this is getting CRAZY.

so, I've narrowed it down to the line:

Code:

---

victim = threads[i];

---

comment that line out, no segfault. note, however, that there is a line following that where I assign a value to threads[i] that it seems to have no problem with. even better, I can do a line as follows with no segfault:

Code:

---

k = threads[i];

---

but, if I then set victim equal to k, segfault. what the deuce?


OK, addendum, even weirder. if I change the next IF statement to use k instead of victim when I set k = threads[i], it segfaults. so, that IF statement is also tied into it. comment out the if statement, no segfault. if I subtitute "threads[i]" in for victim/k in the IF statement, segfault.

so, basically, this line is really giving the segfault:

Code:

---

if (victim == INT_MIN) { /* foo */ }

---

NOTE: this behaviour ONLY occurs if NUM_THREADS is sufficiently high. on my system, at 1758, it faults. 1757, it doesn't. haven't tried it on any others to see if I can replicate those numbers.
NOTE NOTE: i tested it on another system, the behaviour is the same, except for the cutoff for NUM_THREADS is different. everything else is the same

as it stands right now, gdb reports the following for 1758 threads:
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xb6644b70 (LWP 9067)]
0x0011f07a in ?? () from /lib/ld-linux.so.2

Since we don't know which conditionals you've chosen, post the code which you're testing with without all the #ifdefs all over the place.

Also, factoring out some of the work into other functions would make doStuff() more readable.
I mean, you have 2 (or is it 3) nested while(1) loops going on.

Also, I think you're chasing ghosts.
Unusual software bug - Wikipedia, the free encyclopedia
Whilst you may have nailed down a symptom, I don't think you're any nearer understanding a root cause.

Have you tried using say valgrind on a small number of threads?

Also (it's my word of the day), your stack size calculation doesn't take into account things like
- thread infomation
- function call/return overhead
- temporary variable overhead
In short, summing a few sizeof() in one significant stack frame isn't going to give you the total amount needed.

i have no clue how to do a valgrind. anyway, I ran a gcc -E on the code, and here's what I got. a LOT of stuff is taken out, but that's what I expected. it corresponds to SKIPPRINT and RANDOM being defined.

Code:

---

void *DoStuff(void* theData) {
        int loop = 0;
        int valueToWait = 0;
        int i, j;
        int k;
        int myCount;
        int val;
        thread_data * data = (thread_data *) theData;
        my_deque * deque = data->deque;
        int victim = 0;
        int threads[1758 - 1];

        while (1) {
                valueToWait = 1 - valueToWait;
                pthread_mutex_lock(&mut);
                numArrived++;
                myCount = numArrived;

                while ((allDie == 0) && (canGo != valueToWait)) {
                        pthread_cond_wait(&cond, &mut);
                }

                pthread_mutex_unlock(&mut);

                if (allDie != 0) {
                        pthread_exit(((void *)0));
                        break;
                }

                data->writes = 0;
                data->misses = 0;
                data->stealAttempts = 0;
                data->steals = 0;

                while (1) {
                        j = (int)(deque->back);
                        if (j <= 0)
                                break;

                        val = (int)(my_deque_pop_bottom(deque));

                        if (val == 0) {
                                deque->data[j-1] = data->id;
                                data->writes++;
                        }
                        else if (val == -1) {
                                my_deque_push_bottom(deque, 0);
                        }
                        else if (val == (int)-2) {
                        }
                        else
                                break;
                }

                {
                        for (j = 0; j < 1758 - 1; j++) {
                                if (j < data->id-1)
                                        threads[j] = j;
                                else
                                        threads[j] = j+1;
                        }

                        for (j = 0; j < 1758 - 1; j++) {
                                i = rand() % (1758 - 1 - j);
                                victim = threads[i];
                                if (victim == -2147483648 - 1) {
                                        printf("something's wrong, sherlock!\n");
                                        continue;
                                }

                                threads[i] = threads[1758 - 2 - j];
                                threads[1758 - 2 - j] = (-2147483647 - 1);
                        }
                }

                for (i = 0; i < 2048; i++) {
                        if ((deque->data[i] == 0) || (deque->data[i] == -1))
                                data->misses++;
                }

                loop++;
        }
}

---

As for figuring out how much stack space is needed, is there a way I can reasonably figure out what's needed by the rest of that stuff? I gave it the extra 100 as a fudge factor, but it sounds like that wasn't enough fudge...

Quote:

---

Originally Posted by aaronburro

i have no clue how to do a valgrind.

---

Amazingly powerful tool. Learning curve can seem a bit steep, but it's really not that bad. There's lots of tutorials on the web, including one here. If you plan on doing any hardcore C development, valgrind is invaluable.

Quote:

---

As for figuring out how much stack space is needed, is there a way I can reasonably figure out what's needed by the rest of that stuff? I gave it the extra 100 as a fudge factor, but it sounds like that wasn't enough fudge...

---

You would have to do some serious profiling. Valgrind should help with this and there are other tools, I just don't know what they are. I think you may be pre-optimizing. Use the default stack size, and see how many threads you can spin up. You may be able to get enough going and still test your deadlock cases without all this crazy stack size manipulation. If that's insufficient, just overengineer. Take what you think you need and multiply it by 10. That will probably be enough, and still be significantly smaller than your default. If it's still too much, whittle it down until you get the smallest number you can without seg faulting. But be warned, that's a crude method. You would have to test every possible execution path to make sure you wont seg fault from a too-small stack. It should be a very last resort.

Well, I changed the fudge factor to include 10KB extra for the stack size, and that seems to have cleared everything up. Yay for messing with stuff that is over my head at this point. Thx for the help, all.