Thread: pthreads Seg Fault woes

  1. #1
    Registered User
    Join Date
    Apr 2011
    Posts
    6

    pthreads Seg Fault woes

    OK, I'm feeling really dumb right now on a C program I'm working on. I'm getting a segfault that I assume is due to an int array I am creating. I say "i assume" because when I comment out that line of code, the segfaults go away.

    The code is multi-threaded using pthreads, and I change the number of threads by a #define directive. A low number of threads never causes the segfault. I am manually assigning the stack size before creating the threads, and I am including the overhead of the array.

    I've narrowed down the potential area where it occurs and I've gone through and commented out ALL array indexing operations to get eliminate the possibility of bad indexing. It's to the point where I am literally just indexing through for loops.

    here's how I am allocating the thread stack size:
    Code:
    pthread_attr_t attr;
    pthread_attr_init(&attr);
    pthread_attr_getstacksize (&attr, &s);
    s = 7 * sizeof(int) + 1 * sizeof(long) +
    #ifndef SKIPSTEAL
               sizeof(int[NUM_THREADS - 1]) +
    #endif
        	   sizeof(struct1 *) + sizeof(struct2 *) + 100;
    if (s < PTHREAD_STACK_MIN)
         s = PTHREAD_STACK_MIN;
    pthread_attr_setstacksize (&attr, s);
    all of the threads are successfully created before the segfault, and if I run out of memory, the program catches it and aborts.

    I can't get the same place to show up in GDB for the segfault.

    WTF am I missing here?

  2. #2
    and the hat of int overfl Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    39,661
    Why are you messing about trying to set the stack sizes anyway?

    I have to wonder, because your calculation seems to add up to 140 bytes, plus whatever sizeof(int[NUM_THREADS - 1]) evaluates to.
    On my system, PTHREAD_STACK_MIN is 16K.

    One thing you are missing is any error checking on the functions you're calling.
    pthread_attr_setstacksize

    > I can't get the same place to show up in GDB for the segfault.
    But do you get a segfault in gdb?
    Posting the stack trace might help.
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.

  3. #3
    Registered User
    Join Date
    Apr 2011
    Posts
    6
    I'm messing with the stack size because I want to make a large number of threads, sometimes greater than 1000, for the purpose of testing for deadlocks and such. The standard stack size limits me, on my machine, to about 387 threads.

    And, yes, I definitely get the segfault on gdb. Sometimes it occurs in my code, sometimes it occurs in the "compiler code". And it moves around, depending on what code I have enabled. Here's my thread function, and yes, it's frickin nasty with all of the #if's at this point.

    Code:
    void *DoStuff(void* theData) {
    	int loop = 0;
    	int valueToWait = 0;
    	int i, j;
    	int myCount;
    	int val;
    	thread_data * data = (thread_data *) theData;
    	my_deque * deque = data->deque;
    	int victim;
    
    #ifdef DOSTEAL
    	unsigned int t;
    	long oldAge;
    
    #ifdef POPTOPHERE
    	unsigned int localBot;
    	long newAge;
    #endif /* POPTOPHERE */
    #endif /* DOSTEAL */
    #ifndef NOARRAY
    	int threads[NUM_THREADS - 1];
    #endif
    
    	while (1) {
    		#ifdef DEBUG
    		if (DEBUG >= 1)
    			printf("thread %5i approaching barrier in loop %i.  numArrived %i\n", data->id, loop, numArrived);
    		#endif
    
    		// barrier
    		valueToWait = 1 - valueToWait;
    		pthread_mutex_lock(&mut);
    		numArrived++;
    		myCount = numArrived;
    
    		// wait for the signal, man
    		while ((allDie == 0) && (canGo != valueToWait)) {
    			pthread_cond_wait(&cond, &mut);
    		}
    
    		pthread_mutex_unlock(&mut);
    
    		// is it time to go?
    		if (allDie != 0) {
    			#ifdef DEBUG
    			if (DEBUG >= 1)
    				printf("thread %5i dying in loop %i\n", data->id, loop);
    			#endif
    			pthread_exit(NULL);
    			break;
    		}
    
    		// do some work
    		data->writes = 0;
    		data->misses = 0;
    		data->stealAttempts = 0;
    		data->steals = 0;
    #ifndef SKIPNORMAL
    		while (1) {
    			j = (int)(deque->back);
    			if (j <= 0)
    				break;
    
    			val = (int)(my_deque_pop_bottom(deque));
    
    //			printf("j: %i   val: %i   datum->age:  %u   data[j]: %i\n", j, val, datum->age, (int)(datum->data[j]));
    
    			if (val == 0) {
    				deque->data[j-1] = data->id;
    				data->writes++;
    			}
    			else if (val == -1)
    				my_deque_push_bottom(deque, 0);
    			else if (val == (int)NIL) {
    //				printf("Thread %5i got a NIL for slot %5i in loop %i\n", datum->id, j, loop);
    //				break;
    			}
    			else
    				break;
    		}
    #endif /* SKIPNORMAL */
    
    #ifndef SKIPSTEAL
    		// now, try to steal, yo!
    		{
    			// set up my list of threads I've used
    #ifndef SKIPMAKEARRAY
    			for (j = 0; j < NUM_THREADS - 1; j++) {
    				if (j < data->id-1)
    					threads[j] = j;
    				else
    					threads[j] = j+1;
    			}
    
    			// print the array to screen
    #ifdef PRINTTHREADARRAY
    			printf("Thread %5i's threads array:\n", data->id);
    			for (j = 0; j < NUM_THREADS - 1; j++)
    				printf("   threads[%6i]:  %6i\n", j, threads[j]);
    #endif /* PRINTTHREADARRAY */
    #endif /* SKIPMAKEARRAY */
    
    			// go through all the threads
    			for (j = 0; j < NUM_THREADS - 1; j++) {
    				// pick a victim (pictim!)
    #ifdef RANDOM
    				i = (rand() % (NUM_THREADS - 1 - j));
    				victim = threads[i];
    				if (victim == INT_MIN) {
    					printf("something's wrong, sherlock!\n");
    					continue;
    				}
    
    				// swap victim with last number in list
    				threads[i] = threads[NUM_THREADS - 2 - j];                                                              /* changed line of code */
    				threads[NUM_THREADS - 2 - j] = INT_MIN;	// put a dummy value in so we know it's bad                     /* changed line of code */
    #else
    				victim = j;
    #endif /* RANDOM */
    
    #ifdef DOSTEAL
    				// now, start from top and try to steal
    				while (1) {
    					oldAge = threadData[victim].deque->age;
    					t = top(oldAge);
    
    					if (threadData[victim].deque->back > t) {
    						data->stealAttempts++;
    
    						// don't pop it if it's a sentinel
    						if (threadData[victim].deque->data[t] != 0)
    							continue;
    #ifdef POPTOPHERE
    						/* this section of code does the popping right here */
    						localBot = threadData[victim].deque->back;
    						if (localBot <= t) {
    							val = NIL;
    							goto PopDone;
    						}
    						val = threadData[victim].deque->data[t];
    						newAge = oldAge + 1;
    						// can do a ++ here because we are only incrementing the top
    						newAge = __sync_val_compare_and_swap(&(threadData[victim].deque->age), oldAge, newAge);
    						if (oldAge != newAge)
    							val = NIL;
    #else
    						/* this section of code tries to use the pop function */
    						// make sure age hasn't changed, just for kicks
    						// this reduces possibility for a race, but doesn't eliminate it
    						if (oldAge != threadData[victim].deque->age)
    							continue;
    
    						val = (int)(my_deque_pop_top(threadData[victim].deque));
    #endif /* POPTOPHERE */
    					PopDone:
    						if (val == 0) {
    							threadData[victim].deque->data[t] = data->id;
    							data->writes++;
    							data->steals++;
    						}
    // removed this because this caused threads to quit when they got rejected
    //    which isn't always the right thing to do
    //						else	// stop stealing from this guy
    //							break;
    					}
    					else	// this guy is already finished
    						break;
    				}
    #endif	/* DOSTEAL */
    			}
    		}
    #endif /* SKIPSTEAL */
    
    		// calculate number of missed writes
    		for (i = 0; i < MY_DEQUE_ARRAY_SIZE; i++) {
    			if (deque->data[i] == 0)
    				data->misses++;
    		}
    
    		// now, print out the results
    #ifndef SKIPPRINT
    		printf("Thread %5i done with loop %i.  Writes: %5i   Steals Attempts: %10li   Missed Writes: %5i\n", data->id, loop, data->writes, data->steals, data->misses);
    #endif
    
    		loop++;
    	}
    }
    Sometimes it segfaults in the thread, sometimes it segfaults when trying to join the to the threads to end program execution. In case you are curious, yes, this code is for an assignment, and I hope that's kosher to ask on here. I'm more curious about the segfault than any particular way to solve the programming assignment.
    Last edited by aaronburro; 04-27-2011 at 02:28 AM. Reason: change code

  4. #4
    Registered User
    Join Date
    Nov 2010
    Location
    Long Beach, CA
    Posts
    5,909
    I don't know how all your #defines are set up, but you allocate stack size for int[NUM_THREADS - 1] if SKIPSTEAL is not defined, but you actually define the local variable threads if NOARRAY is not defined. It seems that, if NOARRAY is not define, and DoStuff has int threads[NUM_THREADS - 1] on the stack, but SKIPSTEAL is defined, the pthread stack size will not account for threads.

    A couple side notes:
    1. You can use gcc -E to see the result of your C code after the preprocessor handles all your #includes, #ifdefs, etc. There will be a bunch of mess at the top from the standard include headers, but it can be quite helpful.
    2. Negative boolean names (NOARRAY instead of ARRAY) can lead to confusing, double negative boolean statements, like ifndef NOARRAY, which would be logically equivalent to ifdef ARRAY. The only difference is that the latter is usually much easier to grok and keep track of. This can add up with all your conditional compilation code.

  5. #5
    Registered User
    Join Date
    Apr 2011
    Posts
    6
    Yeah, I intentionally set it up so that I can keep the array in even if I skip the stealing section, because that array being defined initially seemed to be what causes the segfault. Thus, I wanted to be able to cause the memory to be allocated without actually using the array to see if the allocation, itself was causing the segfault. This didn't prove to be true, so that's just a hold over.

    I've narrowed it down, right now, to the random section, but that didn't seem to matter before... Right now, as long as the random section doesn't run, then there's no segfault, and the other options don't seem to matter.

    I'm going to edit the code above, if it will let me, with the current version of the thread function.


    Oddly enough, I found something that SHOULD have caused a segfault, but doesn't seem to do so. I changed
    NUM_THREADS - 1 - j
    to
    NUM_THREADS - 2 - j
    in the part where I swap an entry in the threads array. In the other places, it seems to be correct
    the original would have caused the threads array to be read one past the end of the array, which I would think would cause a segfault, but it didn't seem to do so. I fixed it in the code above
    Last edited by aaronburro; 04-27-2011 at 02:32 AM.

  6. #6
    Registered User
    Join Date
    Apr 2011
    Posts
    6
    wow, this is getting CRAZY.

    so, I've narrowed it down to the line:
    Code:
    victim = threads[i];
    comment that line out, no segfault. note, however, that there is a line following that where I assign a value to threads[i] that it seems to have no problem with. even better, I can do a line as follows with no segfault:
    Code:
    k = threads[i];
    but, if I then set victim equal to k, segfault. what the deuce?


    OK, addendum, even weirder. if I change the next IF statement to use k instead of victim when I set k = threads[i], it segfaults. so, that IF statement is also tied into it. comment out the if statement, no segfault. if I subtitute "threads[i]" in for victim/k in the IF statement, segfault.

    so, basically, this line is really giving the segfault:
    Code:
    if (victim == INT_MIN) { /* foo */ }

    NOTE: this behaviour ONLY occurs if NUM_THREADS is sufficiently high. on my system, at 1758, it faults. 1757, it doesn't. haven't tried it on any others to see if I can replicate those numbers.
    NOTE NOTE: i tested it on another system, the behaviour is the same, except for the cutoff for NUM_THREADS is different. everything else is the same

    as it stands right now, gdb reports the following for 1758 threads:
    Program received signal SIGSEGV, Segmentation fault.
    [Switching to Thread 0xb6644b70 (LWP 9067)]
    0x0011f07a in ?? () from /lib/ld-linux.so.2
    Last edited by aaronburro; 04-27-2011 at 03:36 AM.

  7. #7
    and the hat of int overfl Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    39,661
    Since we don't know which conditionals you've chosen, post the code which you're testing with without all the #ifdefs all over the place.

    Also, factoring out some of the work into other functions would make doStuff() more readable.
    I mean, you have 2 (or is it 3) nested while(1) loops going on.

    Also, I think you're chasing ghosts.
    Unusual software bug - Wikipedia, the free encyclopedia
    Whilst you may have nailed down a symptom, I don't think you're any nearer understanding a root cause.

    Have you tried using say valgrind on a small number of threads?

    Also (it's my word of the day), your stack size calculation doesn't take into account things like
    - thread infomation
    - function call/return overhead
    - temporary variable overhead
    In short, summing a few sizeof() in one significant stack frame isn't going to give you the total amount needed.
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.

  8. #8
    Registered User
    Join Date
    Apr 2011
    Posts
    6
    i have no clue how to do a valgrind. anyway, I ran a gcc -E on the code, and here's what I got. a LOT of stuff is taken out, but that's what I expected. it corresponds to SKIPPRINT and RANDOM being defined.

    Code:
    void *DoStuff(void* theData) {
    	int loop = 0;
    	int valueToWait = 0;
    	int i, j;
    	int k;
    	int myCount;
    	int val;
    	thread_data * data = (thread_data *) theData;
    	my_deque * deque = data->deque;
    	int victim = 0;
    	int threads[1758 - 1];
    
    	while (1) {
    		valueToWait = 1 - valueToWait;
    		pthread_mutex_lock(&mut);
    		numArrived++;
    		myCount = numArrived;
    
    		while ((allDie == 0) && (canGo != valueToWait)) {
    			pthread_cond_wait(&cond, &mut);
    		}
    
    		pthread_mutex_unlock(&mut);
    
    		if (allDie != 0) {
    			pthread_exit(((void *)0));
    			break;
    		}
    
    		data->writes = 0;
    		data->misses = 0;
    		data->stealAttempts = 0;
    		data->steals = 0;
    
    		while (1) {
    			j = (int)(deque->back);
    			if (j <= 0)
    				break;
    
    			val = (int)(my_deque_pop_bottom(deque));
    
    			if (val == 0) {
    				deque->data[j-1] = data->id;
    				data->writes++;
    			}
    			else if (val == -1) {
    				my_deque_push_bottom(deque, 0);
    			}
    			else if (val == (int)-2) {
    			}
    			else
    				break;
    		}
    
    		{
    			for (j = 0; j < 1758 - 1; j++) {
    				if (j < data->id-1)
    					threads[j] = j;
    				else
    					threads[j] = j+1;
    			}
    
    			for (j = 0; j < 1758 - 1; j++) {
    				i = rand() % (1758 - 1 - j);
    				victim = threads[i];
    				if (victim == -2147483648 - 1) {
    					printf("something's wrong, sherlock!\n");
    					continue;
    				}
    
    				threads[i] = threads[1758 - 2 - j];
    				threads[1758 - 2 - j] = (-2147483647 - 1);
    			}
    		}
    
    		for (i = 0; i < 2048; i++) {
    			if ((deque->data[i] == 0) || (deque->data[i] == -1))
    				data->misses++;
    		}
    
    		loop++;
    	}
    }

    As for figuring out how much stack space is needed, is there a way I can reasonably figure out what's needed by the rest of that stuff? I gave it the extra 100 as a fudge factor, but it sounds like that wasn't enough fudge...

  9. #9
    Registered User
    Join Date
    Nov 2010
    Location
    Long Beach, CA
    Posts
    5,909
    Quote Originally Posted by aaronburro View Post
    i have no clue how to do a valgrind.
    Amazingly powerful tool. Learning curve can seem a bit steep, but it's really not that bad. There's lots of tutorials on the web, including one here. If you plan on doing any hardcore C development, valgrind is invaluable.

    As for figuring out how much stack space is needed, is there a way I can reasonably figure out what's needed by the rest of that stuff? I gave it the extra 100 as a fudge factor, but it sounds like that wasn't enough fudge...
    You would have to do some serious profiling. Valgrind should help with this and there are other tools, I just don't know what they are. I think you may be pre-optimizing. Use the default stack size, and see how many threads you can spin up. You may be able to get enough going and still test your deadlock cases without all this crazy stack size manipulation. If that's insufficient, just overengineer. Take what you think you need and multiply it by 10. That will probably be enough, and still be significantly smaller than your default. If it's still too much, whittle it down until you get the smallest number you can without seg faulting. But be warned, that's a crude method. You would have to test every possible execution path to make sure you wont seg fault from a too-small stack. It should be a very last resort.

  10. #10
    Registered User
    Join Date
    Apr 2011
    Posts
    6
    Well, I changed the fudge factor to include 10KB extra for the stack size, and that seems to have cleared everything up. Yay for messing with stuff that is over my head at this point. Thx for the help, all.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Pthreads help
    By aydinh in forum C Programming
    Replies: 5
    Last Post: 03-02-2011, 09:31 AM
  2. Pthreads...
    By ropitch in forum C++ Programming
    Replies: 2
    Last Post: 07-14-2009, 05:07 AM
  3. pthreads
    By edesign in forum Linux Programming
    Replies: 8
    Last Post: 05-27-2009, 07:39 AM
  4. pthreads for 64 bit m/c
    By vin_pll in forum C Programming
    Replies: 2
    Last Post: 02-22-2009, 10:25 PM
  5. pthreads woes...
    By Gregthatsme in forum C++ Programming
    Replies: 2
    Last Post: 05-26-2003, 07:41 PM