Signal and exception handling

Hello,

I am running a very computationally expensive program that is parallelised and still runs for many days. It is therefore undesirable for it to stop if a runtime error occurs; precious CPU time on our Linux cluster would be lost. In order to handle abort() and other runtime problems gracefully, I have therefore installed a signal handler that will throw an exception instead of exiting right away. This exception can then be caught and the program can continue without wasting CPU time.

Or so I thought.

This is the problem I have:

I install a signal handler that displays a message, then throw()s an exception. One line of code that is running somewhere calls abort(). This call is within a try statement. The corresponding catch(...) statement, however, is never reached. Instead, the OS calls terminate again, recursively calling the signal handler without ever terminating (until the stack overflows or something). This is the message I get (on stdout/stderr):

Code:

---

NeuroEvolution: Signal 6 (Aborted) received.  Throwing an exception...
terminate called after throwing an instance of 'int'
NeuroEvolution: Signal 6 (Aborted) received.  Throwing an exception...
NeuroEvolution: Signal 6 (Aborted) received.  Throwing an exception...
terminate called recursively
NeuroEvolution: Signal 6 (Aborted) received.  Throwing an exception...
NeuroEvolution: Signal 6 (Aborted) received.  Throwing an exception...
terminate called recursively

---

etc. (you get the picture). The "NeuroEvolution: ..." message is from my signal handler, which looks like this:

Code:

---

2292 void CNeuroEvolution::ThrowingSignalHandler (int signal)
2293 {
2294    cerr << "NeuroEvolution: Signal " << signal
2295          << " (" << strsignal (signal) << ")"
2296          << " received.  Throwing an exception..." << endl;
2297   
2298    throw (signal);
2299 }

---

It was installed using this code:

Code:

---

2302 /// handle signals like ABRT and SEGV by having them throw an exception
2303 void CNeuroEvolution::InstallSignalHandler()
2304 {
2305    struct sigaction signal_handling;
2306   
2307    signal_handling.sa_handler = &ThrowingSignalHandler;
2308
2309    sigfillset(&(signal_handling.sa_mask));
2310    signal_handling.sa_flags = SA_NOCLDSTOP;
2311   
2312    sigaction(SIGILL, &signal_handling, NULL);
2313    sigaction(SIGTRAP, &signal_handling, NULL);
2314    sigaction(SIGABRT, &signal_handling, NULL);
2315    sigaction(SIGFPE, &signal_handling, NULL);
2316    sigaction(SIGSEGV, &signal_handling, NULL);
2317   
2318 }

---

I thought that by using sigfillset etc. I disable any recursive call to the handler? I have also tried sigemptyset at this place with (apparently) no change to the situation.

Here is the catch statement I would like to end up at (the abort occurs in a function called by a function called by a function called by DoSomething):

Code:

---

1682 try
1683 {
1684    // Something that could go wrong
1685    DoSomething(data);
1686 }
1687 catch (...)
1688 {
1689    // something has gone wrong
1690    // avoid a program abort, print a message and do default action
1691
1692    cerr << " NeuroEvolution: Error occured, removing incorrect data... " << endl;
1693
1694    continue;
1695 }

---

This is the relevant part of the stack backtrace:

Code:

---

#32247 0x00002b93bc7ffbbb in PAC::CNeuroEvolution::ThrowingSignalHandler (signal=6) at NeuroEvolution.cpp:2298
#32248 <signal handler called>
#32249 0x00002b93bedb8aa5 in raise () from /lib64/libc.so.6
#32250 0x00002b93bedb9e60 in abort () from /lib64/libc.so.6
#32251 0x00002b93be8afe5b in std::set_unexpected () from /usr/lib64/libstdc++.so.6
#32252 0x00002b93be8af24b in __cxa_bad_cast () from /usr/lib64/libstdc++.so.6
#32253 0x00002b93be8afceb in __gxx_personality_v0 () from /usr/lib64/libstdc++.so.6
#32254 0x00002b93bec84748 in _Unwind_Backtrace () from /lib64/libgcc_s.so.1
#32255 0x00002b93bec848dc in _Unwind_RaiseException () from /lib64/libgcc_s.so.1
#32256 0x00002b93be8aff5d in __cxa_throw () from /usr/lib64/libstdc++.so.6
#32257 0x00002b93bc7ffbbb in PAC::CNeuroEvolution::ThrowingSignalHandler (signal=6) at NeuroEvolution.cpp:2298
#32258 <signal handler called>
#32259 0x00002b93bedb8aa5 in raise () from /lib64/libc.so.6
#32260 0x00002b93bedb9e60 in abort () from /lib64/libc.so.6
#32261 0x00002b93be8afe5b in std::set_unexpected () from /usr/lib64/libstdc++.so.6
#32262 0x00002b93be8af24b in __cxa_bad_cast () from /usr/lib64/libstdc++.so.6
#32263 0x00002b93be8afceb in __gxx_personality_v0 () from /usr/lib64/libstdc++.so.6
#32264 0x00002b93bec84748 in _Unwind_Backtrace () from /lib64/libgcc_s.so.1
#32265 0x00002b93bec848dc in _Unwind_RaiseException () from /lib64/libgcc_s.so.1
#32266 0x00002b93be8aff5d in __cxa_throw () from /usr/lib64/libstdc++.so.6
#32267 0x00002b93bc7ffbbb in PAC::CNeuroEvolution::ThrowingSignalHandler (signal=6) at NeuroEvolution.cpp:2298
#32268 <signal handler called>
#32269 0x00002b93bedb8aa5 in raise () from /lib64/libc.so.6
#32270 0x00002b93bedb9e60 in abort () from /lib64/libc.so.6
#32271 0x00002b93bedb2246 in __assert_fail () from /lib64/libc.so.6
#32272 0x00002b93bc7e7045 in CLinearGenome::SortSubNetworks (this=0x7fffee3e6720, Start=0, End=108,
    Visited=0x7fffee3e6450) at LinearGenome.cpp:1106

---

I have spent a lot of time trying to get it to run and would be very happy if you could point me to a solution.

Please let me know if you need any more details. I am using gcc (GCC) 4.1.0 (SUSE Linux).

Many thanks in advance,

Nils.

PS: Please do not suggest to change the cause for the abort() call, I had that idea myself already :-) Most problems occur in a part of the code I have not written myself and it would be too time-consuming to debug all of that.

Signal handlers must return normally. Because they're abnormally called, you cannot expect the C++ exception mechanism to handle this correctly.

Basically, those two don't go together. Truth is, there is very little you can do in signal handlers.

Dear CornedBee,

Thanks for your reply!

Quote:

---

Originally Posted by CornedBee

Signal handlers must return normally. Because they're abnormally called, you cannot expect the C++ exception mechanism to handle this correctly.

---

How can a signal handler that was called by abort() return? If it does, where would it go? Please elaborate a little bit on the reason so that I can understand it better.

Quote:

---

Originally Posted by CornedBee

Basically, those two don't go together. Truth is, there is very little you can do in signal handlers.

---

So, do you have any other suggestion what I might do in this case?

Many thanks and regards,

Nils.

So in the midst of all this crashing, what makes you think that you're going to get the correct answer (eventually)?

If the s/w is bad enough to cause exceptions (like the ones you're getting), it's definitely bad enough to give you the wrong answers.

Hang on... A call to abort is usually because the code is "horribly confused" and "can't figure out what to do". Surely the RESULT after "ignoring" such a abort() call would be completely unedefined and unpredictable, and just result in further computation of undefined results? Or do you have some way of "figuring out which parts where affected and thus recover some valuable data"?

I'm just sort of expecting the code that calls abort() to not be able to continue (in a meaningful way) where it came from anyways, even if you override the abort itself.

--
Mats

Dear Salem,

Quote:

---

Originally Posted by Salem

So in the midst of all this crashing, what makes you think that you're going to get the correct answer (eventually)?

If the s/w is bad enough to cause exceptions (like the ones you're getting), it's definitely bad enough to give you the wrong answers.

---

That's a good point, thanks for your feedback. However, in this case it really would make sense to continue: To sum it up, I am generating random data structures in a loop, then testing them by using them. If using them generates an error I discard the data structure, then continue the loop looking for a datum that does not create an error.

So, it really does make sense in this special case. The data that causes the problem is discarded and the code works fine with the other data.

Also, as I said earlier, I simply do not have the time to debug the faulty code (which I have not written myself) since it is too complex.

If anybody has an idea how to do it in any other way please let me know.

Dear Mats,

Quote:

---

Originally Posted by matsp

Hang on... A call to abort is usually because the code is "horribly confused" and "can't figure out what to do". Surely the RESULT after "ignoring" such a abort() call would be completely unedefined and unpredictable, and just result in further computation of undefined results? Or do you have some way of "figuring out which parts where affected and thus recover some valuable data"?

---

Exactly that, as I just wrote in my reply above. The problems only occur with specific data -- which, however, cannot easily be tested for this feature except by using it. This, in turn, creates the signal, which I can then use as a measure for the data, and remove it.

Quote:

---

Originally Posted by matsp

I'm just sort of expecting the code that calls abort() to not be able to continue (in a meaningful way) where it came from anyways, even if you override the abort itself.

---

... and this is why I got the idea with the exception! The catch statement knows exactly what to do and is placed at the correct level to continue in a meaningful way.

Ok, obviously me and Salem posted at the same time.

So, presumably this is something like "the calculation of the input values lead to incorrect result [obvious ones are: divide by zero, negative square root, but of course any partial result that is invalid for this particular application]" under some circumstances, and when the (large external) code find this, it calls abort().

Is there many different places that cause this call to abort? Would it be possible to catch this in a different manner?

I'm just dubious that the current approach will actually work right.

--
Mats

This probably isn't possible, or at least not easy, but is there a way to redefine abort() to call your own abort_exception() function?

I've seen some C code that does something like this:

Code:

---

#define malloc  myMalloc

---

I'm guessing you'd have to do that in all the files that call abort() though... So I guess I'm wondering if there's a way to globally re-link all the object files to call your abort() instead of the regular C abort()?

cpjust makes a good point: just replace "abort()" with your own function - and you don't really need a different name, as long as you link that object file before the standard C library, it should take your abort instead of the other one.

--
Mats

Quote:

---

Originally Posted by matsp

cpjust makes a good point: just replace "abort()" with your own function - and you don't really need a different name, as long as you link that object file before the standard C library, it should take your abort instead of the other one.

--
Mats

---

This sounds like an interesting experiment, so I'll probably try it out when I get some time.
Do you know if that would still work if the code that's calling abort() was in a separate .lib file that's being linked or would the .lib file need to be recompiled first?

Also, wouldn't you get Linker errors about abort() already defined in file blah...? I've always found those kinds of errors a pain in the ass to get rid of.

Quote:

---

Originally Posted by cpjust

This probably isn't possible, or at least not easy, but is there a way to redefine abort() to call your own abort_exception() function?

I've seen some C code that does something like this:

Code:

---

#define malloc  myMalloc

---

I'm guessing you'd have to do that in all the files that call abort() though... So I guess I'm wondering if there's a way to globally re-link all the object files to call your abort() instead of the regular C abort()?

---

Dear cpjust,
Dear Mats,

Thanks for your input. Maybe it is possible to do something along those lines. I am unsure whether it is possible to shadow abort since I am not calling it directly. Also, sometimes the code gets SEGV. However, I will try to work in that direction and post the outcome. I was also thinking I could set some flag instead of throwing an exception (if exception handling needs a valid stack to backtrace and the signal handler caller corrupts it), then hope that the code somehow reaches the place where my catch is, and query the flag there instead of catch()ing.

Thanks again for you help, everyone who has replied!

Kind regards,

Nils.

Quote:

---

Originally Posted by CornedBee

Signal handlers must return normally. Because they're abnormally called, you cannot expect the C++ exception mechanism to handle this correctly.

---

There have been some attempts to make it work. None have been 100% successful that I've heard of. The GNU C library allows you to longjmp() out of a signal handler, so it seems theoretically possible to throw an exception from a signal handler, too.

However, any such method would be fraught with difficulty. Another thing to consider is that if the signal handler is processing an ASYNCHRONOUS signal, you have no idea when this signal occurs. You might be in the middle of constructing an object. You might be in the middle of a memory allocation or something. Normal C++ exceptions only occur at well-defined points in execution, but an exception triggered from an asynchronous signal handler violates these rules.

In short, I don't think this is going to work at all. Instead, rely on the tried-and-true method of setting some volatile flag variable inside your signal handler, which you periodically check (probably in some mid-level loop somewhere). If you see that the flag is set, THEN you throw an exception.

Quote:

---

Originally Posted by nts

Thanks for your input. Maybe it is possible to do something along those lines. I am unsure whether it is possible to shadow abort since I am not calling it directly.

---

Nothing should be calling it. I don't know of any C library routines which call abort(). If you're not calling it, nobody is calling it.

Quote:

---

Also, sometimes the code gets SEGV.

---

Under UNIX, the result of ignoring a SIGSEGV signal which was not generated with raise() or kill() is undefined. It will probably lead to an infinite loop. Here's what happens:

1. Code attempts to access bogus memory location
2. Processor signal interrupt, calls SIGSEGV handler
3. Handler returns, CPU goes to the same instruction again
4. Go to step 1

Quote:

---

I was also thinking I could set some flag instead of throwing an exception

---

yes, Yes, YES.

Dear Brewbuck,

Quote:

---

Originally Posted by brewbuck

There have been some attempts to make it work. None have been 100% successful that I've heard of. (...)

---

Thanks, I also googled about that and had still some hope left that I could make it work. I suspect that the exception handling tries something that fails because it relies on a clean state of the stack, like backtracking instead of doing a longjump-like thing to reach the catch?

Quote:

---

Originally Posted by brewbuck

In short, I don't think this is going to work at all. Instead, rely on the tried-and-true method of setting some volatile flag variable inside your signal handler, which you periodically check (probably in some mid-level loop somewhere). If you see that the flag is set, THEN you throw an exception.

---

That's probably the best way, thanks very much.

I will try some more and keep (all of) you posted.

Best regards,

Nils.

Quote:

---

Originally Posted by brewbuck

Nothing should be calling it. I don't know of any C library routines which call abort(). If you're not calling it, nobody is calling it.

---

assert() does. But then assert could also be easily replaced my myassert() which in my code would unfortunately only solve a small part of the problem.

Quote:

---

Originally Posted by brewbuck

Under UNIX, the result of ignoring a SIGSEGV signal which was not generated with raise() or kill() is undefined. It will probably lead to an infinite loop. Here's what happens:

1. Code attempts to access bogus memory location
2. Processor signal interrupt, calls SIGSEGV handler
3. Handler returns, CPU goes to the same instruction again
4. Go to step 1

---

Oh, it goes to the same instruction? That may make sense from the point of view of the OS but here it creates trouble. So, in that case any signal handler that sets a flag should not return, or should it? But if it waits and a different thread throws an exception, our thread will not go to the catch either, or will it?!

Nils.

Quote:

---

Originally Posted by nts

Oh, it goes to the same instruction? That may make sense from the point of view of the OS but here it creates trouble.

---

What other behavior would make any sense? The idea is to trap a bad memory access, do something that hopefully fixes the situation, then try again. If it just moved on to the next instruction, then that memory access acts like it never existed in the first place, which means your program has just become a big mess of "undefined."

Quote:

---

So, in that case any signal handler that sets a flag should not return, or should it? But if it waits and a different thread throws an exception, our thread will not go to the catch either, or will it?!

---

Exceptions only propagate within a single thread. So the exception has to be thrown by the same thread which will ultimately handle it. As for whether the signal handler should return... It must always return. How could it not?

Quote:

---

Originally Posted by brewbuck

What other behavior would make any sense? The idea is to trap a bad memory access, do something that hopefully fixes the situation, then try again. If it just moved on to the next instruction, then that memory access acts like it never existed in the first place, which means your program has just become a big mess of "undefined."

---

Yes, that's what I meant, from an OS point of view it does make sense. From a programmer's point of view retrying that instruction would mean that the source of the problem must be fixed first. Since we are not dealing with a page fault but with something unknown in an unknown (to the signal handler) context it would be better for us to return elsewhere... But that would require contextual information and cannot be guessed by the compiler, I do realise that.

Quote:

---

Originally Posted by brewbuck

Exceptions only propagate within a single thread. So the exception has to be thrown by the same thread which will ultimately handle it. As for whether the signal handler should return... It must always return. How could it not?

---

Well, since the data is apparently faulty we really do not want it to return to the method where the signal occurred in order to avoid an infinite signal/catcher loop. Instead we really want the IP to point to the catch() clause or some other defined position. Maybe with a long jump we could do that, but in each stack state (i.e. method, possibly in each for loop etc?) we would then need to register the address to jump to so that the signal handler would only jump within the "same stack state" location. Sounds like our own try-catch construct, too bad the real one does not do it :-(

OK, I will try something but tomorrow -- right now my head is spinning...

Quote:

---

How can a signal handler that was called by abort() return?

---

Let me reword: they must return normally or not at all.

GCC's exception implementation uses DWARF information, the instruction pointer and a stack deconstruction for unwinding. So yes, the stack definitely needs to be clean.


Can't you write code that checks the datum's correctness (as in, won't cause the lib to crash) before passing it in?

Quote:

---

Originally Posted by nts

OK, I will try something but tomorrow -- right now my head is spinning...

---

You say that the model here is: Test piece of data. If fault caught, discard data and try something else. Why not test each piece of data in its own thread. If the thread crashes, who cares? You don't have to create a new thread for every single piece of data you test. Just make one, go into your loop, and run until you crash, logging your progress all the while. When the crash inevitably occurs, spawn a new thread and start your loop again where you left off.

Quote:

---

Originally Posted by CornedBee

GCC's exception implementation uses DWARF information, the instruction pointer and a stack deconstruction for unwinding. So yes, the stack definitely needs to be clean.

---

I suspected so, looking at the backtrace (posted above). That's unfortunate for us.

Quote:

---

Originally Posted by CornedBee

Can't you write code that checks the datum's correctness (as in, won't cause the lib to crash) before passing it in?

---

The thing is, the datum really is OK but the lib code is not. The lib code will work with most data but in 0.5% of the cases there will be an otherwise valid datum that makes it crash.

Thinking about my possibilities I may try to modify the lib to at least throw exceptions (directly!) instead of doing [things that lead to] aborts, even if I do not have the time to debug it completely.

Thanks again everyone for your input!!!

Kind regards,

Nils.

Quote:

---

Originally Posted by brewbuck

You say that the model here is: Test piece of data. If fault caught, discard data and try something else. Why not test each piece of data in its own thread. If the thread crashes, who cares? You don't have to create a new thread for every single piece of data you test. Just make one, go into your loop, and run until you crash, logging your progress all the while. When the crash inevitably occurs, spawn a new thread and start your loop again where you left off.

---

Brewbuck, you're a genius! That sounds like a great idea. It may even run faster if I parallelise it :-)

Thanks!

Quote:

---

Originally Posted by nts

Brewbuck, you're a genius! That sounds like a great idea. It may even run faster if I parallelise it :-)

Thanks!

---

The way signals work in POSIX threads means you have to do some trickery to make the SIGSEGV actually kill the proper thread. Without some special code it will simply kill the entire process including all its threads. I wrote up a small example which shows how it might work.

Code:

---

#include <signal.h>
#include <pthread.h>
#include <stdio.h>

pthread_t thr;

void do_segv(int x);

void *thread_func(void *arg)
{
  int *null = NULL;
  int x;

  pthread_setcancelstate(PTHREAD_CANCEL_ENABLE, NULL);
  pthread_setcanceltype(PTHREAD_CANCEL_ASYNCHRONOUS, NULL);

  x = *null;
  return NULL;
}

void do_segv(int x)
{
  pthread_cancel(thr);
}

int main()
{
  signal(SIGSEGV, do_segv);

  for(;;)
  {
      pthread_create(&thr, NULL, thread_func, NULL);
      pthread_join(thr, NULL);
      printf("Thread died\n");
  }
}

---

This loops forever, spawning new threads then waiting for them to die. The thread which is spawned immediately causes a segfault by dereferencing a NULL pointer. This triggers a SIGSEGV, which is delivered to do_segv(). do_segv() calls pthread_cancel() to kill the thread.

In order for cancellation to work as we want here, the two calls to pthread_setcancelstate() and pthread_setcanceltype() are necessary. Otherwise delivery of the cancellation will only occur at a "cancellation point." But we want it to die right away.

Note the need for a global variable which holds the thread handle. This is so the signal handler can access it. It doesn't need to be volatile, since the signal handler doesn't change its value.

There are drawbacks. If some other part of your app throws a SIGSEGV, it will kill the wrong thread. You can probably imagine ways to deal with that.

Or don't use threads. Just fork a new process and test the exit code.