Linux Putting my children to sleep with futex!?
Hey all, I'm not the most seasoned person writing in C however I'd like to think I'm coming along :-) I'm just about finished with a program I've been writing for a little while now that is a basic little daemon. Which basically listens for a connection, forks, and sends the request to a child process then goes back to listening.
In order to ensure that I don't get too many children processes I check a shared piece of memory (which gets incremented by the deamon when it forks, and decremented by the child when it finishes) to make sure I don't go over a certain number of children. After I detect that I already have, for example 5 children running the daemon sends a signal to all of the children to stop what they're doing, decrement the shared memory counter and exit. All the while the daemon is waiting for the children to stop.
Everything works great until I run my little load testing script that basically hammers my daemon with several thousands of requests as fast as it possibly can.
The problem is that after my main big load test there is sometimes a child process that is in a "sleeping" state(and never got to decrement the counter so the daemon waits forever). Doing an strace (following forks) on the daemon during the load test when the child goes to sleep and even after the child is sleeping I find the last run call was a futex FUTEX_WAIT. I'm not using futexes anywhere in my code and am protecting my shared memory with very very basic spinlocks. So it appears my OS is arbitrarily picking one of my children and putting it to sleep by making it call futex.
Has anyone ever heard of this, or know if there is a way to keep it from happening? I've even looked at different gcc options but am not seeing anything that appears to relate to this.
I spent most of yesterday googling and all I found was information about what a futex is and how they work, but nothing about Linux putting certain procs to sleep... something somewhere is trying to be way too helpful. Any ideas?