Thread: Linux Putting my children to sleep with futex!?

  1. #1
    Registered User
    Join Date
    Feb 2009
    Posts
    10

    Post Linux Putting my children to sleep with futex!?

    Hey all, I'm not the most seasoned person writing in C however I'd like to think I'm coming along :-) I'm just about finished with a program I've been writing for a little while now that is a basic little daemon. Which basically listens for a connection, forks, and sends the request to a child process then goes back to listening.

    In order to ensure that I don't get too many children processes I check a shared piece of memory (which gets incremented by the deamon when it forks, and decremented by the child when it finishes) to make sure I don't go over a certain number of children. After I detect that I already have, for example 5 children running the daemon sends a signal to all of the children to stop what they're doing, decrement the shared memory counter and exit. All the while the daemon is waiting for the children to stop.

    Everything works great until I run my little load testing script that basically hammers my daemon with several thousands of requests as fast as it possibly can.

    The problem is that after my main big load test there is sometimes a child process that is in a "sleeping" state(and never got to decrement the counter so the daemon waits forever). Doing an strace (following forks) on the daemon during the load test when the child goes to sleep and even after the child is sleeping I find the last run call was a futex FUTEX_WAIT. I'm not using futexes anywhere in my code and am protecting my shared memory with very very basic spinlocks. So it appears my OS is arbitrarily picking one of my children and putting it to sleep by making it call futex.

    Has anyone ever heard of this, or know if there is a way to keep it from happening? I've even looked at different gcc options but am not seeing anything that appears to relate to this.

    I spent most of yesterday googling and all I found was information about what a futex is and how they work, but nothing about Linux putting certain procs to sleep... something somewhere is trying to be way too helpful. Any ideas?

  2. #2
    Registered User Codeplug's Avatar
    Join Date
    Mar 2003
    Posts
    4,981
    >> and am protecting my shared memory with very very basic spinlocks.
    Sounds like you're not using proper synchronization. A Posix or SVR4 semaphore would be a natural choice for limiting the number of children.

    If you would like to post a minimal, but complete, example of your forking and shm access, we can point out anything that's "not right".

    gg

  3. #3
    Registered User
    Join Date
    Feb 2009
    Posts
    10
    Sure thing.

    Here is my code snippet that does the checking and and forking etc... My scontext is a struct with two integers.. children and updating.
    Code:
    if (strcasecmp(context.throttle_procs,"on") == 0 && scontext->children+1 > context.int_throttle_procs_max) {
            sprintf(buf,"%s Max processes(%d) reached, sending SIGUSR1 signal to children",context.ident,context.int_throttle_procs_max);
            log_message(context.log_file,buf,'e');                                                                                       
            kill(0,SIGUSR1);                                                                                                             
            while (scontext->children !=0) {                                                                                             
                    usleep(250000);                                                                                                      
                    //Wait for procs to exit before continueing                                                                          
            }                                                                                                                            
    }                                                                                                                                    
    if ((pid=fork()) >= 0) {                                                                                                             
            //**--This is the Child Process--**//                                                                                        
            if (pid == 0) {                                                                                                              
                    cpid=getpid();                                                                                                       
                    sprintf(context.ident,"child[%d]:",cpid);                                                                            
                    context.child=scontext->children+1;                                                                                  
                    dispatcher(client_sockfd,hostname);                                                                                  
                    sprintf(buf,"%s process #%d exiting",context.ident,context.child);                                                   
                    log_message(context.log_file,buf,'d');                                                                               
                    //Protext critical section with simple spinlock//                                                                    
                    while (scontext->updating == TRUE) {                                                                                 
                            //spin waiting for lock                                                                                      
                            sprintf(buf,"%s process #%d waiting for lock.",context.ident,context.child);                                 
                            log_message(context.log_file,buf,'d');                                                                       
                            usleep(200000+context.child);
                    }
                    scontext->updating = TRUE;
                    scontext->children--;
                    //unlock critical section//
                    scontext->updating = FALSE;
                    close(client_sockfd);
                    exit(0);
            }
            //**--This is the parent Process--**//
            else {
                    //Protext critical section with simple spinlock//
                    while (scontext->updating == TRUE) {
                            //spin waiting for lock
                            sprintf(buf,"%s process #%d waiting for lock.",context.ident,context.child);
                            log_message(context.log_file,buf,'d');
                            usleep(199000);
                    }
                    scontext->updating = TRUE;
                    scontext->children++;
                    //unlock critical section//
                    scontext->updating = FALSE;
                    sprintf(buf,"%s Sent to child(%d) process #%d",context.ident,pid,scontext->children);
                    log_message(context.log_file,buf,'d');
                    close(client_sockfd);
            }
    }
    when a SIGUSR1 is recieved my signal handler checks to see if it is the daemone process or not... if if it's not (child) then it run this function:
    Code:
    void sig_handle_force_exit(void) {
            char buf[KB];
            struct shrd_context *scontext;
    
            context.running = FALSE;
            close_all_fd();
            scontext = (struct shrd_context *)get_shrd_context(context.shm_id);
            while (scontext->updating == TRUE) {
                    //spin waiting for lock
                    sprintf(buf,"%s process #%d waiting for lock.",context.ident,context.child);
                    log_message(context.log_file,buf,'d');
                    usleep(250000+context.child);
            }
            scontext->updating = TRUE;
            scontext->children--;
            //unlock critical section//
            scontext->updating = FALSE;
            det_shrd_context(scontext);
            sprintf(buf,"%s process #%d forced exit.",context.ident,context.child);
            log_message(context.log_file,buf,'w');
            exit(-1);
    }
    Thank you so much for the eyes. I know it's probably not the cleanest code to look at, and it will probably make you cringe, but like I said... I'm still pretty green :-) Thanks

  4. #4
    Officially An Architect brewbuck's Avatar
    Join Date
    Mar 2007
    Location
    Portland, OR
    Posts
    7,396
    Quote Originally Posted by Codeplug View Post
    >> and am protecting my shared memory with very very basic spinlocks.
    Sounds like you're not using proper synchronization. A Posix or SVR4 semaphore would be a natural choice for limiting the number of children.
    Well, strictly a spinlock is perfectly fine to protect a shared resource, it just chows down on CPU while it does so.
    Code:
    //try
    //{
    	if (a) do { f( b); } while(1);
    	else   do { f(!b); } while(1);
    //}

  5. #5
    Registered User
    Join Date
    Feb 2009
    Posts
    10
    right, from what I've read since getting in and only incrementing/decrementing a counter is such a small and fast operation a spinlock is ideal as it doesn't require any context switches etc... and can actually be faster than having to put a proc to sleep, wake up, and resume. But that assumes something else doesn't come along and do it for you :-/

  6. #6
    Officially An Architect brewbuck's Avatar
    Join Date
    Mar 2007
    Location
    Portland, OR
    Posts
    7,396
    Futexes are used internally by the C library to implement certain kinds of wait states. They are a Linux-specific feature which are meant to be used as a building block for more friendly synchronization objects.

    I can't see what in your code might be using a futex. But your "spinlocks" are not actual spinlocks, since they are not atomic test-and-set. I imagine you have a simple deadlock caused by not using proper locks.

    A proper spinlock is fairly simple, but I won't show one because it's not what you should be using here. Better options would be pthread mutexes or UNIX semaphores.
    Code:
    //try
    //{
    	if (a) do { f( b); } while(1);
    	else   do { f(!b); } while(1);
    //}

  7. #7
    Registered User
    Join Date
    Feb 2009
    Posts
    10
    brewbuck, thank you for your comments... Since I'm doing everything with fork, I would need to do a considerable rewrite to get this working using threads instead, so I'd like to avoid pthreads. Would you recommend a Unix semaphore then? And can you tell me why you recommend again using a spinlock in this situation? The criteria you're basing this off of would help me to make a better decision in the future.

  8. #8
    Registered User Codeplug's Avatar
    Join Date
    Mar 2003
    Posts
    4,981
    What you have is mult-threaded. Each thread just happens to be in a separate process.

    >> //unlock critical section//
    >> scontext->updating = FALSE;
    This does not provide any kind of meaningful (working) synchronization - even on a single core processor. Under Posix, there are strict rules to how multiple threads can access the same memory location:
    http://www.opengroup.org/onlinepubs/...html#tag_04_10

    You also need to make sure you're calling safe functions within your signal handler:
    http://www.opengroup.org/onlinepubs/...l#tag_02_04_03 (towards the bottom of 2.4.3)

    As for a solution - the easiest thing to do would be to use a single Posix semaphore with a count of 1 as your "critical section" provider - instead of the unsafe 'updating' variable.
    http://www.opengroup.org/onlinepubs/.../sem_init.html
    So to "enter the critical section" you would call sem_wait(). To "leave the critical section" you would call sem_post().

    Now that you have a proper synchronization object, you can use it to protect all "access" to shared memory. If you need to read the value of 'children', you:
    [enter CS]
    [read a copy]
    [exit CS]
    And likewise for increment and decrement.

    Now, if there is shared memory which is read-only (never modified after threads are created), then synchronization isn't necessary - you can read away. All other accesses must use proper synchronization.

    >> context.child=scontext->children+1;
    Keep in mind, that even with proper synchronization there is no guarantee that 'context.child' will be unique per child.

    gg

  9. #9
    Registered User
    Join Date
    Feb 2009
    Posts
    10
    Thank you so much for those links, I'm still going through them. And am going to take a stab at implementing it with a semaphore. I'll post back with my results. Another question though... With my loop that waits for the counter to come back down to 0, because it's not doing any modification of the value does it need to also be protected?

    And yes I know that setting context.child=scotext->children+1 doesn't guarantee it to be unique per child, I had put that in place for more debugging so I could see which pid was running as which child number in my log. That does bring up another question for me though... when I make that assignment, that would also need to be within the CS, to guarantee I get the proper value right?

  10. #10
    Registered User Codeplug's Avatar
    Join Date
    Mar 2003
    Posts
    4,981
    All reads and writes ("accesses") should be protected.

    Instead of polling for 0 in the parent, you could create an additional semaphore, let's call "lastchild", with an initial value of 0. The parent would then sem_wait(lastchild) to wait for all the children to exit. The last child to exit (scontext->children == 1) would then sem_post(lastchild), unblocking the parent, letting the parent know that the last child is going down.

    >> when I make that assignment, that would also need to be within the CS
    Yeah, since that constitutes an access to a shared memory location. You could just use a local variable in the parent to communicate a "child number" to the child.

    gg

  11. #11
    Registered User
    Join Date
    Feb 2009
    Posts
    10
    Well, it's interesting... I'm now using semaphores to protect the shared memory and I'm still getting the same behavior as before where for somereason one of the child processes goes to sleep... this time however using strace, it appears that the last called sys call is
    Code:
    futex(0x7f21e73005d4, FUTEX_WAIT_PRIVATE, 2, NULL
    Instead of the regular FUTEX_WAIT.

    When I attach to the child with gdb and do a backtrace I it looks its running a function that it can't find (which makes sense as it looks like the process is being put to sleep by some other process(maybe OS?)). Here's the output from the gdb backtrace:
    Code:
    GNU gdb (GDB; openSUSE 11.1) 6.8.50.20081120-cvs
    Copyright (C) 2008 Free Software Foundation, Inc.
    License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
    This is free software: you are free to change and redistribute it.           
    There is NO WARRANTY, to the extent permitted by law.  Type "show copying"   
    and "show warranty" for details.                                             
    This GDB was configured as "x86_64-suse-linux".                              
    For bug reporting instructions, please see:                                  
    <http://bugs.opensuse.org/>.                                                 
    Attaching to process 27973                                                   
    Reading symbols from /home/miscem/dev/app_status/app_status...done.          
    Reading symbols from /lib64/librt.so.1...done.                               
    Loaded symbols for /lib64/librt.so.1                                         
    Reading symbols from /lib64/libc.so.6...done.                                
    Loaded symbols for /lib64/libc.so.6                                          
    Reading symbols from /lib64/libpthread.so.0...done.                          
    [Thread debugging using libthread_db enabled]                                
    Loaded symbols for /lib64/libpthread.so.0                                    
    Reading symbols from /lib64/ld-linux-x86-64.so.2...done.                     
    Loaded symbols for /lib64/ld-linux-x86-64.so.2                               
    0x00007f21e7089e6e in ?? () from /lib64/libc.so.6                            
    (gdb) bt                                                                     
    #0  0x00007f21e7089e6e in ?? () from /lib64/libc.so.6                        
    #1  0x00007f21e703e9ed in ?? () from /lib64/libc.so.6                        
    #2  0x00007f21e703e7a6 in ?? () from /lib64/libc.so.6                        
    #3  0x00007f21e703cd00 in ctime_r () from /lib64/libc.so.6                   
    #4  0x00000000004020be in log_message (filename=0x6092cc "/var/log/app_status", message=0x7fffef7282c0 "child[27973]: 10.41.1.187: Recieved SIGUSR1, Max procs reached!?", t=101 'e')
        at includes/cust_utils.h:223                                                                                                                                                     
    #5  0x0000000000404c35 in signal_handler (sig=10) at includes/init_utils.h:561                                                                                                       
    #6  <signal handler called>                                                                                                                                                          
    #7  0x00007f21e703c7b0 in ?? () from /lib64/libc.so.6                                                                                                                                
    #8  0x00007f21e703e832 in ?? () from /lib64/libc.so.6
    #9  0x00007f21e703cd00 in ctime_r () from /lib64/libc.so.6
    #10 0x00000000004020be in log_message (filename=0x6092cc "/var/log/app_status", message=0x7fffef728c50 "child[27973]: 10.41.1.187: dispatcher: request: \"GET /app HTTP/1.0\"", t=100 'd')
        at includes/cust_utils.h:223
    #11 0x0000000000405f20 in dispatcher (clientfd=6, hostname=0x7fffef72a4c0 "10.41.1.187") at includes/http_utils.h:319
    #12 0x0000000000406574 in main (argc=1, argv=0x7fffef72aab8) at app_status.c:151
    (gdb)
    One interesting thing I noticed, is that when I compiled it as a 32bit executable, it didn't have the problem nearly as often. But once it did the gdb backtrace showed this:
    Code:
    GNU gdb (GDB; openSUSE 11.1) 6.8.50.20081120-cvs
    Copyright (C) 2008 Free Software Foundation, Inc.
    License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
    This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
    and "show warranty" for details.
    This GDB was configured as "x86_64-suse-linux".
    For bug reporting instructions, please see:
    <http://bugs.opensuse.org/>.
    Attaching to process 10695
    Reading symbols from /home/miscem/dev/app_status/app_status...done.
    Reading symbols from /lib/librt.so.1...done.
    Loaded symbols for /lib/librt.so.1
    Reading symbols from /lib/libc.so.6...done.
    Loaded symbols for /lib/libc.so.6
    Reading symbols from /lib/libpthread.so.0...done.
    [Thread debugging using libthread_db enabled]
    Loaded symbols for /lib/libpthread.so.0
    Reading symbols from /lib/ld-linux.so.2...done.
    Loaded symbols for /lib/ld-linux.so.2
    0xffffe430 in __kernel_vsyscall ()
    (gdb) bt
    #0  0xffffe430 in __kernel_vsyscall ()
    #1  0xf7e90e93 in ?? () from /lib/libc.so.6
    #2  0xf7e3e44b in ?? () from /lib/libc.so.6
    Backtrace stopped: previous frame identical to this frame (corrupt stack?)
    (gdb)
    Which looks dubious. Also here is my modified code snippets...maybe I missed something or I'm not implementing the semaphores correctly??

    Added this for the creation of the semaphores... I removed the int "updating" from the shared context struct I was using before and added two sem_t types to it called mod_sem, and lc_sem.
    Code:
    //--Create Semaphore(s)--//                                                                
    if (sem_init(&scontext->mod_sem,1,1) == 0) {                                               
            if (sem_init(&scontext->lc_sem,1,0) != 0) {                                        
                    sprintf(buf,"%s sem_init(lc_sem): %s",context.ident,strerror(errno));      
                    log_message(context.log_file,buf,'e');                                     
                    return 1;                                                                  
            }                                                                                  
    }                                                                                          
    else {                                                                                     
            sprintf(buf,"%s sem_init(mod_sem): %s",context.ident,strerror(errno));             
            log_message(context.log_file,buf,'e');                                             
            return 1;                                                                          
    }
    Here is the updated snippet showing the checking for the number of child processes, the fork, increment and decrement.
    Code:
    sem_wait(&scontext->mod_sem); //Enter CS
    cur_children = scontext->children; //Get Copy of val
    sem_post(&scontext->mod_sem); //Leave CS            
    if (strcasecmp(context.throttle_procs,"on") == 0 && cur_children+1 > context.int_throttle_procs_max) {
            sprintf(buf,"%s Max processes(%d) reached, sending SIGUSR1 signal to children",context.ident,context.int_throttle_procs_max);
            log_message(context.log_file,buf,'e');                                                                                       
            kill(0,SIGUSR1);                                                                                                             
            sem_wait(&scontext->lc_sem); //wait for last child to signal no more children                                                
    }                                                                                                                                    
    if ((pid=fork()) >= 0) {                                                                                                             
            //**--This is the Child Process--**//                                                                                        
            if (pid == 0) {                                                                                                              
                    cpid=getpid();                                                                                                       
                    sprintf(context.ident,"child[%d]:",cpid);                                                                            
                    sem_wait(&scontext->mod_sem); //Enter CS                                                                             
                    context.child=scontext->children+1;                                                                                  
                    sem_post(&scontext->mod_sem); //Leave CS                                                                             
                    dispatcher(client_sockfd,hostname);                                                                                  
                    sprintf(buf,"%s process #%d exiting",context.ident,context.child);                                                   
                    log_message(context.log_file,buf,'d');                                                                               
                    sem_wait(&scontext->mod_sem); //Enter CS                                                                             
                    scontext->children--;                                                                                                
                    sem_post(&scontext->mod_sem); //Leave CS                                                                             
                    det_shrd_context(scontext);                                                                                          
                    close(client_sockfd);                                                                                                
                    exit(0);                                                                                                             
            }                                                                                                                            
            //**--This is the parent Process--**//                                                                                       
            else {                                                                                                                       
                    sem_wait(&scontext->mod_sem); //Enter CS                                                                             
                    scontext->children++;                                                                                                
                    sprintf(buf,"%s Sent to child(%d) process #%d",context.ident,pid,scontext->children);                                
                    sem_post(&scontext->mod_sem); //Leave CS                                                                             
                    log_message(context.log_file,buf,'d');                                                                               
                    close(client_sockfd);                                                                                                
            }                                                                                                                            
    }
    And here is the function that gets called by the children on a SIGUSR1
    Code:
    void sig_handle_force_exit(void) {
            char buf[KB];             
            struct shrd_context *scontext;
    
            context.running = FALSE;
            close_all_fd();         
            scontext = (struct shrd_context *)get_shrd_context(context.shm_id);
            sem_wait(&scontext->mod_sem); //Enter CS                           
            scontext->children--;                                              
            if (scontext->children == 0 ) {                                    
                    sprintf(buf,"%s I'm the last child, signaling....",context.ident);
                    log_message(context.log_file,buf,'d');                            
                    sem_post(&scontext->lc_sem); //Signal daemon waiting for last child
            }                                                                          
            //unlock critical section//                                                
            sem_post(&scontext->mod_sem); //Leave CS                                   
            det_shrd_context(scontext);                                                
            sprintf(buf,"%s process #%d forced exit.",context.ident,context.child);    
            log_message(context.log_file,buf,'w');                                     
            exit(-1);                                                                  
    }
    Thanks again for taking time to help me as I learn these things. I've already learned a ton from whats been posted already.

  12. #12
    Officially An Architect brewbuck's Avatar
    Join Date
    Mar 2007
    Location
    Portland, OR
    Posts
    7,396
    Quote Originally Posted by Abs View Post
    brewbuck, thank you for your comments... Since I'm doing everything with fork, I would need to do a considerable rewrite to get this working using threads instead, so I'd like to avoid pthreads. Would you recommend a Unix semaphore then? And can you tell me why you recommend again using a spinlock in this situation? The criteria you're basing this off of would help me to make a better decision in the future.
    On Linux at least, pthread mutexes can be shared between processes. You have to initialize the mutex within a shared memory segment (something you already have), and set the attributes with:

    Code:
    pthread_mutexattr_setpshared(&attr, PTHREAD_PROCESS_SHARED);
    As far as I know, you can't currently do this with condition variables, only mutexes.
    Code:
    //try
    //{
    	if (a) do { f( b); } while(1);
    	else   do { f(!b); } while(1);
    //}

  13. #13
    Registered User Codeplug's Avatar
    Join Date
    Mar 2003
    Posts
    4,981
    You're still trying to do way too much in your signal handler. This easiest (and safest) thing to do is to boil down your signal handler to one line:
    > g_time_to_die = 1;
    Where 'g_time_to_die' is of type "volatile sig_atomic_t". Then dispatcher() just needs to poll it ever so often.

    Another option is to use a different IPC mechanism for communicating "time to die" to your children. If dispatcher() spends most of its time in a poll() or select(), then an additional socket or a pipe could be added to the fd_set/pollfd so that when the parent writes to it, the children know to die. This approach could be expanded to send other "commands" to all or individual children.

    gg

  14. #14
    Registered User
    Join Date
    Feb 2009
    Posts
    10
    Is there an easy way to share a volatile sig_atomic_t between processes? If I have to add it to my shared memory segment then I would need to still attach to the shared memory and update the value. And those functions aren't listed as async safe functions to use in a signal handler... which may be part of my problem in the first place... I think you're right on, that the problem lies in my signal handler.

    For anyone else reading I also ran across these two links that were very informative for me, complete with examples:

    https://www.securecoding.cert.org/co...ignal+handlers

    https://www.securecoding.cert.org/co...ignal+handlers

  15. #15
    Registered User
    Join Date
    Feb 2009
    Posts
    10
    Nevermind, I just realized I don't need to share that between processes as the signal handler only cares about the current running process. Maybe I'll forget the whole decrement forced exit...so that once the SIGUSR1 is sent to the children, then just call _exit or maybe abort (both of which are async safe. Then set the counter back to 0 and assume that the children exited.

    I don't know if I really want to use poll or select because I want them to actually get interrupted in whatever they're doing to just go away.

    Does anyone forsee any problem with this or am I doing yet another dumb thing?

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Which distribution of Linux should I get?
    By joshdick in forum A Brief History of Cprogramming.com
    Replies: 50
    Last Post: 01-19-2003, 09:26 AM
  2. Linux for Windows!
    By Strut in forum Linux Programming
    Replies: 2
    Last Post: 12-25-2002, 11:36 AM
  3. Linux? Windows Xp?
    By VooDoo in forum Linux Programming
    Replies: 15
    Last Post: 07-31-2002, 08:18 AM
  4. Linux Under Windows
    By Strut in forum Linux Programming
    Replies: 3
    Last Post: 05-27-2002, 08:09 PM
  5. linux vs linux?
    By Dreamerv3 in forum Linux Programming
    Replies: 5
    Last Post: 01-22-2002, 09:39 AM

Tags for this Thread