Thread: ICP using shared memory - how to sync child processes?

  1. #1
    Registered User
    Join Date
    Jun 2009
    Posts
    101

    ICP using shared memory - how to sync child processes?

    I wrote a small app that uses shared memory (using shm_open and mmap). It launches say, four child processes (via four threads) that all look at a shared memory location for data. The way it is now, the data is just a line of characters. I need to have each child process write data to disk, but it has to be in a specific order. Since all four are launched simultaneously, I don't know which will finish its task first. With threads, this isn't a problem as I just use pthread_join() to block until the threads are finished and perform the write in the main thread. But with child processes, there is no such function that I know of. My thought was to set up another shared memory location that holds an int. That int would be used to send control messages between processes.

    Here is my code so far:
    Code:
    #define MAXCHRS 255
    
    char mempath[] = "/tmp/shmem0";
    int fd;
    char *membuffer;
    
    fd = shm_open(mempath, O_CREAT | O_RDWR, S_IRUSR | S_IWUSR); 
    membuffer = (char*)mmap(0, MAXCHRS, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
    fd(close);
    The code works fine passing characters to/from the shared memory location, but I couldn't get it to work by just passing one int.

  2. #2
    and the hat of int overfl Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    39,660
    What did you try?

    Did you try
    int *membuffer;
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.

  3. #3
    Registered User
    Join Date
    Jun 2009
    Posts
    101
    Quote Originally Posted by Salem View Post
    What did you try?

    Did you try
    int *membuffer;
    Here's the code from the host process:
    Code:
    #define CTRL_PATH = "/tmp/control"
    
    shm_unlink(CTRL_PATH); //in case of previous crash
    int fd_ctl = shm_open(CTRL_PATH, O_CREAT | O_RDWR, S_IRUSR | S_IWUSR);		
    ftruncate(fd_ctl, sizeof(int));
    int *control_code_ptr = (int*)mmap(0, sizeof(int), PROT_READ | PROT_WRITE, MAP_SHARED, fd_ctl, 0);
    close(fd_ctl);
    int ctrl_code;
    control_code_ptr = &ctrl_code;
    ctrl_code = 123;
    fprintf(stderr, "Control code set to %d\n",*control_code_ptr); //works fine
    And the code from the receiving process:
    Code:
    int fd_ctl = shm_open(CTRL_PATH, O_RDWR,  S_IRUSR | S_IWUSR);
    int *control_code_ptr = (int*)mmap(0, sizeof(int), PROT_READ | PROT_WRITE, MAP_SHARED, fd_ctl, 0);
    close(fd_ctl);
    
    fprintf(stderr, "Control code received was %d\n",*control_code_ptr); //always returns 0
    The control code I get is always 0. However, if I set the code from within the child process, that works. That doesn't help me though with communicating between processes -- each process needs to be able to read and write values read/written from other processes.

  4. #4
    spurious conceit MK27's Avatar
    Join Date
    Jul 2008
    Location
    segmentation fault
    Posts
    8,300
    It's not clear to me why you would need to use shared memory between threads (and nb, threads are not "child processes"), since threads share memory anyway by their very nature.

    But if you are, I would think you would still need to use a mutex to lock access, because otherwise there is the possibility that one thread will (eg) read half of an int (2 bytes), then another thread completes a write on the last two bytes, then the other thread reads those, and receives an number that was never written by any thread.* Instead, it's two bytes from one and two bytes from another. Whoops!

    So without the mutex you will not be able to get consistent results. Not to say that is the explanation for your problem, but it would be a precursor to dealing with it.

    *shmem may deal with this issue inherently (I've never used it) but you need to find out one way or another if it does, and if it does so only between processes (which threads, again, are not processes), because doing so between processes does not mean there is any guard against a race condition within a single process, which if you have four threads in a single process, it is still a single process, not a parent with four children. A little bit of googling implied to me that mutexes are necessary in this case.
    Last edited by MK27; 06-12-2011 at 07:09 AM.
    C programming resources:
    GNU C Function and Macro Index -- glibc reference manual
    The C Book -- nice online learner guide
    Current ISO draft standard
    CCAN -- new CPAN like open source library repository
    3 (different) GNU debugger tutorials: #1 -- #2 -- #3
    cpwiki -- our wiki on sourceforge

  5. #5
    and the hat of int overfl Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    39,660
    close(fd_ctl);
    int ctrl_code;
    control_code_ptr = &ctrl_code;

    1. You close your shared memory fd
    2. You trash the pointer you set up
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.

  6. #6
    Registered User
    Join Date
    Jun 2009
    Posts
    101
    Quote Originally Posted by Salem View Post
    close(fd_ctl);
    int ctrl_code;
    control_code_ptr = &ctrl_code;

    1. You close your shared memory fd
    2. You trash the pointer you set up
    Yeah I thought that was the problem, but it actually doesn't affect the shared memory area at all, ostensibly because the "file" has already been written and can be accessed by path (/tmp/control).

    Anyway, I think I figured it out. Instead of setting my control code like this:
    Code:
    int ctrl_code;
    control_code_ptr = &ctrl_code;
    ctrl_code = 123;
    I did this instead:
    Code:
    int ctrl_code = 123;
    memcpy(control_code_ptr,&ctrl_code,sizeof(int));
    That works now!

  7. #7
    Registered User
    Join Date
    Jun 2009
    Posts
    101
    Looks like I spoke too soon -- if I set gcc's optimization level to any value (-O1 - -Os), I can't receive the control codes via shared memory. They always return 0 in the receiving process. By the way, I am on gcc 4.2 compiling for x86_64.

  8. #8
    and the hat of int overfl Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    39,660
    The optimiser is most likely removing code which is either unreachable, or has no observable effect.

    For example, just writing to a memory location and never reading from it.

    Consider something like
    Code:
    volatile int *control_code_ptr;
    ...
    // and in place of the memcpy
    *control_code_ptr = ctrl_code;
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.

  9. #9
    Registered User
    Join Date
    Jun 2009
    Posts
    101
    Quote Originally Posted by Salem View Post
    The optimiser is most likely removing code which is either unreachable, or has no observable effect.

    For example, just writing to a memory location and never reading from it.

    Consider something like
    Code:
    volatile int *control_code_ptr;
    ...
    // and in place of the memcpy
    *control_code_ptr = ctrl_code;
    Thanks. Adding volatile to the var fixed it!

  10. #10
    Registered User Codeplug's Avatar
    Join Date
    Mar 2003
    Posts
    4,981
    >> Adding volatile to the var fixed it!
    Nope, it's still incorrect. volatile has nothing to do with synchronization.
    The correct way is to use synchronization primitives provided by your OS or threading library.

    gg

  11. #11
    Registered User
    Join Date
    Jun 2009
    Posts
    101
    Quote Originally Posted by Codeplug View Post
    >> Adding volatile to the var fixed it!
    Nope, it's still incorrect. volatile has nothing to do with synchronization.
    The correct way is to use synchronization primitives provided by your OS or threading library.
    Is it not true that by NOT using volatile, the optimizer makes assumptions about whether certain things are needed or not? The buffers I am using are shared with other processes, so there is no way the optimizer would know that the data would change in these buffers because they are changed by other processes. That's why it optimizes properly now using "volatile" for those buffers' pointers. Does that not make sense?

  12. #12
    Registered User
    Join Date
    Apr 2006
    Posts
    2,149
    Volitile only protects against some errors that occur with sharing variables in memory, not all. It does not prevent race conditions. It was never designed to provide synchronization. It was designed to interface with memory mapped hardware.
    Last edited by King Mir; 06-14-2011 at 08:33 AM.
    It is too clear and so it is hard to see.
    A dunce once searched for fire with a lighted lantern.
    Had he known what fire was,
    He could have cooked his rice much sooner.

  13. #13
    Registered User
    Join Date
    Jun 2009
    Posts
    101
    Quote Originally Posted by King Mir View Post
    Volitile only protects against some errors that occur with sharing variables in memory, not all. It does not prevent race conditions. It was never designed to provide synchronization. It was designed to interface with memory mapped hardware.
    Yes, I am doing some simple syncing with calls to pthread_join(). But, this is only to sync threads that launch each child process. The processes are synchronized by passing values between them via shared memory and my own (simple) implementation. The issue was that the optimizer was messing up parts of the code that relied on this synch variable (the one updated from child processes). Since it was being written to by processes other than the main process, I think the optimizer was making incorrect assumptions. Only compiling with -O0 made the program work. Using "volatile" fixed it so I could optimize at levels higher than 0.

  14. #14
    spurious conceit MK27's Avatar
    Join Date
    Jul 2008
    Location
    segmentation fault
    Posts
    8,300
    Quote Originally Posted by synthetix View Post
    Yes, I am doing some simple syncing with calls to pthread_join(). But, this is only to sync threads that launch each child process. The processes are synchronized by passing values between them via shared memory and my own (simple) implementation. The issue was that the optimizer was messing up parts of the code that relied on this synch variable (the one updated from child processes). Since it was being written to by processes other than the main process, I think the optimizer was making incorrect assumptions. Only compiling with -O0 made the program work. Using "volatile" fixed it so I could optimize at levels higher than 0.
    As Codeplug, and I, have already pointed out, you only have ONE choice here:

    USE MUTEX LOCKS. Any other form of voodoo -- such as a home brewed sync variable -- will fail you, if not right away then later. You cannot have a variable readable by any thread without a lock, if you want it to be reliably readable. It doesn't matter where you put it. Sans that, do not pass go.

    If you are too lazy to do this, do not keep asking for hack answers to a problem that already has a very normative, clearly defined one that you cannot be bothered to apply.
    Last edited by MK27; 06-14-2011 at 10:28 AM.
    C programming resources:
    GNU C Function and Macro Index -- glibc reference manual
    The C Book -- nice online learner guide
    Current ISO draft standard
    CCAN -- new CPAN like open source library repository
    3 (different) GNU debugger tutorials: #1 -- #2 -- #3
    cpwiki -- our wiki on sourceforge

  15. #15
    Registered User
    Join Date
    Apr 2006
    Posts
    2,149
    Quote Originally Posted by synthetix View Post
    Yes, I am doing some simple syncing with calls to pthread_join(). But, this is only to sync threads that launch each child process. The processes are synchronized by passing values between them via shared memory and my own (simple) implementation. The issue was that the optimizer was messing up parts of the code that relied on this synch variable (the one updated from child processes). Since it was being written to by processes other than the main process, I think the optimizer was making incorrect assumptions. Only compiling with -O0 made the program work. Using "volatile" fixed it so I could optimize at levels higher than 0.
    All you've done is turned a repeatable error into a sporadic, hard to track down error.
    It is too clear and so it is hard to see.
    A dunce once searched for fire with a lighted lantern.
    Had he known what fire was,
    He could have cooked his rice much sooner.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. multiple processes accessing same shared library
    By yogeeshgv in forum Linux Programming
    Replies: 1
    Last Post: 07-28-2010, 02:18 PM
  2. accessing shared memory via forked processes
    By rklockow in forum C Programming
    Replies: 7
    Last Post: 06-30-2010, 05:44 PM
  3. Replies: 7
    Last Post: 02-06-2009, 12:27 PM
  4. sockets and child processes
    By Elkvis in forum Linux Programming
    Replies: 2
    Last Post: 03-06-2008, 04:03 PM
  5. shared libraries, datasegment and multiple processes
    By ashim_k1 in forum Linux Programming
    Replies: 1
    Last Post: 02-28-2008, 02:23 PM