Thread: Shared-memory synchronization issues

What is the best way to ensure synchronization and memory consistency in the following scenario:
- Two processes are running on a multi-core machine
- The machine also has an RDMA-capable network (InifniBand)
- The two processes have read/write access to a window of memory in shared-memory. This window is also registered with the network interface. So, the data in the shared-memory region can be changed by either of the processes on the node and by the network.

Suppose, P1 is running on core0. P2 is running on core1.

P2 initializes its shared-memory window by doing a
memset(.., 1, size); P1 writes 0's into this shared-memory window through the network interface (RDMA, loop-back). I get the required network completion-event and as far as the network is concerned, the 0's have been written into the shared-memory. When P2 tries to read this memory region, it should now read "0", instead of "1". But, I am seeing a race condition here and this is not always the case.

I have tried the following cases:
- pthread-mutex:
1. I have a pthread_mutex in this shared-memory window, initialized with the attribute : PTHREAD_PROCESS_SHARED.
2. P1 locks the mutex and does the network operation. When process P0 gets the network completion event, it releases the lock.
3. P2 then locks the mutex and reads the data from the shared-memory

- memory barriers:
I have tried inserting the following memory barriers, just after P2 acquires the lock and just before it reads the data. It still doesnt seem to help.

__asm__ __volatile__ ( "lfence" ::: "memory" );
__asm__ __volatile__ ( "sfence" ::: "memory" );
__asm__ __volatile__ ( "mfence" ::: "memory" );
__asm__ __volatile__ ( "sfence" ::: "memory" );
__sync_synchronize();


Any suggestions as to how such a situation can be handled to ensure correctness? I am using GNU compilers, and the compute node is an Intel Westmere machine.

Thanks,
--K

Hi Ledow,
The write is happening. In some cases, about 100 iterations complete correctly, with P2 seeing the right data in the shared-memory region. Also, if I force P2 to sleep when it detects the error, and if I attach to P2 through gdb to check its data, it is correct. But, the data was not consistent during the execution, when P2 was checking its buffer. I am yet to try a simple memcopy(). I can do that and post an update.
I missed one point earlier, the shared-memory window is basically a file that I have opened in /dev/shm:

fd = shm_open(shm_file, O_CREAT | O_RDWR | O_EXCL, S_IRWXU);

The two processes mmap this file in the following manner;

mem_ptr = mmap(0, shm_file_size, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_POPULATE | MAP_LOCKED,
fd, 0);



Thanks for your help,
--K

Hi Codeplug, I am using InfiniBand verbs API. So, the code involves calls like ibv_reg_mr() for registering/pinning the memory. ibv_post_send() to do the RDMA transfers. With RDMA, the accepted way for a process to check if it has received the data on a pending receive operation is to poll the memory. However, the twist in my case is that the memory is actually an mmaped() file in /dev/shm. I even tried doing a msync() to force the data to be flushed back to the file in /dev/shm, so that P2 can read the right data. It doesnt seem to help much.

Ledow: I am yet to try just doing a simple localcopy between the memory regions. Thats on my todo list. It may help to isolate the cause of this issue.