Thread: Problem with asynchronous read/write

Hi,

I have discovered that a certain feature of aio in Linux does not work as I expect. From what I understand, it is possible to start read and then write asynchronous operations, and expect them to complete in ANY order. For example, suppose that when a client connects I want to send it a greeting as well as start listening for any data asynchronously. If I start async write and then async read everything works as expected. However, if I start async read and then async write, the write operation does not proceed until the read operation completes. I do not understand why this happens. If a client does not send any data, I still should expect the write operation to complete, despite that it was started second. Even though there is no data available from the client, there still should be possibility to send it something.

Below I have a sample program to test what I mean. The program listens on port 7777. When connection arrives it starts async read and then initiates async write to send "HELLO" greetings to the client. When I use telnet to connect to this, I expect to see "HELLO" right after the connection completes, even though I haven't sent any data. However, this is not what happens. I have to send some data to the server and only then does the write operation proceeds and I receive my long awaited "HELLO". If I reverse the two IO operations (write followed by a read), everything works as expected. I tested this on Debian and Suse linux distributions with the same result (kernel 2.6.18 ).

Please, if somebody could clarify to me why this happens and how I should go about it, I would be extremely grateful!
Thank you in advance.

Code:

#include <iostream>
using namespace std;

#include <aio.h>
#include <unistd.h>
#include <string.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <errno.h>


int main()
{
  sockaddr_in addr;
  memset(&addr, 0, sizeof(addr));
  addr.sin_port = htons(7777);
  addr.sin_family = AF_INET;
  addr.sin_addr.s_addr = INADDR_ANY;
  int fd = socket(AF_INET, SOCK_STREAM, 0);
  assert(fd > 0);

  int opt = 1;
  assert(setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt)) == 0);
  assert(bind(fd, (struct sockaddr*)&addr, sizeof(addr)) == 0);
  assert(listen(fd, 5) == 0);
  int cfd = accept(fd, NULL, 0);
  assert(cfd > 0);
  //  READING:
  aiocb read_cb;
  char readBuf[1024];
  memset(&read_cb, 0, sizeof(read_cb));
  read_cb.aio_fildes = cfd;
  read_cb.aio_offset = 0;
  read_cb.aio_buf = readBuf;
  read_cb.aio_nbytes = 1023;
  // start asynch read first
  assert(aio_read(&read_cb) == 0);
  //  WRITING:
  aiocb write_cb;
  char strToSend[] = "HELLO\n";
  size_t strLen = strlen(strToSend)+1;
  memset(&write_cb, 0, sizeof(write_cb));
  write_cb.aio_fildes = cfd;
  write_cb.aio_offset = 0;
  write_cb.aio_buf = strToSend;
  write_cb.aio_nbytes = strLen;
  // start asynch write second
  assert(aio_write(&write_cb) == 0);

  size_t const numIO = 2;
  aiocb* cba[numIO] = {0};
  cba[0] = &write_cb;
  cba[1] = &read_cb;

  cout <<"Start waiting for IO completion" <<endl;
  for (size_t done=0; done < numIO; ) {
    //int ret = aio_suspend(cba, numIO, NULL);  // uncommenting this does not help
    //assert(ret == 0);

    // look for completed io:
    done = 0;
    for (size_t c=0; c < numIO; ++c) {
      aiocb *currIO = cba[c];
      if (currIO == NULL) {
        ++done;
        continue;
      }
      int errorIO = aio_error(currIO);
      // IO has completed:
      if (errorIO == 0) {
        ssize_t retIO = aio_return(currIO);
        if (currIO == &write_cb) {
          cout <<"WRITE COMPLETED: ret=" <<retIO <<endl;
        }
        else if (currIO == &read_cb) {
          readBuf[retIO] = 0;
          cout <<"READ COMPLETED : ret=" <<retIO <<" buf=" <<readBuf <<endl;
        }
        else {
          assert(0);
        }
        cba[c] = NULL;
        ++done;
      }
      else 
        assert(errorIO == EINPROGRESS); // don't deal with other errors
    }
  }
  close(cfd);

  close(fd);
  return 0;
}

(To compile this you have to link against librt: 'g++ -Wall -o server server.cpp -lrt')

And thus the read operation indefinitely blocks the write operation.

I've come across the same problem when porting an ACE Proactor application from VxWorks/Windows to Linux.

Attached is a sample program that tries to use a non-blocking socket. It still doesn't work though, busy-looping with EAGAIN/EWOULDBLOCK status from the read operation.

I'd appreciate help if you were able to solve this full-duplex socket problem without resorting to something different than AIO.

P.S. The attached program has the same behavior on Centos5.5, WindRiver Linux 1.4 and WindRiver Linux 3.0.