Quick Winsock question

[Hunter2]

Hey, just a quick question. When a client send()'s data to a server, and the server calls select(), will the server know data is waiting to be received until the entire message has been uploaded to a service provider and the whole thing is ready to be downloaded or something? Will the server, if it calls receive(), have to receive it say, 1 byte per second if that's how fast the client is uploading it, or will the client upload the whole thing at 1 byte per second then the server download it at say 150k per second?

[Hunter2]

The select() call will return when data is available to be read from the socket, regardless of how much there is.

Hmm, maybe I didn't give enough info. I know this much, but in my implementation I call select() with a timeout of 0 in a "hasData()" function, so that if no data was sent by a client, the server can attend to others.. but say I called: send("Hello"). If the socket is blocking (BTW, are all TCP sockets blocking by default or something?), then the server's supposed to receive the whole message with one receive() call, right? But if the client has a really bad internet connection, will the server have to receive the message at a slow rate?

[johnnie2]

Originally posted by Hunter2
(BTW, are all TCP sockets blocking by default or something?)

Yes.

If the socket is blocking, then the server's supposed to receive the whole message with one receive() call, right?

Not necessarily. The server is not guaranteed to receive the entire string in one call to recv(); it may take up to five server-side invocations (not including the NULL terminator) to secure that message depending on connection status.

But if the client has a really bad internet connection, will the server have to receive the message at a slow rate?

The server hardware will attempt to receive the client's data at its normal speed of 150 Kb/s. However, if the client is uploading at 1 Kb/s, the server will have no choice but to receive data at 1 Kb/s because the data is simply not available at a faster rate.

[Hunter2]

However, if the client is uploading at 1 Kb/s, the server will have no choice but to receive data at 1 Kb/s because the data is simply not available at a faster rate.

Ahh shoot, so select() won't wait for all the data to be available before letting you know, huh?

it may take up to five server-side invocations (not including the NULL terminator) to secure that message depending on connection status.

Hmm, so how would I implement this? Would I do something like:
-loop until at least 4 bytes have been read
-convert the 1st 4 bytes into an unsigned long, the size of the msg
-loop until I have received (size) bytes

Your TCP implementation will try to transmit more than one byte at time though, nevertheless the cost of a system call every time is not acceptable.

So, would it be much better to lump all possible data together into 1 message and send that all at once, then parse it on the server, or would the performance difference be not worth the trouble of doing so?

[Hammer]

>>better to lump all possible data together into 1 message
Yes, do as few system calls as possible by sending chunks of data.

>>so how would I implement this?
There are a few ways, revolving around message length indicators and message terminators.

For example:

At the start of each logical message, put a length indicator. Something like:
>005Hello005World
This contains two logical messages. The length of the length field is ALWAYS 3 bytes, so you read that many first, determine the actual message length, then read that many bytes. Of course, you might need to do more than one read to get that many.

OR

Use a terminating character, like \0:
>Hello\0World\0
This is two messages again, each terminated with a \0. Read in a chunck of data (arbiturary size), and parse the buffer for the \0.

Here is some sample source for reading:

/* readn - read exactly n bytes */
int readn(SOCKET fd, char *bp, size_t len)
{
  int cnt;
  int rc;

  cnt = len;
  while (cnt > 0)
  {
    rc = recv(fd, bp, cnt, 0);

    if (rc < 0)           /* read error? */
    {
      if (errno == EINTR) /* interrupted? */
        continue;         /* restart the read */
      return(-1);         /* return error */
    }

    if (rc == 0)          /* EOF? */
      return(len - cnt);  /* return short count */
    bp += rc;
    cnt -= rc;
  }

  return(len);
}

/* readline - read a newline terminated record */
int readline(SOCKET fd, char *bufptr, size_t len)
{
  char        *bufx = bufptr;
  static char *bp;
  static int  cnt = 0;
  static char b[1500];
  char        c;

  while (--len > 0)
  {
    if (--cnt <= 0)
    {
      cnt = recv(fd, b, sizeof(b), 0);
      if (cnt < 0)
      {
        if (errno == EINTR)
        {
          len++;  /* the while will decrement */
          continue;
        }

        return(-1);
      }

      if (cnt == 0) return(0);
      bp = b;
    }

    c = *bp++;
    *bufptr++ = c;
    if (c == '\n')
    {
      *bufptr = '\0';
      return(bufptr - bufx);
    }
  }

  set_errno(EMSGSIZE);
  return(-1);
}

/* readcrlf - read a CR/LF terminated line */
int readcrlf(SOCKET s, char *buf, size_t len)
{
  char  *bufx = buf;
  int   rc;
  char  c;
  char  lastc = 0;

  while (len > 0)
  {
    if ((rc = recv(s, &c, 1, 0)) != 1)
    {
      /*
			 *  If we were interrupted, keep going,
			 *  otherwise, return EOF or the error.
			 */
      if (rc < 0 && errno == EINTR) continue;
      return(rc);
    }

    if (c == '\n')
    {
      if (lastc == '\r') buf--;
      *buf = '\0';  /* don't include <CR><LF> */
      return(buf - bufx);
    }

    *buf++ = c;
    lastc = c;
    len--;
  }

  set_errno(EMSGSIZE);
  return(-1);
}

These samples are from this book by Jon Snader

[novacain]

the other thing is that the client could be faster than the server and so you get 1.5 msg's in one recv(). Then have to process one msg then wait until the rest of the second msg arrives.

If sending multiple data types (or have multiple clients) I suggest a packet header struct and if using large data compression with something like zlib.

The packet header struct would give you a standard size to recv() before each data chunk. It would contain details of the data chunk inc sender, size of items, number of items, total compressed size, type or error codes, CRC checks, time out if data does not all arrive, ect. This struc can then be passed to help data processing.

[Hunter2]

the other thing is that the client could be faster than the server and so you get 1.5 msg's in one recv().

Oh, right. I never really thought about that one But if I only recv() say 4 bytes, for the size of the message, then recv() only the number of bytes contained in the message, this should be a non-issue, right? Or would the double-recv be very costly?

[Hammer]

The last app I wrote used the double-recv approach. You could also read a buffer full of data, and process it in memory, which makes the reading easier, but you have to worry about the buffer contents and handle it correctly.

[Hunter2]

Ok, thanks alot for the help

[Hunter2]

Oh yeah, I forgot one thing. I'm trying to make a file transfer program, and earlier on I've managed to put something together that will read/send a file and the server will receive/save the file. But since files are often pretty big (i.e. 30-150 mb for an installer), I had it broken down into smaller bits and sent a whole bunch of messages instead of just 1 big one, for a number of reasons involving max message sizes and the amount of available RAM and other stuff like that. But I wasn't sure about what size the messages should be, since the "max" returned by getsockopt() was something like 4gb and the compiler wouldn't let me create a character array that big (). Should the size be in the 1kb or 10kb range maybe, or does it really matter (as long as it's not too extremely large or small)?

[Hunter2]

Ok, thanks. That makes sense