recv large amounts of data

I'm writing a chat bot for PvPGN BnetD. The format will look similar to this:

Quote:

1010 NAME kpreston
1018 INFO "Welcome to PvPGN BnetD 1.5.0!"
1018 INFO "Type /help for a list of commands."
1002 JOIN kpreston 0001
1003 LEAVE somedude 0001
1003 LEAVE somedude(1) 0001
1002 JOIN somedude(2) 0001
1003 LEAVE somedude(2) 0001

The server is restarted every 3-5 hours. When this happens, approx. 1000 users/bots will reconnect. The issue I'm having is when recv() returns this:

Quote:

1010 NAME kpreston\r\n
1018 INFO "Welcome to PvPGN BnetD 1.5.0!"\r\n
1018 INFO "Type /help for a list of commands."\r\n
1002 JOIN kpreston 0001\r\n
1003 LEAVE somedude 0001\r\n
1003 LEAVE somedude(1) 0001\r\n
1002 JOIN somedu

The buffer gets parsed by "\r\n," and I have an incomplete packet at the end, which will cause undefined behaivor if I don't have a ton of error checking.

How would you solve this?

Edit: Speed and efficiency is the MOST important aspect to consider.

Keep recv'ing until you find a \r\n and process that as a command.

Quote:

Originally Posted by kpreston

which will cause undefined behaivor if I don't have a ton of error checking.

I think you just answered your own question.

Quote:

Originally Posted by prog-bman

Keep recv'ing until you find a \r\n and process that as a command.

Would I just continue to realloc more space? Would I want to free that memory and malloc a smaller buffer after I've parsed the huge chunk of data?

Quote:

I think you just answered your own question.

No I didn't.

Quote:

Originally Posted by kpreston

Edit: Speed and efficiency is the MOST important aspect to consider.

I doubt it. Processors are fast. Networks are slow.

There is no way around the fact that you have to queue up the data while you wait for the line terminators. The code is more complicated but I think your performance worries are unfounded. You aren't considering doing it "wrong" to actually be an option, are you?

Quote:

Originally Posted by brewbuck

I doubt it. Processors are fast. Networks are slow.

There is no way around the fact that you have to queue up the data while you wait for the line terminators. The code is more complicated but I think your performance worries are unfounded. You aren't considering doing it "wrong" to actually be an option, are you?

That's the problem. I don't know what "wrong" is. I'm not a very strong C coder, which is why I'm starting this project. What I've thought of doing is similar to this:

Code:

char *buffer = NULL; char tmp[128] = ""; int buffersize = 128; int n = 0; buffer = malloc(buffersize * sizeof(char)); buffer[0] = 0; while ((n = recv(s, tmp, 128, 0)) > 0) { tmp[n] = 0; if ((strlen(tmp) + strlen(buffer)) >= buffersize) { buffersize = buffersize * 2; buffer = realloc(buffer, buffersize * sizeof(char)); } strcat(buffer, tmp); if (buffer[strlen(buffer) - 1] == '\n') { ParseData(buffer); buffersize = 128; free(buffer); buffer = malloc(buffersize * sizeof(char)); buffer[0] = 0; } }

But there's nothing to say that 3 message fragments might be

1010 NAME kpreston\r\n1018 INFO
"Welcome to PvPGN BnetD 1.5.0!"\r
\n1018 INFO "Type /help for a list of commands."\r\n

Or for that matter, a single fragment with

1010 NAME kpreston\r\n1018 INFO "Welcome to PvPGN BnetD 1.5.0!"\r\n1018 INFO "Type /help for a list of commands."\r\n

If you're expecting to deal with 3 separate lines like

1010 NAME kpreston\r\n
1018 INFO "Welcome to PvPGN BnetD 1.5.0!"\r\n
1018 INFO "Type /help for a list of commands."\r\n

then you need to have a function which receives data, and returns lines when one (or more) newline pairs are found (and keep the rest of the message safe for next time).

Quote:

Originally Posted by Salem

But there's nothing to say that 3 message fragments might be

1010 NAME kpreston\r\n1018 INFO
"Welcome to PvPGN BnetD 1.5.0!"\r
\n1018 INFO "Type /help for a list of commands."\r\n

Or for that matter, a single fragment with

1010 NAME kpreston\r\n1018 INFO "Welcome to PvPGN BnetD 1.5.0!"\r\n1018 INFO "Type /help for a list of commands."\r\n

If you're expecting to deal with 3 separate lines like

1010 NAME kpreston\r\n
1018 INFO "Welcome to PvPGN BnetD 1.5.0!"\r\n
1018 INFO "Type /help for a list of commands."\r\n

then you need to have a function which receives data, and returns lines when one (or more) newline pairs are found (and keep the rest of the message safe for next time).

All three examples are possible.

Speed is an important consideration to make however what good is speed going to do when you keep timing out? How important is fast when you swiftly lose connection? You have to take a more balanced approach my friend.

I've got it. The above example I typed up quickly in notepad actually compiled and worked perfectly.

Bonus question!

Code:

token += 11;

This is my half-assed attempt at string manipulation. I read an example that used it years ago. Why does it work? Is there a "better" way?

You are skipping over the data inside the header. You are leaping over a constant size, so always double check documentation periodically to see if that constant ever changes.

Quote:

Originally Posted by master5001

You are skipping over the data inside the header.

You lost me.

:) You are skipping over some of the stuff that was received.

Quote:

Originally Posted by master5001

:) You are skipping over some of the stuff that was received.

Are you responding to the token += 11 thing?

Yep.

Quote:

Originally Posted by master5001

Yep.

Oh. Battle.net's CHAT protocol hasn't changed in over a decade.

Then the +11 thing should be fine :) I just thought I would give you "the talk" about skipping constants like that since if I don't someone else surely will.

Its good to know you are fundamentally having something like this occur.

Code:

RESP: 123456 xkpreston RESP: 092342 ymaster5001

And simply noticing "Gee, I don't really need that 'RESP: ' crap, do I? Plus it seems to always be the same. So I guess I can skip the first 6 characters."

Same idea with the +11 thing.

Bonus question #2!

I need connect to timeout. I'm trying to convert what I already have to use non-blocking sockets and select(), but I'm failing. It claims to be connected, but it never actually logs on or receives data. Am I using select() correctly?

Code:

FD_ZERO(&sockets); FD_SET(s, &sockets); timeout.tv_sec = 0; timeout.tv_usec = 100; printf("Connecting...\n"); connect(s, (struct sockaddr*)&sin, sizeof(sin)); while (select(0, NULL, &sockets, NULL, &timeout) == SOCKET_ERROR) { printf("[%d] ", WSAGetLastError()); printf("Retrying connection...\n"); connect(s, (struct sockaddr*)&sin, sizeof(sin)); FD_ZERO(&sockets); FD_SET(s, &sockets); }

Edit1:

Code:

FD_ZERO(&sockets); FD_SET(s, &sockets); timeout.tv_sec = 0; timeout.tv_usec = 900; printf("Connecting...\n"); connect(s, (struct sockaddr*)&sin, sizeof(sin)); ret = select(s+1, NULL, &sockets, NULL, &timeout); while (ret == 0 || ret == SOCKET_ERROR) { printf("[%d] ", WSAGetLastError()); printf("Retrying connection...\n"); connect(s, (struct sockaddr*)&sin, sizeof(sin)); FD_ZERO(&sockets); FD_SET(s, &sockets); ret = select(s+1, NULL, &sockets, NULL, &timeout); }

And it works. However, if I set tv_usec to 999 and under, I get [10035] Retrying connection, [10037] Retrying connection, [10037] Retrying connection, [10037] Retrying connection, Connected!. Why is this? Do I have to close the socket and create a new, every time I retry the connection?

Edit2: Durrrr. Microseconds.

Do I have to close the socket and create a new, every time I retry the connection?

MSVS comes with a program to tell you what these errors mean... But here you go

Quote:

Originally Posted by 10035

Resource temporarily unavailable. This error is returned from operations on non-blocking sockets that cannot be completed immediately, for example recv() when no data is queued to be read from the socket. It is a nonfatal error, and the operation should be retried later. It is normal for WSAEWOULDBLOCK to be reported as the result from calling connect() on a non-blocking SOCK_STREAM socket, since some time must elapse for the connection to be established.

Quote:

Originally Posted by 10037

Operation already in progress. An operation was attempted on a non-blocking socket with an operation already in progress - that is, calling connect() a second time on a non-blocking socket that is already connecting, or canceling an asynchronous request (WSAAsyncGetXbyY) that has already been canceled or completed.

Sockets on windows can be such a piece of crap at times. This is so much easier to do on linux.

So... I DO have to close the socket and create a new one if I want to stop a connection in progress and reconnect?

P.S. Thank you for being so patient with me. :)