Thread: recv large amounts of data

  1. #1
    Registered User
    Join Date
    Aug 2008
    Posts
    67

    recv large amounts of data

    I'm writing a chat bot for PvPGN BnetD. The format will look similar to this:

    1010 NAME kpreston
    1018 INFO "Welcome to PvPGN BnetD 1.5.0!"
    1018 INFO "Type /help for a list of commands."
    1002 JOIN kpreston 0001
    1003 LEAVE somedude 0001
    1003 LEAVE somedude(1) 0001
    1002 JOIN somedude(2) 0001
    1003 LEAVE somedude(2) 0001
    The server is restarted every 3-5 hours. When this happens, approx. 1000 users/bots will reconnect. The issue I'm having is when recv() returns this:

    1010 NAME kpreston\r\n
    1018 INFO "Welcome to PvPGN BnetD 1.5.0!"\r\n
    1018 INFO "Type /help for a list of commands."\r\n
    1002 JOIN kpreston 0001\r\n
    1003 LEAVE somedude 0001\r\n
    1003 LEAVE somedude(1) 0001\r\n
    1002 JOIN somedu
    The buffer gets parsed by "\r\n," and I have an incomplete packet at the end, which will cause undefined behaivor if I don't have a ton of error checking.

    How would you solve this?

    Edit: Speed and efficiency is the MOST important aspect to consider.
    Last edited by kpreston; 10-16-2008 at 12:12 PM.

  2. #2
    Sweet
    Join Date
    Aug 2002
    Location
    Tucson, Arizona
    Posts
    1,820
    Keep recv'ing until you find a \r\n and process that as a command.
    Woop?

  3. #3
    Registered User
    Join Date
    Oct 2001
    Posts
    2,129
    Quote Originally Posted by kpreston View Post
    which will cause undefined behaivor if I don't have a ton of error checking.
    I think you just answered your own question.

  4. #4
    Registered User
    Join Date
    Aug 2008
    Posts
    67
    Quote Originally Posted by prog-bman View Post
    Keep recv'ing until you find a \r\n and process that as a command.
    Would I just continue to realloc more space? Would I want to free that memory and malloc a smaller buffer after I've parsed the huge chunk of data?

    I think you just answered your own question.
    No I didn't.

  5. #5
    Officially An Architect brewbuck's Avatar
    Join Date
    Mar 2007
    Location
    Portland, OR
    Posts
    7,396
    Quote Originally Posted by kpreston View Post
    Edit: Speed and efficiency is the MOST important aspect to consider.
    I doubt it. Processors are fast. Networks are slow.

    There is no way around the fact that you have to queue up the data while you wait for the line terminators. The code is more complicated but I think your performance worries are unfounded. You aren't considering doing it "wrong" to actually be an option, are you?
    Code:
    //try
    //{
    	if (a) do { f( b); } while(1);
    	else   do { f(!b); } while(1);
    //}

  6. #6
    Registered User
    Join Date
    Aug 2008
    Posts
    67
    Quote Originally Posted by brewbuck View Post
    I doubt it. Processors are fast. Networks are slow.

    There is no way around the fact that you have to queue up the data while you wait for the line terminators. The code is more complicated but I think your performance worries are unfounded. You aren't considering doing it "wrong" to actually be an option, are you?
    That's the problem. I don't know what "wrong" is. I'm not a very strong C coder, which is why I'm starting this project. What I've thought of doing is similar to this:

    Code:
    char 	*buffer 	= NULL;
    char 	tmp[128] 	= "";
    int 	buffersize 	= 128;
    int 	n		= 0;
    
    buffer = malloc(buffersize * sizeof(char));
    buffer[0] = 0;
    
    while ((n = recv(s, tmp, 128, 0)) > 0) {
    	tmp[n] = 0;
    	if ((strlen(tmp) + strlen(buffer)) >= buffersize) {
    		buffersize = buffersize * 2;
    		buffer = realloc(buffer, buffersize * sizeof(char));
    	}
    	strcat(buffer, tmp);
    	if (buffer[strlen(buffer) - 1] == '\n') {
    		ParseData(buffer);
    		buffersize = 128;
    		free(buffer);
    		buffer = malloc(buffersize * sizeof(char));
    		buffer[0] = 0;
    	}
    }

  7. #7
    and the hat of int overfl Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    39,660
    But there's nothing to say that 3 message fragments might be

    1010 NAME kpreston\r\n1018 INFO
    "Welcome to PvPGN BnetD 1.5.0!"\r
    \n1018 INFO "Type /help for a list of commands."\r\n

    Or for that matter, a single fragment with

    1010 NAME kpreston\r\n1018 INFO "Welcome to PvPGN BnetD 1.5.0!"\r\n1018 INFO "Type /help for a list of commands."\r\n

    If you're expecting to deal with 3 separate lines like

    1010 NAME kpreston\r\n
    1018 INFO "Welcome to PvPGN BnetD 1.5.0!"\r\n
    1018 INFO "Type /help for a list of commands."\r\n

    then you need to have a function which receives data, and returns lines when one (or more) newline pairs are found (and keep the rest of the message safe for next time).
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.

  8. #8
    Registered User
    Join Date
    Aug 2008
    Posts
    67
    Quote Originally Posted by Salem View Post
    But there's nothing to say that 3 message fragments might be

    1010 NAME kpreston\r\n1018 INFO
    "Welcome to PvPGN BnetD 1.5.0!"\r
    \n1018 INFO "Type /help for a list of commands."\r\n

    Or for that matter, a single fragment with

    1010 NAME kpreston\r\n1018 INFO "Welcome to PvPGN BnetD 1.5.0!"\r\n1018 INFO "Type /help for a list of commands."\r\n

    If you're expecting to deal with 3 separate lines like

    1010 NAME kpreston\r\n
    1018 INFO "Welcome to PvPGN BnetD 1.5.0!"\r\n
    1018 INFO "Type /help for a list of commands."\r\n

    then you need to have a function which receives data, and returns lines when one (or more) newline pairs are found (and keep the rest of the message safe for next time).
    All three examples are possible.

  9. #9
    Banned master5001's Avatar
    Join Date
    Aug 2001
    Location
    Visalia, CA, USA
    Posts
    3,685
    Speed is an important consideration to make however what good is speed going to do when you keep timing out? How important is fast when you swiftly lose connection? You have to take a more balanced approach my friend.

  10. #10
    Registered User
    Join Date
    Aug 2008
    Posts
    67
    I've got it. The above example I typed up quickly in notepad actually compiled and worked perfectly.

    Bonus question!

    Code:
    token += 11;
    This is my half-assed attempt at string manipulation. I read an example that used it years ago. Why does it work? Is there a "better" way?

  11. #11
    Banned master5001's Avatar
    Join Date
    Aug 2001
    Location
    Visalia, CA, USA
    Posts
    3,685
    You are skipping over the data inside the header. You are leaping over a constant size, so always double check documentation periodically to see if that constant ever changes.

  12. #12
    Registered User
    Join Date
    Aug 2008
    Posts
    67
    Quote Originally Posted by master5001 View Post
    You are skipping over the data inside the header.
    You lost me.

  13. #13
    Banned master5001's Avatar
    Join Date
    Aug 2001
    Location
    Visalia, CA, USA
    Posts
    3,685
    You are skipping over some of the stuff that was received.

  14. #14
    Registered User
    Join Date
    Aug 2008
    Posts
    67
    Quote Originally Posted by master5001 View Post
    You are skipping over some of the stuff that was received.
    Are you responding to the token += 11 thing?

  15. #15
    Banned master5001's Avatar
    Join Date
    Aug 2001
    Location
    Visalia, CA, USA
    Posts
    3,685
    Yep.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. xor linked list
    By adramalech in forum C Programming
    Replies: 23
    Last Post: 10-14-2008, 10:13 AM
  2. Large Data management in using File
    By mohsan1987 in forum C Programming
    Replies: 6
    Last Post: 09-14-2006, 06:41 AM
  3. Reading large complicated data files
    By dodzy in forum C Programming
    Replies: 16
    Last Post: 05-17-2006, 04:57 PM
  4. gcc problem
    By bjdea1 in forum Linux Programming
    Replies: 13
    Last Post: 04-29-2002, 06:51 PM
  5. accepting large amounts of data
    By Sekti in forum C++ Programming
    Replies: 1
    Last Post: 04-05-2002, 05:45 PM