String Tokenizing

This is a discussion on String Tokenizing within the C++ Programming forums, part of the General Programming Boards category; That last solution works fine for my first case of reading in a blank 1st field, but now if I ...

  1. #16
    Registered User
    Join Date
    May 2003
    Posts
    11
    That last solution works fine for my first case of reading in a blank 1st field, but now if I have something like:

    pMsg = field1|field2||field4

    with a blank field in the middle somewhere, then everything is still hosing up after that first strtok.

    token1 is now being assigned the empty string like I want it
    token2 is getting field2
    token3 is getting field4
    token4 is NULL

    I would think you could just do the same thing 4 times, but I believe the way strtok is working, pMsg isn't getting moved over. I'm terrible with words. If anyone understands what I'm saying, help is greatly appreciated. If not, I'll keep fighting until I get it figured out. Thanks

  2. #17
    Magically delicious LuckY's Avatar
    Join Date
    Oct 2001
    Posts
    856
    Troi oi...
    Here, I just wrote this real quick. Maybe it'll work...
    Code:
    const char* pipeTokenize(const char *str, char *tokenized, int bufSize) {
      *tokenized = NULL;
      int size = 0;
    
      while (*str != NULL && size < bufSize) {
    
        if (*str == '|') {
          ++str;
          break;
        }
    
        ++size;
        *tokenized = *str;
         *++tokenized = NULL;
    
        ++str;
      }
    
      return *str == NULL ? NULL : str;
    }
    
    int main() {
      string delimited = "|field2||field4";
      char buffer[256];
    
      const  char *p  = delimited.c_str();
      while (p != NULL) {
        p = pipeTokenize(p, buffer, 256);
        
        cout << "Hey dummy, here's a field: " << buffer << endl;
      }
    
      return 0;
    }

  3. #18
    Cat without Hat CornedBee's Avatar
    Join Date
    Apr 2003
    Posts
    8,893
    What's wrong with my function splitting to a vector of strings? After all we're using C++ here, right?
    All the buzzt!
    CornedBee

    "There is not now, nor has there ever been, nor will there ever be, any programming language in which it is the least bit difficult to write bad code."
    - Flon's Law

  4. #19
    The Pantless Man CheesyMoo's Avatar
    Join Date
    Jan 2003
    Posts
    262
    This was on the AP Computer Science exam I took today, sorry if it's already been said.
    If you ever need a hug, just ask.

  5. #20
    Magically delicious LuckY's Avatar
    Join Date
    Oct 2001
    Posts
    856
    Originally posted by CheesyMoo
    This was on the AP Computer Science exam I took today, sorry if it's already been said.
    Yeah, several people have come on the board telling us about your AP computer science exam... It's funny you didn't notice.

    Can I have a hug?

  6. #21
    Registered User
    Join Date
    Aug 2001
    Posts
    223

    string tok

    I had the same problem because I was reading in from a tab delimited file.... Here is what I did... I used the CStrings replace function to replace "\t\t" with "\t \t" so that there would be at least one space in between tabs and therefore the toke would be picked up as a blank space. Later if I found a blank space I would just copy an empty sting into that token. Of course this means that you have to move the buffer into a string and then back into the buffer. I am not quite sure what the equivalent functions of CString are in string to accomplish this but I am sure that they exist.

    Code:
    while( cFile.ReadString( strInline ) == TRUE)
    {
          //Trime all spaces to the right in case of empty string
          strInline.TrimRight();
    			
         if( strInline <= " ")
    	continue;
    
        //Remove all back to back tabs with space in between them
        while( strInline.Find("\t\t") > -1)
                strInline.Replace("\t\t", "\t \t");
    
        char *buffer;
        char *pToken;
    
        //Create a new buffer
        buffer = new char[ strInline.GetLength()+1];
    
        strcpy(buffer, strInline.GetBuffer( strInline.GetLength()+1 ));
    
        pToken = strtok(buffer, "\t");
        int i = 0;
        CString strTokens[31];
    
        strTokens[i++] = pToken;
        if(strTokens[i-1] <= " ")
        strTokens[i-1] = _T("");
    
       while( pToken != NULL && i < 30)
       {
              /* Get next token: */
              pToken = strtok( NULL, "\t");
              strTokens[i++] = pToken;
    
              if( strTokens[i-1] <= " ")
    	strTokens[i-1] = _T("");
       }
    zMan

  7. #22
    Registered User
    Join Date
    Feb 2002
    Posts
    329
    I guess i'll have to write the whole thing:
    Code:
    if(*pMsg=='|'){	// Handle blank #1 strings
    	strtok(pMsg, "|");
    	token1 = "";
    }else
    	token1 = strtok(pMsg, "|");
    	
    // Next
    pToken = strtok(NULL, "|");
    token2 = ((pToken-token1)>1)?pToken:"";
    // And next
    pToken = strtok(NULL, "|");
    token3 = ((pToken-token2)>1)?pToken:"";
    // And next
    pToken = strtok(NULL, "|");
    token4 = ((pToken-token3)>1)?pToken:"";
    This should handle empty strings too. I thought you'd see how to do continue the function..

Page 2 of 2 FirstFirst 12
Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Inheritance Hierarchy for a Package class
    By twickre in forum C++ Programming
    Replies: 7
    Last Post: 12-08-2007, 04:13 PM
  2. String issues
    By The_professor in forum C++ Programming
    Replies: 7
    Last Post: 06-12-2007, 10:11 AM
  3. Compile Error that i dont understand
    By bobthebullet990 in forum C++ Programming
    Replies: 5
    Last Post: 05-05-2006, 10:19 AM
  4. Classes inheretance problem...
    By NANO in forum C++ Programming
    Replies: 12
    Last Post: 12-09-2002, 03:23 PM
  5. Warnings, warnings, warnings?
    By spentdome in forum C Programming
    Replies: 25
    Last Post: 05-27-2002, 07:49 PM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21