C++ String Tokenizer

This is a discussion on C++ String Tokenizer within the Game Programming forums, part of the General Programming Boards category; I am working on a text RPG and I need a way to parse the user's commands and I think ...

  1. #1
    Registered User
    Join Date
    Jul 2005
    Location
    MA
    Posts
    7

    Arrow C++ String Tokenizer

    I am working on a text RPG and I need a way to parse the user's commands and I think tokenizing the input would be the easiest so I can piece together the bits. However, it is not working well for me. Here is the piece of my code so far:
    Code:
    cout << "> ";
    cin >> command;
    str = strtok(command, " ");
    
    while (1)
    {
    /* extract string from string sequence */
    str = strtok(NULL, " ");
    
    /* check if there is nothing else to extract */
    if (str == NULL)
    {
    cout << "Tokenizing complete\n";
    exit(1);
    }
    
    /* print string after tokenized */
    printf("%i: %s\n", x, str);
    x++;
    }
    This is a test running from a tokenizer I got at metalshell.com and I want to get it to parse a user input, which it won't do. It runs and I input text, then it says "Toeknizing Complete" then it ends.

    I am fairly new to C++ so I don't know how to fix it. And what do the %i and %s do?

    My goal here is that I can parse the string, then, for example, read the first token as "take", then read the second token as "club" and go from there.

    Thanks for any help.

  2. #2

    Join Date
    May 2005
    Posts
    1,041
    Hints:

    -%i is for parsing integers

    -strtok modifies the character array that you pass in

    -look into sscanf on msdn from the C runtime library
    I'm not immature, I'm refined in the opposite direction.

  3. #3
    Registered User
    Join Date
    Jul 2005
    Location
    MA
    Posts
    7
    I'm using C++ so the printf shouldn't be there. I figured that %i was integer and %s is string.

    But where are the tokens stored and how can I access them?

  4. #4
    Registered User
    Join Date
    Jul 2005
    Location
    MA
    Posts
    7
    I researched this a bit and for the life of me I can't seem to figure it out and get it to work.

    Example:
    I want to parse a string, like "take club" into "take" and "club", storing those into some variable I can access.

    I'm close with this, but it just won't work. I'm new at C++ so I must be overlooking something simple...

  5. #5

    Join Date
    May 2005
    Posts
    1,041
    search it on msdn.microsoft.com

    IIRC strtok returns a pointer to the first segment of the string before the token

    strtok

    sscanf
    I'm not immature, I'm refined in the opposite direction.

  6. #6
    Registered User
    Join Date
    Jul 2005
    Location
    MA
    Posts
    7
    From the MSDN site, here is some code I modified:

    Code:
    #include <string.h>
    #include <stdio.h>
    #include <iostream.h>
    
    char *token;
    char stringx[32];
    
    void main(void)
    {
    		cin >> stringx;
    		token = strtok(stringx," "); /*There are two delimiters here*/
            while (token != NULL){
                    printf("Token is: %s\n", token);
                    token = strtok(NULL," ");
            }
    }
    The original used commas instead of spaces and it worked, mine doesn't work and it does if I use commas instead.... I don't see why spaces won't work...

  7. #7
    Frequently Quite Prolix dwks's Avatar
    Join Date
    Apr 2005
    Location
    Canada
    Posts
    8,045
    I would change void main() to int main().

    Does cin read spaces? I know scanf() doesn't unless you use it specially.
    dwk

    Seek and ye shall find. quaere et invenies.

    "Simplicity does not precede complexity, but follows it." -- Alan Perlis
    "Testing can only prove the presence of bugs, not their absence." -- Edsger Dijkstra
    "The only real mistake is the one from which we learn nothing." -- John Powell


    Other boards: DaniWeb, TPS
    Unofficial Wiki FAQ: cpwiki.sf.net

    My website: http://dwks.theprogrammingsite.com/
    Projects: codeform, xuni, atlantis, nort, etc.

  8. #8
    Magically delicious LuckY's Avatar
    Join Date
    Oct 2001
    Posts
    856
    A fundamental flaw in your algorithm is that you want to parse by space, yet you're reading the input with the extraction operator which skips all whitespace. So if someone entered "take club", stringx would contain only "take" leaving nothing to parse. Use getline() to read an entire line of text, then you can parse it.

  9. #9
    Frequently Quite Prolix dwks's Avatar
    Join Date
    Apr 2005
    Location
    Canada
    Posts
    8,045
    So it doesn't read spaces . . . now I know.
    dwk

    Seek and ye shall find. quaere et invenies.

    "Simplicity does not precede complexity, but follows it." -- Alan Perlis
    "Testing can only prove the presence of bugs, not their absence." -- Edsger Dijkstra
    "The only real mistake is the one from which we learn nothing." -- John Powell


    Other boards: DaniWeb, TPS
    Unofficial Wiki FAQ: cpwiki.sf.net

    My website: http://dwks.theprogrammingsite.com/
    Projects: codeform, xuni, atlantis, nort, etc.

  10. #10
    Registered User
    Join Date
    Jul 2005
    Posts
    1
    For a more C++ feel, here is an algorithm I borrowed and modified it to make it work for me:
    Code:
    string buffer = "This is my string that I am trying to parse";
    string delimiter = " "; // or whatever delimiter you rather use
    
    vector<string> tokenList;
    string token;
    string::size_type startPos = 0;
    string::size_type pos = buffer.find(delimiter);
    string::size_type delLen = delimiter.length();
    while (pos != string::npos) {
      token = buffer.substr(startPos, pos - startPos);
      tokenList.push_back(token);
      startPos = pos + delLen;
      pos = buffer.find(delimiter, startPos);
    }
    token = buffer.substr(startPos, buffer.length() - startPos);
    if (!token.empty())
      tokenList.push_back(token);
    Good luck...
    Note: I do not recall where exactly I got the original algorithm but it's been modified since.

  11. #11
    ¡Amo fútbol!
    Join Date
    Dec 2001
    Posts
    2,136
    Easiest way would be to use a stringstream.

    Code:
    std::string command;
    std::string temp;
    std::vector<std::string> parsedList;
    getline(command, std::cin); //can't use cin.getline with strings
    
    std::istringstream blah(command); //yes, i'm creative with var names
    while(blah>>temp)
         parsedList.push_back(temp);
    Something like that should do the trick. No compiler around so as usual, check for minor errors.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. char Handling, probably typical newbie stuff
    By Neolyth in forum C Programming
    Replies: 16
    Last Post: 06-21-2009, 04:05 AM
  2. String tokenizer and delimiters
    By John_L in forum C Programming
    Replies: 5
    Last Post: 11-06-2007, 06:22 PM
  3. RicBot
    By John_ in forum C++ Programming
    Replies: 8
    Last Post: 06-13-2006, 06:52 PM
  4. Classes inheretance problem...
    By NANO in forum C++ Programming
    Replies: 12
    Last Post: 12-09-2002, 02:23 PM
  5. Warnings, warnings, warnings?
    By spentdome in forum C Programming
    Replies: 25
    Last Post: 05-27-2002, 06:49 PM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21