C++ String Tokenizer

I am working on a text RPG and I need a way to parse the user's commands and I think tokenizing the input would be the easiest so I can piece together the bits. However, it is not working well for me. Here is the piece of my code so far:

Code:

cout << "> ";
cin >> command;
str = strtok(command, " ");

while (1)
{
/* extract string from string sequence */
str = strtok(NULL, " ");

/* check if there is nothing else to extract */
if (str == NULL)
{
cout << "Tokenizing complete\n";
exit(1);
}

/* print string after tokenized */
printf("%i: %s\n", x, str);
x++;
}

This is a test running from a tokenizer I got at metalshell.com and I want to get it to parse a user input, which it won't do. It runs and I input text, then it says "Toeknizing Complete" then it ends.

I am fairly new to C++ so I don't know how to fix it. And what do the %i and %s do?

My goal here is that I can parse the string, then, for example, read the first token as "take", then read the second token as "club" and go from there.

Thanks for any help.

Hints:

-%i is for parsing integers

-strtok modifies the character array that you pass in

-look into sscanf on msdn from the C runtime library

I'm using C++ so the printf shouldn't be there. I figured that %i was integer and %s is string.

But where are the tokens stored and how can I access them?

I researched this a bit and for the life of me I can't seem to figure it out and get it to work.

Example:
I want to parse a string, like "take club" into "take" and "club", storing those into some variable I can access.

I'm close with this, but it just won't work. I'm new at C++ so I must be overlooking something simple...

search it on msdn.microsoft.com

IIRC strtok returns a pointer to the first segment of the string before the token

strtok

sscanf

From the MSDN site, here is some code I modified:

Code:

#include <string.h>
#include <stdio.h>
#include <iostream.h>

char *token;
char stringx[32];

void main(void)
{
		cin >> stringx;
		token = strtok(stringx," "); /*There are two delimiters here*/
        while (token != NULL){
                printf("Token is: %s\n", token);
                token = strtok(NULL," ");
        }
}

The original used commas instead of spaces and it worked, mine doesn't work and it does if I use commas instead.... I don't see why spaces won't work...

I would change void main() to int main().

Does cin read spaces? I know scanf() doesn't unless you use it specially.

A fundamental flaw in your algorithm is that you want to parse by space, yet you're reading the input with the extraction operator which skips all whitespace. So if someone entered "take club", stringx would contain only "take" leaving nothing to parse. Use getline() to read an entire line of text, then you can parse it.

So it doesn't read spaces . . . now I know.

For a more C++ feel, here is an algorithm I borrowed and modified it to make it work for me:

Code:

string buffer = "This is my string that I am trying to parse";
string delimiter = " "; // or whatever delimiter you rather use

vector<string> tokenList;
string token;
string::size_type startPos = 0;
string::size_type pos = buffer.find(delimiter);
string::size_type delLen = delimiter.length();
while (pos != string::npos) {
  token = buffer.substr(startPos, pos - startPos);
  tokenList.push_back(token);
  startPos = pos + delLen;
  pos = buffer.find(delimiter, startPos);
}
token = buffer.substr(startPos, buffer.length() - startPos);
if (!token.empty())
  tokenList.push_back(token);

Good luck...
Note: I do not recall where exactly I got the original algorithm but it's been modified since.

Easiest way would be to use a stringstream.

Code:

std::string command;
std::string temp;
std::vector<std::string> parsedList;
getline(command, std::cin); //can't use cin.getline with strings

std::istringstream blah(command); //yes, i'm creative with var names
while(blah>>temp)
     parsedList.push_back(temp);

Something like that should do the trick. No compiler around so as usual, check for minor errors.