Thread: C++ String Tokenizer

I am working on a text RPG and I need a way to parse the user's commands and I think tokenizing the input would be the easiest so I can piece together the bits. However, it is not working well for me. Here is the piece of my code so far:

Code:

cout << "> ";
cin >> command;
str = strtok(command, " ");

while (1)
{
/* extract string from string sequence */
str = strtok(NULL, " ");

/* check if there is nothing else to extract */
if (str == NULL)
{
cout << "Tokenizing complete\n";
exit(1);
}

/* print string after tokenized */
printf("%i: %s\n", x, str);
x++;
}

This is a test running from a tokenizer I got at metalshell.com and I want to get it to parse a user input, which it won't do. It runs and I input text, then it says "Toeknizing Complete" then it ends.

I am fairly new to C++ so I don't know how to fix it. And what do the %i and %s do?

My goal here is that I can parse the string, then, for example, read the first token as "take", then read the second token as "club" and go from there.

Thanks for any help.

I'm using C++ so the printf shouldn't be there. I figured that %i was integer and %s is string.

But where are the tokens stored and how can I access them?

I researched this a bit and for the life of me I can't seem to figure it out and get it to work.

Example:
I want to parse a string, like "take club" into "take" and "club", storing those into some variable I can access.

I'm close with this, but it just won't work. I'm new at C++ so I must be overlooking something simple...

From the MSDN site, here is some code I modified:

Code:

#include <string.h>
#include <stdio.h>
#include <iostream.h>

char *token;
char stringx[32];

void main(void)
{
		cin >> stringx;
		token = strtok(stringx," "); /*There are two delimiters here*/
        while (token != NULL){
                printf("Token is: %s\n", token);
                token = strtok(NULL," ");
        }
}

The original used commas instead of spaces and it worked, mine doesn't work and it does if I use commas instead.... I don't see why spaces won't work...

For a more C++ feel, here is an algorithm I borrowed and modified it to make it work for me:

Code:

string buffer = "This is my string that I am trying to parse";
string delimiter = " "; // or whatever delimiter you rather use

vector<string> tokenList;
string token;
string::size_type startPos = 0;
string::size_type pos = buffer.find(delimiter);
string::size_type delLen = delimiter.length();
while (pos != string::npos) {
  token = buffer.substr(startPos, pos - startPos);
  tokenList.push_back(token);
  startPos = pos + delLen;
  pos = buffer.find(delimiter, startPos);
}
token = buffer.substr(startPos, buffer.length() - startPos);
if (!token.empty())
  tokenList.push_back(token);

Good luck...
Note: I do not recall where exactly I got the original algorithm but it's been modified since.

Easiest way would be to use a stringstream.

Code:

std::string command;
std::string temp;
std::vector<std::string> parsedList;
getline(command, std::cin); //can't use cin.getline with strings

std::istringstream blah(command); //yes, i'm creative with var names
while(blah>>temp)
     parsedList.push_back(temp);

Something like that should do the trick. No compiler around so as usual, check for minor errors.