Tokenizing a C++ string

[Lurker]

How can I token a C++ string object? The only thing I can think of is strtok(str.c_str(), /*...*/), but I know that's a horrible idea. Any thoughts? Thanks in advance!

[ggs]

convert it into a c-string and then roll your own tokenizer!

edit: i am only advocating the writing of your own lower-level tokenizing constructs, i feel that using library functions to achieve this is the incorrect approach >

[jlou]

There are lots of member functions with std::string that might be helpful. For example, find_first_of, find_first_not_of, substr, etc. Of course, you can always use existing tokenizers (there is one from boost) if you are so inclined.

I do agree with you Lurker, using "strtok(str.c_str()" isn't a good idea. If you wanted to use C style string functions, you could just use C style strings! I don't think it's that hard to write the equivalent of strtok for strings. There might even be examples on this board.

[Prelude]

>If you wanted to use C style string functions, you could just use C style strings!
The only way to call strtok with an std::string would be to call c_str, but it returns a const char * and strtok modifies its argument. Such code wouldn't be conductive to smart programming.

Here is a tokenizer class that I wrote up quickly a while ago. It comes in handy when you don't want to use a more complex object such as the boost tokenizer:

#include <sstream>
#include <string>
#include <vector>

class Tokenizer {
public:
  explicit Tokenizer ( const std::string& s, char delim = ' ' );
  explicit Tokenizer ( const std::string& s, const std::string& delim );
public:
  std::string next() { return !done() ? *current++ : std::string(); }
  bool done() const { return current == tokens.end(); }
private:
  std::vector<std::string>           tokens;
  std::vector<std::string>::iterator current;
};

Tokenizer::Tokenizer ( const std::string& s, char delim )
{
  std::istringstream grabber ( s );
  std::string        token;

  while ( getline ( grabber, token, delim ) ) {
    if ( !token.empty() )
      tokens.push_back ( token );
  }
  current = tokens.begin();
}

Tokenizer::Tokenizer ( const std::string& s, const std::string& delim )
{
  std::string            token;
  std::string::size_type front = 0;
  std::string::size_type back = 0;

  while ( true ) {
    if ( back == std::string::npos )
      break;
    front = s.find_first_not_of ( delim, front );
    if ( front == std::string::npos )
      break;
    back = s.find_first_of ( delim, front );
    token = s.substr ( front, back - front );
    tokens.push_back ( token );
    front = back + delim.length();
  }
  current = tokens.begin();
}