Tokenizing a C++ string

This is a discussion on Tokenizing a C++ string within the C++ Programming forums, part of the General Programming Boards category; How can I token a C++ string object? The only thing I can think of is strtok(str.c_str(), /*...*/), but I ...

  1. #1
    The Defective GRAPE Lurker's Avatar
    Join Date
    Feb 2003
    Posts
    949

    Tokenizing a C++ string

    How can I token a C++ string object? The only thing I can think of is strtok(str.c_str(), /*...*/), but I know that's a horrible idea. Any thoughts? Thanks in advance!
    Do not make direct eye contact with me.

  2. #2
    ggs
    ggs is offline
    C > C++ duders ggs's Avatar
    Join Date
    Aug 2001
    Posts
    435
    convert it into a c-string and then roll your own tokenizer!

    edit: i am only advocating the writing of your own lower-level tokenizing constructs, i feel that using library functions to achieve this is the incorrect approach >
    Last edited by ggs; 04-06-2004 at 09:29 PM. Reason: post script
    .sect signature

  3. #3
    Registered User jlou's Avatar
    Join Date
    Jul 2003
    Posts
    1,088
    There are lots of member functions with std::string that might be helpful. For example, find_first_of, find_first_not_of, substr, etc. Of course, you can always use existing tokenizers (there is one from boost) if you are so inclined.

    I do agree with you Lurker, using "strtok(str.c_str()" isn't a good idea. If you wanted to use C style string functions, you could just use C style strings! I don't think it's that hard to write the equivalent of strtok for strings. There might even be examples on this board.

  4. #4
    Code Goddess Prelude's Avatar
    Join Date
    Sep 2001
    Posts
    9,796
    >If you wanted to use C style string functions, you could just use C style strings!
    The only way to call strtok with an std::string would be to call c_str, but it returns a const char * and strtok modifies its argument. Such code wouldn't be conductive to smart programming.

    Here is a tokenizer class that I wrote up quickly a while ago. It comes in handy when you don't want to use a more complex object such as the boost tokenizer:
    Code:
    #include <sstream>
    #include <string>
    #include <vector>
    
    class Tokenizer {
    public:
      explicit Tokenizer ( const std::string& s, char delim = ' ' );
      explicit Tokenizer ( const std::string& s, const std::string& delim );
    public:
      std::string next() { return !done() ? *current++ : std::string(); }
      bool done() const { return current == tokens.end(); }
    private:
      std::vector<std::string>           tokens;
      std::vector<std::string>::iterator current;
    };
    
    Tokenizer::Tokenizer ( const std::string& s, char delim )
    {
      std::istringstream grabber ( s );
      std::string        token;
    
      while ( getline ( grabber, token, delim ) ) {
        if ( !token.empty() )
          tokens.push_back ( token );
      }
      current = tokens.begin();
    }
    
    Tokenizer::Tokenizer ( const std::string& s, const std::string& delim )
    {
      std::string            token;
      std::string::size_type front = 0;
      std::string::size_type back = 0;
    
      while ( true ) {
        if ( back == std::string::npos )
          break;
        front = s.find_first_not_of ( delim, front );
        if ( front == std::string::npos )
          break;
        back = s.find_first_of ( delim, front );
        token = s.substr ( front, back - front );
        tokens.push_back ( token );
        front = back + delim.length();
      }
      current = tokens.begin();
    }
    My best code is written with the delete key.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Inheritance Hierarchy for a Package class
    By twickre in forum C++ Programming
    Replies: 7
    Last Post: 12-08-2007, 04:13 PM
  2. String issues
    By The_professor in forum C++ Programming
    Replies: 7
    Last Post: 06-12-2007, 10:11 AM
  3. Compile Error that i dont understand
    By bobthebullet990 in forum C++ Programming
    Replies: 5
    Last Post: 05-05-2006, 10:19 AM
  4. Classes inheretance problem...
    By NANO in forum C++ Programming
    Replies: 12
    Last Post: 12-09-2002, 03:23 PM
  5. Warnings, warnings, warnings?
    By spentdome in forum C Programming
    Replies: 25
    Last Post: 05-27-2002, 07:49 PM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21