boost tokenizer problem

This is a discussion on boost tokenizer problem within the C++ Programming forums, part of the General Programming Boards category; Hello.. Whenever I compile this code, I get boost compile error, though I have no idea what could be wrong. ...

  1. #1
    l2u
    l2u is offline
    Registered User
    Join Date
    May 2006
    Posts
    630

    boost tokenizer problem

    Hello..

    Whenever I compile this code, I get boost compile error, though I have no idea what could be wrong.

    Code:
    #include <string>
    #include <iostream>
    #include <boost/tokenizer.hpp>
    
    using namespace std;
    
    int parse(string data) {
    
    	typedef boost::tokenizer <boost::escaped_list_separator<char> > tokenizer;
    	tokenizer tokens(data);
    
    	for (tokenizer::iterator it = tokens.begin(); it != tokens.end(); it++) {
    		cout << *it << endl;
    
    		cout << *(it - 1) << endl;  //here i want to print the previous iterator (if it exists)
    	}
    
    	return 0;
    }
    Thanks a lot for help
    Last edited by l2u; 09-11-2006 at 10:36 PM.

  2. #2
    Cat
    Cat is offline
    Registered User
    Join Date
    May 2003
    Posts
    1,571
    You can't do (it - 1) in the code.

    The iterator you are working with will ONLY support ++. Not --, not +, not -.

    One solution is to copy the elements out of the tokenizer into a standard container. For example:

    Code:
    #include <string>
    #include <vector>
    #include <iostream>
    #include <boost/tokenizer.hpp>
    
    
    int parse(std::string data) {
    
    	typedef boost::tokenizer <boost::escaped_list_separator<char> > tokenizer;
    	tokenizer tokens(data);
    
    	std::vector<std::string> vectorTokens;
    
    	std::copy(tokens.begin(),tokens.end(),std::back_inserter(vectorTokens));
    
    
    
    	for (std::vector<std::string>::iterator it = vectorTokens.begin(); it != vectorTokens.end(); ++it) {
    		std::cout << *it << std::endl;
    
    		std::cout << *(it - 1) << std::endl;  //here i want to print the previous iterator (if it exists)
    	}
    
    	return 0;
    }
    Of course, you need to do something special to handle the very first iteration through the loop, or you'll try to walk off the beginning of the vector with the it-1.
    Last edited by Cat; 09-12-2006 at 12:12 AM.
    You ever try a pink golf ball, Wally? Why, the wind shear on a pink ball alone can take the head clean off a 90 pound midget at 300 yards.

  3. #3
    Cat without Hat CornedBee's Avatar
    Join Date
    Apr 2003
    Posts
    8,892
    Easier to just save the token for one iteration and print it again.
    All the buzzt!
    CornedBee

    "There is not now, nor has there ever been, nor will there ever be, any programming language in which it is the least bit difficult to write bad code."
    - Flon's Law

  4. #4
    l2u
    l2u is offline
    Registered User
    Join Date
    May 2006
    Posts
    630
    One more thing guys...

    This is the code for escaped_list_separator tokenizer, but I would like to have first escaped list separator and then char separator. What would be the best way to use both one after another?

    Lets say I have string "here is \"my string\" some,text".
    I should get <here> <is> <my string> <some> <text>

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. WS_POPUP, continuation of old problem
    By blurrymadness in forum Windows Programming
    Replies: 1
    Last Post: 04-20-2007, 06:54 PM
  2. Problem With Use of Unicode + Boost
    By Tonto in forum C++ Programming
    Replies: 0
    Last Post: 04-05-2007, 08:55 PM
  3. problem with boost random func (again)
    By l2u in forum C++ Programming
    Replies: 5
    Last Post: 11-06-2006, 02:22 AM
  4. boost shared_ptr problem
    By l2u in forum C++ Programming
    Replies: 5
    Last Post: 09-11-2006, 09:02 AM
  5. beginner problem
    By The_Nymph in forum C Programming
    Replies: 4
    Last Post: 03-05-2002, 04:46 PM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21