tokenise read in file

This is a discussion on tokenise read in file within the C++ Programming forums, part of the General Programming Boards category; I want to write a tokenizer for c++ string. I am reading in a file which is delimited by each ...

  1. #1
    Registered User
    Join Date
    Sep 2002
    Posts
    24

    tokenise read in file

    I want to write a tokenizer for c++ string. I am reading in a file
    which is delimited by each line is delimited by '|' and on the first and every sixteenth the delimiter is '\n'. Is there a class or example known to anyone available??

  2. #2
    Code Goddess Prelude's Avatar
    Join Date
    Sep 2001
    Posts
    9,796
    You can find a number of tokenizing classes on the net, but writing one really is trivial:
    Code:
    /*
     * Disclaimer:
     *   All unique code supplied in this post is given as-is.
     *   The author of this program is not responsible for any
     *   errors, warnings, inconsistencies, or general lack of
     *   usefulness concerning the actual problem in question.
     *
     *   The following code is given as an example only. Any
     *   resemblance to the complete solution of the given
     *   problem is purely coincidence.
     */
    #include <iostream>
    #include <string>
    #include <vector>
    
    using namespace std;
    
    vector<string> tokenize(string s, string delim)
    {
      vector<string> rv;
      string::size_type spos;
      string::size_type epos;
    
      spos = 0;
      while (1) {
        epos = s.find_first_of(delim, spos); 
        rv.push_back(s.substr(spos, epos - spos));
        if (epos == string::npos)
          return rv;
        spos = epos + 1;
      }
    }
    
    int main()
    {
      vector<string> rv;
    
      rv = tokenize("This is a test", " ");
      for (int i = 0; i < rv.size(); i++)
        cout<< rv[i] <<endl;
    }
    My best code is written with the delete key.

  3. #3
    Confused Magos's Avatar
    Join Date
    Sep 2001
    Location
    Sweden
    Posts
    3,145
    Shouldn't main return something?
    MagosX.com

    Give a man a fish and you feed him for a day.
    Teach a man to fish and you feed him for a lifetime.

  4. #4
    Guest Sebastiani's Avatar
    Join Date
    Aug 2001
    Location
    Waterloo, Texas
    Posts
    5,600
    >> I want to write a tokenizer for c++ string.

    >> Is there a class or example known to anyone available??

    Are you wanting to write one or find one precoded? Perhaps it would be a good exercise for you to give it a go for yourself?
    Code:
    if( numeric_limits< byte >::digits != bits_per_byte )
        error( "program requires bits_per_byte-bit bytes" );
    24bbs.cpp

  5. #5
    Code Goddess Prelude's Avatar
    Join Date
    Sep 2001
    Posts
    9,796
    >Shouldn't main return something?
    Not unless one uses a very old compiler, which I don't. Modern C++ returns a value for success automatically if no explicit return in main is present. C99 does the same thing, but so few people use C99, so the old advice concerning undefined behavior still stands.
    My best code is written with the delete key.

  6. #6
    Cat
    Cat is offline
    Registered User
    Join Date
    May 2003
    Posts
    1,571

    Re: tokenise read in file

    Originally posted by nomes
    I want to write a tokenizer for c++ string. I am reading in a file
    which is delimited by each line is delimited by '|' and on the first and every sixteenth the delimiter is '\n'. Is there a class or example known to anyone available??
    Boost has a good string tokenizer class.

  7. #7
    Registered User
    Join Date
    Sep 2002
    Posts
    24
    I have the tokenizer working until I want to copy each element of
    the vector to a string variable (at moment is cout). I am getting a segmentation fault in the "else", the file reads a line and passes to function. if the vector has only one element store in country else will store in other variables, this goes thru once until the 2nd country is read by file and I get the segmentation fault. If I dont have the else in the code the program works fine. Not sure if I get the segmentation fault as am not deleting the vector each time??

    Code:
    string playerCountry;
    	vector<string> tokens;
    	std::vector<string>::const_iterator p;
    	Tokenize(line, tokens,"|");
    	p=tokens.begin();
    	
    	if(tokens.size()==1)
    	{       
    		cout<<"Should be country  "<<*p<<endl;
    		
    	}
    	
    	else
                    {       
       	   cout<<"Token [0]: "<<p[0]<<endl;
       	   cout<<"Token [1]: "<<p[1]<<endl;
       	   cout<<"Token [2]: "<<p[2]<<endl;
       	   cout<<"Token [3]: "<<p[3]<<endl;
       	   cout<<"Token [4]: "<<p[4]<<endl;
       	}

  8. #8
    Disturbed Boy gustavosserra's Avatar
    Join Date
    Apr 2003
    Posts
    244
    Probaly because your vector holds "non-dynamic" strings, which are deallocated right after you return from the Tokenize function.
    Try using std::vector<string*> and allocate them in your Tokenize function. Donīt forget to deallocated them manually.
    If still doesnīt work, post your Tokenize function code.

    Hope that helps.
    Nothing more to tell about me...
    Happy day =)

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. File Writing Problem
    By polskash in forum C Programming
    Replies: 3
    Last Post: 02-13-2009, 09:47 AM
  2. Inventory records
    By jsbeckton in forum C Programming
    Replies: 23
    Last Post: 06-28-2007, 04:14 AM
  3. Replies: 3
    Last Post: 03-04-2005, 01:46 PM
  4. c script help to read a file & update
    By indy in forum C Programming
    Replies: 8
    Last Post: 12-01-2003, 10:32 AM
  5. Hmm....help me take a look at this: File Encryptor
    By heljy in forum C Programming
    Replies: 3
    Last Post: 03-23-2002, 09:57 AM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21