I want to write a tokenizer for c++ string. I am reading in a file
which is delimited by each line is delimited by '|' and on the first and every sixteenth the delimiter is '\n'. Is there a class or example known to anyone available??
I want to write a tokenizer for c++ string. I am reading in a file
which is delimited by each line is delimited by '|' and on the first and every sixteenth the delimiter is '\n'. Is there a class or example known to anyone available??
You can find a number of tokenizing classes on the net, but writing one really is trivial:
Code:/* * Disclaimer: * All unique code supplied in this post is given as-is. * The author of this program is not responsible for any * errors, warnings, inconsistencies, or general lack of * usefulness concerning the actual problem in question. * * The following code is given as an example only. Any * resemblance to the complete solution of the given * problem is purely coincidence. */ #include <iostream> #include <string> #include <vector> using namespace std; vector<string> tokenize(string s, string delim) { vector<string> rv; string::size_type spos; string::size_type epos; spos = 0; while (1) { epos = s.find_first_of(delim, spos); rv.push_back(s.substr(spos, epos - spos)); if (epos == string::npos) return rv; spos = epos + 1; } } int main() { vector<string> rv; rv = tokenize("This is a test", " "); for (int i = 0; i < rv.size(); i++) cout<< rv[i] <<endl; }
My best code is written with the delete key.
Shouldn't main return something?
MagosX.com
Give a man a fish and you feed him for a day.
Teach a man to fish and you feed him for a lifetime.
>> I want to write a tokenizer for c++ string.
>> Is there a class or example known to anyone available??
Are you wanting to write one or find one precoded? Perhaps it would be a good exercise for you to give it a go for yourself?
Code:#include <cmath> #include <complex> bool euler_flip(bool value) { return std::pow ( std::complex<float>(std::exp(1.0)), std::complex<float>(0, 1) * std::complex<float>(std::atan(1.0) *(1 << (value + 2))) ).real() < 0; }
>Shouldn't main return something?
Not unless one uses a very old compiler, which I don't. Modern C++ returns a value for success automatically if no explicit return in main is present. C99 does the same thing, but so few people use C99, so the old advice concerning undefined behavior still stands.
My best code is written with the delete key.
Boost has a good string tokenizer class.Originally posted by nomes
I want to write a tokenizer for c++ string. I am reading in a file
which is delimited by each line is delimited by '|' and on the first and every sixteenth the delimiter is '\n'. Is there a class or example known to anyone available??
I have the tokenizer working until I want to copy each element of
the vector to a string variable (at moment is cout). I am getting a segmentation fault in the "else", the file reads a line and passes to function. if the vector has only one element store in country else will store in other variables, this goes thru once until the 2nd country is read by file and I get the segmentation fault. If I dont have the else in the code the program works fine. Not sure if I get the segmentation fault as am not deleting the vector each time??
Code:string playerCountry; vector<string> tokens; std::vector<string>::const_iterator p; Tokenize(line, tokens,"|"); p=tokens.begin(); if(tokens.size()==1) { cout<<"Should be country "<<*p<<endl; } else { cout<<"Token [0]: "<<p[0]<<endl; cout<<"Token [1]: "<<p[1]<<endl; cout<<"Token [2]: "<<p[2]<<endl; cout<<"Token [3]: "<<p[3]<<endl; cout<<"Token [4]: "<<p[4]<<endl; }
Probaly because your vector holds "non-dynamic" strings, which are deallocated right after you return from the Tokenize function.
Try using std::vector<string*> and allocate them in your Tokenize function. Donīt forget to deallocated them manually.
If still doesnīt work, post your Tokenize function code.
Hope that helps.
Nothing more to tell about me...
Happy day =)