Split string up into single words

[Todd88]

As some of you well know (since I have been asking so many questions in the forum) I am creating an HTML book given a simple text file as input. I am also creating an index of certain terms in the book. Right now I have the book output implemented and am now starting on the index portion. I have this loop right here that takes each line and adds the line to a vector:

Page current;
      current.setPageNum(PageNo);

      for(int lineNumber = 0; lineNumber < MAX_LINES_PER_PAGE; lineNumber++)
      {
          string line = book.readLine();
          current.addLine(line);
      }

      current.output(out, book.theTitle, pageNo, nextPageNo, prevPageNo);

Now I need to put a function in here that basically splits the "line" variable up into single words then takes those words and passes them to my index function named "addWord." I have tried using substr and doing line.find(' ') and it gives runtime errors. Any help would be appreciated.

[matsp]

It would help quite a bit if you posted your code that actually does the splitting and failing...

--
Mats

[Todd88]

Okay here is the failing code:

Page current;
      Index create;
      current.setPageNum(PageNo);

      for(int lineNumber = 0; lineNumber < MAX_LINES_PER_PAGE; lineNumber++)
      {
          string line = book.readLine();
          current.addLine(line);
          position = line.find(' ');
          create.addWord(line.substr(position), PageNo);
      }

      current.output(out, book.theTitle, pageNo, nextPageNo, prevPageNo);

[tabstop]

What if a page is shorter than MAX_LINES_PER_PAGE? Does readLine fail in a nice way?

[C_ntua]

Guessing time!

string line;
int start = line.find(' ', 0) + 1;
int end = line.find(' ', start);
string word = line.substr(line, start, end - start);

to get a word from the beginning. My guess it that you foget the +1? Or you put end as the second parameter in substr?

Here is a more C way (if you prefer):

char* cline = line.c_str();
string word("");
vector<string> words;
bool start = false;
for (int i = 0; i < line.size; ++i)
{
    if (cline[i] == ' ') {
              if (word != "") 
                   words.push_back(word);
              word = "";
         }
    }
    if (cline[i] == \n || cline == EOF) {
         if (word != "") 
             words.push_back(word);
         break;
    }
    word += cline[i];
}

Don't know if the above is 100% correct, but you get the idea. Just scan through the string and do exactly what you want.

[anon]

Exactly what word do you want to index? The first? Entire line except first word? All words?

If it is the first word that you want, perhaps you need something like:

create.addWord(line.substr(0, position), PageNo);

If you want to split the string entirely up into words, then perhaps boost::split can be of help.

[Todd88]

Guessing time!

string line;
int start = line.find(' ', 0) + 1;
int end = line.find(' ', start);
string word = line.substr(line, start, end - start);

to get a word from the beginning. My guess it that you foget the +1? Or you put end as the second parameter in substr?

Here is a more C way (if you prefer):

char* cline = line.c_str();
string word("");
vector<string> words;
bool start = false;
for (int i = 0; i < line.size; ++i)
{
    if (cline[i] == ' ') {
              if (word != "") 
                   words.push_back(word);
              word = "";
         }
    }
    if (cline[i] == \n || cline == EOF) {
         if (word != "") 
             words.push_back(word);
         break;
    }
    word += cline[i];
}

Don't know if the above is 100% correct, but you get the idea. Just scan through the string and do exactly what you want.

Alright that works but how to I move through the line instead of just getting the start of every line?

[Todd88]

this prints out the same letter like 50 times too.

[Todd88]

Exactly what word do you want to index? The first? Entire line except first word? All words?

If it is the first word that you want, perhaps you need something like:

create.addWord(line.substr(0, position), PageNo);

If you want to split the string entirely up into words, then perhaps boost::split can be of help.

I want to index the entire line if the word is > 3 letters.

[C_ntua]

OK. Here is a working code (compiled and run it with notepad+command prompt just for you)

#include <iostream>
#include <string>
#include <vector>
using namespace std;

int main(){
string line("hey dude dad");
const char* cline = line.c_str();
string word("");
vector<string> words;
int size = line.size();
for (int i = 0; ;++i)
{
    if (cline[i] == ' ') {
              if (word != "") 
                   words.push_back(word);
              word = "";
              continue;
    }
    if (cline[i] == '\n' || cline[i] == '\0') {
         if (word != "") 
             words.push_back(word);
         break;
    }
    word += cline[i];
}
cout << "WORDS" << endl;
for (int i = 0; i < words.size(); ++i)
     cout << words[i] << endl;
}

The previous one had a lot of bugs. Some obvious some not. But I said it wasn't working.

A C++ version:

#include <iostream>
#include <string>
#include <vector>
using namespace std;

int main(){
string line("hey dude dad ");
string word;
vector<string> words;
int start = 0, end;
while ((end = line.find(' ', start)) != string::npos)
{
   word = line.substr(start, end - start);
   if (word != "")
       words.push_back(word);
   start = end + 1;
}
word = line.substr(start, line.size() - start);
if (word != "")
   words.push_back(word);
cout << "WORDS" << endl;
for (int i = 0; i < words.size(); ++i)
     cout << words[i] << endl;
}

[Todd88]

OK. Here is a working code (compiled and run it with notepad+command prompt just for you)

#include <iostream>
#include <string>
#include <vector>
using namespace std;

int main(){
string line("hey dude dad");
const char* cline = line.c_str();
string word("");
vector<string> words;
int size = line.size();
for (int i = 0; ;++i)
{
    if (cline[i] == ' ') {
              if (word != "") 
                   words.push_back(word);
              word = "";
              continue;
    }
    if (cline[i] == '\n' || cline[i] == '\0') {
         if (word != "") 
             words.push_back(word);
         break;
    }
    word += cline[i];
}
cout << "WORDS" << endl;
for (int i = 0; i < words.size(); ++i)
     cout << words[i] << endl;
}

The previous one had a lot of bugs. Some obvious some not. But I said it wasn't working.

A C++ version:

#include <iostream>
#include <string>
#include <vector>
using namespace std;

int main(){
string line("hey dude dad ");
string word;
vector<string> words;
int start = 0, end;
while ((end = line.find(' ', start)) != string::npos)
{
   word = line.substr(start, end - start);
   if (word != "")
       words.push_back(word);
   start = end + 1;
}
word = line.substr(start, line.size() - start);
if (word != "")
   words.push_back(word);
cout << "WORDS" << endl;
for (int i = 0; i < words.size(); ++i)
     cout << words[i] << endl;
}

YOU ARE THE MAN!!!! Thanks so much!

[laserlight]

hmm... but if your definition of "word" just means "a group of consecutive non-whitespace characters", then a stringstream will suffice:

#include <iostream>
#include <string>
#include <vector>
#include <sstream>

int main()
{
    using namespace std;

    string line("hey dude dad ");
    vector<string> words;

    stringstream ss(line);
    string word;
    while (ss >> word)
    {
        words.push_back(word);
    }

    for (vector<string>::size_type i = 0, size = words.size(); i != size; ++i)
    {
        cout << words[i] << endl;
    }
}

[C_ntua]

Look at this also if you want http://www.cprogramming.com/faq/cgi-...&id=1044780608

So:
1) strinstreams for whitespace separated words
2) string functions for other separators
3) a char by char scan (good old C style)

In your case you will probably want option 2 or 3, since you have something more general. Whatever seems more easy for you.

I can tell you right away that if you want words to be separated also with , ., !, < etc etc then you could add some || to my C style code. And then you can read a whole file (NOT including the EOF) and have a vector with all the words

[laserlight]

I can tell you right away that if you want words to be separated also with , ., !, < etc etc then you could add some || to my C style code.

By changing your "more" C++ version to use find_first_of() instead of find(), something similiar can be applied.

[prog-bman]

You can also use getline on the stream to build a series of word.

#include <iostream>
#include <vector>
#include <string>
#include <sstream>

int main()
{
	//Will be used to determine how to split words up.
	const char WordDeliminter = ' ';

	//
	//Build the word string and load it into the string stream
	//
	std::string wordString = "Hello how are you today?";
	std::stringstream wordStream(wordString);

	//Will be used to hold all the words.
	std::vector<std::string> wordContainer;

	//
	//Use getline with the stream to get all words seperated by space. Add the words to the vector.
	//
	std::string word;
	while(std::getline(wordStream, word, WordDeliminter)){
		wordContainer.push_back(word);
	}//while

	//
	//Loop through and output all the words.
	//
	for(std::vector<std::string>::size_type i = 0; i < wordContainer.size(); i++){
		std::cout<<wordContainer[i]<<std::endl;
	}//for

	std::cin.get();
	return 0;
}