Split string up into single words

As some of you well know (since I have been asking so many questions in the forum) I am creating an HTML book given a simple text file as input. I am also creating an index of certain terms in the book. Right now I have the book output implemented and am now starting on the index portion. I have this loop right here that takes each line and adds the line to a vector:

Code:

---

Page current;
      current.setPageNum(PageNo);

      for(int lineNumber = 0; lineNumber < MAX_LINES_PER_PAGE; lineNumber++)
      {
          string line = book.readLine();
          current.addLine(line);
      }

      current.output(out, book.theTitle, pageNo, nextPageNo, prevPageNo);

---

Now I need to put a function in here that basically splits the "line" variable up into single words then takes those words and passes them to my index function named "addWord." I have tried using substr and doing line.find(' ') and it gives runtime errors. Any help would be appreciated.

It would help quite a bit if you posted your code that actually does the splitting and failing...

--
Mats

Okay here is the failing code:

Code:

---

Page current;
      Index create;
      current.setPageNum(PageNo);

      for(int lineNumber = 0; lineNumber < MAX_LINES_PER_PAGE; lineNumber++)
      {
          string line = book.readLine();
          current.addLine(line);
          position = line.find(' ');
          create.addWord(line.substr(position), PageNo);
      }

      current.output(out, book.theTitle, pageNo, nextPageNo, prevPageNo);

---

What if a page is shorter than MAX_LINES_PER_PAGE? Does readLine fail in a nice way?

Guessing time!

Code:

---

string line;
int start = line.find(' ', 0) + 1;
int end = line.find(' ', start);
string word = line.substr(line, start, end - start);

---

to get a word from the beginning. My guess it that you foget the +1? Or you put end as the second parameter in substr?

Here is a more C way (if you prefer):

Code:

---

char* cline = line.c_str();
string word("");
vector<string> words;
bool start = false;
for (int i = 0; i < line.size; ++i)
{
    if (cline[i] == ' ') {
              if (word != "")
                  words.push_back(word);
              word = "";
        }
    }
    if (cline[i] == \n || cline == EOF) {
        if (word != "")
            words.push_back(word);
        break;
    }
    word += cline[i];
}

---

Don't know if the above is 100% correct, but you get the idea. Just scan through the string and do exactly what you want.

Exactly what word do you want to index? The first? Entire line except first word? All words?

If it is the first word that you want, perhaps you need something like:

Code:

---

create.addWord(line.substr(0, position), PageNo);

---

If you want to split the string entirely up into words, then perhaps boost::split can be of help.

Quote:

---

Originally Posted by C_ntua

Guessing time!

Code:

---

string line;
int start = line.find(' ', 0) + 1;
int end = line.find(' ', start);
string word = line.substr(line, start, end - start);

---

to get a word from the beginning. My guess it that you foget the +1? Or you put end as the second parameter in substr?

Here is a more C way (if you prefer):

Code:

---

char* cline = line.c_str();
string word("");
vector<string> words;
bool start = false;
for (int i = 0; i < line.size; ++i)
{
    if (cline[i] == ' ') {
              if (word != "")
                  words.push_back(word);
              word = "";
        }
    }
    if (cline[i] == \n || cline == EOF) {
        if (word != "")
            words.push_back(word);
        break;
    }
    word += cline[i];
}

---

Don't know if the above is 100% correct, but you get the idea. Just scan through the string and do exactly what you want.

---

Alright that works but how to I move through the line instead of just getting the start of every line?

this prints out the same letter like 50 times too.

Quote:

---

Originally Posted by anon

Exactly what word do you want to index? The first? Entire line except first word? All words?

If it is the first word that you want, perhaps you need something like:

Code:

---

create.addWord(line.substr(0, position), PageNo);

---

If you want to split the string entirely up into words, then perhaps boost::split can be of help.

---

I want to index the entire line if the word is > 3 letters.

OK. Here is a working code (compiled and run it with notepad+command prompt just for you)

Code:

---

#include <iostream>
#include <string>
#include <vector>
using namespace std;

int main(){
string line("hey dude dad");
const char* cline = line.c_str();
string word("");
vector<string> words;
int size = line.size();
for (int i = 0; ;++i)
{
    if (cline[i] == ' ') {
              if (word != "")
                  words.push_back(word);
              word = "";
              continue;
    }
    if (cline[i] == '\n' || cline[i] == '\0') {
        if (word != "")
            words.push_back(word);
        break;
    }
    word += cline[i];
}
cout << "WORDS" << endl;
for (int i = 0; i < words.size(); ++i)
    cout << words[i] << endl;
}

---

The previous one had a lot of bugs. Some obvious some not. But I said it wasn't working.

A C++ version:

Code:

---

#include <iostream>
#include <string>
#include <vector>
using namespace std;

int main(){
string line("hey dude dad ");
string word;
vector<string> words;
int start = 0, end;
while ((end = line.find(' ', start)) != string::npos)
{
  word = line.substr(start, end - start);
  if (word != "")
      words.push_back(word);
  start = end + 1;
}
word = line.substr(start, line.size() - start);
if (word != "")
  words.push_back(word);
cout << "WORDS" << endl;
for (int i = 0; i < words.size(); ++i)
    cout << words[i] << endl;
}

---

Quote:

---

Originally Posted by C_ntua

OK. Here is a working code (compiled and run it with notepad+command prompt just for you)

Code:

---

#include <iostream>
#include <string>
#include <vector>
using namespace std;

int main(){
string line("hey dude dad");
const char* cline = line.c_str();
string word("");
vector<string> words;
int size = line.size();
for (int i = 0; ;++i)
{
    if (cline[i] == ' ') {
              if (word != "")
                  words.push_back(word);
              word = "";
              continue;
    }
    if (cline[i] == '\n' || cline[i] == '\0') {
        if (word != "")
            words.push_back(word);
        break;
    }
    word += cline[i];
}
cout << "WORDS" << endl;
for (int i = 0; i < words.size(); ++i)
    cout << words[i] << endl;
}

---

The previous one had a lot of bugs. Some obvious some not. But I said it wasn't working.

A C++ version:

Code:

---

#include <iostream>
#include <string>
#include <vector>
using namespace std;

int main(){
string line("hey dude dad ");
string word;
vector<string> words;
int start = 0, end;
while ((end = line.find(' ', start)) != string::npos)
{
  word = line.substr(start, end - start);
  if (word != "")
      words.push_back(word);
  start = end + 1;
}
word = line.substr(start, line.size() - start);
if (word != "")
  words.push_back(word);
cout << "WORDS" << endl;
for (int i = 0; i < words.size(); ++i)
    cout << words[i] << endl;
}

---

---

YOU ARE THE MAN!!!! Thanks so much! :)

hmm... but if your definition of "word" just means "a group of consecutive non-whitespace characters", then a stringstream will suffice:

Code:

---

#include <iostream>
#include <string>
#include <vector>
#include <sstream>

int main()
{
    using namespace std;

    string line("hey dude dad ");
    vector<string> words;

    stringstream ss(line);
    string word;
    while (ss >> word)
    {
        words.push_back(word);
    }

    for (vector<string>::size_type i = 0, size = words.size(); i != size; ++i)
    {
        cout << words[i] << endl;
    }
}

---

Look at this also if you want http://www.cprogramming.com/faq/cgi-...&id=1044780608

So:
1) strinstreams for whitespace separated words
2) string functions for other separators
3) a char by char scan (good old C style)

In your case you will probably want option 2 or 3, since you have something more general. Whatever seems more easy for you.

I can tell you right away that if you want words to be separated also with , ., !, < etc etc then you could add some || to my C style code. And then you can read a whole file (NOT including the EOF) and have a vector with all the words

Quote:

---

Originally Posted by C_ntua

I can tell you right away that if you want words to be separated also with , ., !, < etc etc then you could add some || to my C style code.

---

By changing your "more" C++ version to use find_first_of() instead of find(), something similiar can be applied.

You can also use getline on the stream to build a series of word.

Code:

---

#include <iostream>
#include <vector>
#include <string>
#include <sstream>

int main()
{
        //Will be used to determine how to split words up.
        const char WordDeliminter = ' ';

        //
        //Build the word string and load it into the string stream
        //
        std::string wordString = "Hello how are you today?";
        std::stringstream wordStream(wordString);

        //Will be used to hold all the words.
        std::vector<std::string> wordContainer;

        //
        //Use getline with the stream to get all words seperated by space. Add the words to the vector.
        //
        std::string word;
        while(std::getline(wordStream, word, WordDeliminter)){
                wordContainer.push_back(word);
        }//while

        //
        //Loop through and output all the words.
        //
        for(std::vector<std::string>::size_type i = 0; i < wordContainer.size(); i++){
                std::cout<<wordContainer[i]<<std::endl;
        }//for

        std::cin.get();
        return 0;
}

---

Quote:

---

Originally Posted by prog-bman

You can also use getline on the stream to build a series of word.

---

However, getline() only allows a single char as the delimiter, hence if a range of chars are needed to act as delimiters, C_ntua's suggestion with my proposed modification would be more applicable... or anon's suggestion of boost::split could be used instead.

Indeed.

I am sure someone could roll there own split function from getline or use the options you said above.

There's also Boost.Tokenizer which might be even more appropriate, since you don't want to get a list of words but each word individually to determine what to do with it.

Sample usage:

Code:

---

#include<iostream>
#include<boost/tokenizer.hpp>
#include <boost/foreach.hpp>
#include<string>

void add_index_entry(const std::string& s, unsigned page_n)
{
    std::cout << "adding: " << s << ": page " << page_n << '\n';
}

int main(){
    std::string line("Oh, this line - unfortunately - contains some punctuation...");
    unsigned page_n = 1;

    boost::tokenizer<> tokens(line);
    BOOST_FOREACH(const std::string& word, tokens) {
        if (word.size() > 3)
            add_index_entry(word, page_n);
    }
}

---

If the aim is not to implement everything yourself, boost - among other things - is rather helpful for string processing (splitting, case insensitive comparisons, etc)

Using boost is cheating :p

Yeah sorry I can't use boost.

Okay I found something. How could I use this:

Code:

---

/* strtok example */
#include <stdio.h>
#include <string.h>

int main ()
{
  char str[] ="- This, a sample string.";
  char * pch;
  printf ("Splitting string \"&#37;s\" into tokens:\n",str);
  pch = strtok (str," ,.-");
  while (pch != NULL)
  {
    printf ("%s\n",pch);
    pch = strtok (NULL, " ,.-");
  }
  return 0;
}

---

This is my attempt at making it work but it doesn't:

Code:

---

void Index::addWord(string word, int pageNumber)
{
    cout << word;
    char * pch;
    pch = &word[0];
    pch = strtok (pch," :(#;[]""\"()!`'?,.-");
    while (pch != NULL)
    {
        //printf ("%s\n",pch);
        pch = strtok (NULL, " :(#;[]""\"()!`'?,.-");
        data.push_back(pch);
    }
}

---

I need to take the passed string, split it into a character array, then store the split words into the vector.

Perhaps the problem is that you are adding data after a call to strtok without checking what it returned. data.push_back should occur where your printf statement is.

Still there is something immoral about gaining non-constant access to a string's internal buffer :)

Another approach is to use std::algorithms - IMO, the iterator interface is a bit more convenient than the indices interface of std::string (no special npos to check for), and you can use predicates:

Code:

---

#include <iostream>
#include <string>
#include <functional>
#include <algorithm>
#include <cctype>
using namespace std;
void addWords(const string& line, int )
{
    cout << line << '\n';
    std::string::const_iterator word_start = line.begin(), word_end = line.begin();
   
    while (
        word_start = std::find_if(word_end, line.end(), std::ptr_fun<int, int>(std::isalnum)),
        word_end = std::find_if(word_start, line.end(), std::not1(std::ptr_fun<int, int>(std::isalnum))),
        word_start != word_end
        ) {
        std::cout << std::string(word_start, word_end) << '\n';
    }
}

int main()
{
    std::string s("- This is an example string (which contains punctuation)! -");
    addWords(s, 10);
}

---

(Pardon the somewhat unorthodox use of comma operator :) )