Split string up into single words

Quote:

---

Originally Posted by prog-bman

You can also use getline on the stream to build a series of word.

---

However, getline() only allows a single char as the delimiter, hence if a range of chars are needed to act as delimiters, C_ntua's suggestion with my proposed modification would be more applicable... or anon's suggestion of boost::split could be used instead.

Indeed.

I am sure someone could roll there own split function from getline or use the options you said above.

There's also Boost.Tokenizer which might be even more appropriate, since you don't want to get a list of words but each word individually to determine what to do with it.

Sample usage:

Code:

---

#include<iostream>
#include<boost/tokenizer.hpp>
#include <boost/foreach.hpp>
#include<string>

void add_index_entry(const std::string& s, unsigned page_n)
{
    std::cout << "adding: " << s << ": page " << page_n << '\n';
}

int main(){
    std::string line("Oh, this line - unfortunately - contains some punctuation...");
    unsigned page_n = 1;

    boost::tokenizer<> tokens(line);
    BOOST_FOREACH(const std::string& word, tokens) {
        if (word.size() > 3)
            add_index_entry(word, page_n);
    }
}

---

If the aim is not to implement everything yourself, boost - among other things - is rather helpful for string processing (splitting, case insensitive comparisons, etc)

Using boost is cheating :p

Yeah sorry I can't use boost.

Okay I found something. How could I use this:

Code:

---

/* strtok example */
#include <stdio.h>
#include <string.h>

int main ()
{
  char str[] ="- This, a sample string.";
  char * pch;
  printf ("Splitting string \"&#37;s\" into tokens:\n",str);
  pch = strtok (str," ,.-");
  while (pch != NULL)
  {
    printf ("%s\n",pch);
    pch = strtok (NULL, " ,.-");
  }
  return 0;
}

---

This is my attempt at making it work but it doesn't:

Code:

---

void Index::addWord(string word, int pageNumber)
{
    cout << word;
    char * pch;
    pch = &word[0];
    pch = strtok (pch," :(#;[]""\"()!`'?,.-");
    while (pch != NULL)
    {
        //printf ("%s\n",pch);
        pch = strtok (NULL, " :(#;[]""\"()!`'?,.-");
        data.push_back(pch);
    }
}

---

I need to take the passed string, split it into a character array, then store the split words into the vector.

Perhaps the problem is that you are adding data after a call to strtok without checking what it returned. data.push_back should occur where your printf statement is.

Still there is something immoral about gaining non-constant access to a string's internal buffer :)

Another approach is to use std::algorithms - IMO, the iterator interface is a bit more convenient than the indices interface of std::string (no special npos to check for), and you can use predicates:

Code:

---

#include <iostream>
#include <string>
#include <functional>
#include <algorithm>
#include <cctype>
using namespace std;
void addWords(const string& line, int )
{
    cout << line << '\n';
    std::string::const_iterator word_start = line.begin(), word_end = line.begin();
   
    while (
        word_start = std::find_if(word_end, line.end(), std::ptr_fun<int, int>(std::isalnum)),
        word_end = std::find_if(word_start, line.end(), std::not1(std::ptr_fun<int, int>(std::isalnum))),
        word_start != word_end
        ) {
        std::cout << std::string(word_start, word_end) << '\n';
    }
}

int main()
{
    std::string s("- This is an example string (which contains punctuation)! -");
    addWords(s, 10);
}

---

(Pardon the somewhat unorthodox use of comma operator :) )