Thread: Split string up into single words

Originally Posted by prog-bman

You can also use getline on the stream to build a series of word.

However, getline() only allows a single char as the delimiter, hence if a range of chars are needed to act as delimiters, C_ntua's suggestion with my proposed modification would be more applicable... or anon's suggestion of boost::split could be used instead.

Indeed.

I am sure someone could roll there own split function from getline or use the options you said above.

There's also Boost.Tokenizer which might be even more appropriate, since you don't want to get a list of words but each word individually to determine what to do with it.

Sample usage:

Code:

#include<iostream>
#include<boost/tokenizer.hpp>
#include <boost/foreach.hpp>
#include<string>

void add_index_entry(const std::string& s, unsigned page_n)
{
    std::cout << "adding: " << s << ": page " << page_n << '\n';
}

int main(){
    std::string line("Oh, this line - unfortunately - contains some punctuation...");
    unsigned page_n = 1;

    boost::tokenizer<> tokens(line);
    BOOST_FOREACH(const std::string& word, tokens) {
        if (word.size() > 3)
            add_index_entry(word, page_n);
    }
}

If the aim is not to implement everything yourself, boost - among other things - is rather helpful for string processing (splitting, case insensitive comparisons, etc)

Perhaps the problem is that you are adding data after a call to strtok without checking what it returned. data.push_back should occur where your printf statement is.

Still there is something immoral about gaining non-constant access to a string's internal buffer

Another approach is to use std::algorithms - IMO, the iterator interface is a bit more convenient than the indices interface of std::string (no special npos to check for), and you can use predicates:

Code:

#include <iostream>
#include <string>
#include <functional>
#include <algorithm>
#include <cctype>
using namespace std;
void addWords(const string& line, int )
{
    cout << line << '\n';
    std::string::const_iterator word_start = line.begin(), word_end = line.begin();
    
    while (
        word_start = std::find_if(word_end, line.end(), std::ptr_fun<int, int>(std::isalnum)),
        word_end = std::find_if(word_start, line.end(), std::not1(std::ptr_fun<int, int>(std::isalnum))),
        word_start != word_end
        ) {
        std::cout << std::string(word_start, word_end) << '\n';
    }
}

int main()
{
    std::string s("- This is an example string (which contains punctuation)! -");
    addWords(s, 10);
}

(Pardon the somewhat unorthodox use of comma operator )