eliminating \n and \r from string [boost::iostreams]

Hello

I use boost::iostreams input filter stream to filter the input file.

Heres the example code:

Code:

using namespace std;
using namespace boost::iostreams;
namespace io = boost::iostreams;

class long_line_counter : public boost::iostreams::line_filter {
public:
 explicit long_line_counter(int max_length = 80)
  : max_(max_length), count_(0) { }

 int count() const { return count_; }

 vector<string> m_list;

private:
 std::string do_filter(const std::string& line) {
  if (line.size() < max_) m_list.push_back(line);
  ++count_;
  return line;
 }

 int max_;
 int count_;
};

int main() {
 try {
  filtering_istream fin;
  long_line_counter llc(80);

  file_source fss("outest.fil", ios::in | ios::binary);

  fin.push(boost::ref(llc));
  fin.push(boost::ref(fss));

  char buf[4012];
  while ( fin.read(buf, 4012) ) {
  }

  std::cout << llc.count() << "\n";
  std::cout << llc.mail_list.size() << "\n";
 }
 catch (exception &e) {
  cout << "exception: " << e.what() << std::endl;
 }

 return 0;
}

Now the problem is that the boost::iostreams line_filter will pass the line (string) along with the \r and \n symbols to the do_filter() function.. So if it reads line "customer: Steve Porter\r\n" it will pass the same string to the function instead of string without '\r\n'.

I've been wondering whats the most efficient way to eliminate/remove '\r' and '\n' symbols from this std::string.

Should I do char by char check, or maybe use regex? Is there any more efficient way since I have to do this many thousand times in a short period.

Thanks for help!

Seems like you want a trim_right. String_Algo has one.

I'm confused. From my brief reading of the Boost documentation, it looks like it should be removing the line terminator character already. So maybe you just have the line termination mode set incorrectly.

Originally Posted by brewbuck

I'm confused. From my brief reading of the Boost documentation, it looks like it should be removing the line terminator character already. So maybe you just have the line termination mode set incorrectly.

You are right.

Thats how I open the stream:

Code:

line_filter m_line_filter; //filter class that derives from line_filter
boost::iostreams::filtering_istream m_filter_stream;
m_filter_stream.push(boost::ref(m_line_filter));
m_filter_stream.push(boost::iostreams::file_source("file.txt", std::ios::in | std::ios::binary));

Not sure why it doesnt remove line termintator '\r' although I open it in binary mode. Any idea?

Thanks a lot!

In binary mode the line terminator is just '\n' not "\r\n". Multi-character line terminators are a feature of NON-binary access modes.

Originally Posted by brewbuck

In binary mode the line terminator is just '\n' not "\r\n". Multi-character line terminators are a feature of NON-binary access modes.

I meant with the code above it will still pass "someline\r" instead of "someline" to do_filter() function.

Yes, because you're in binary mode.

Originally Posted by CornedBee

Yes, because you're in binary mode.

Which way should I open it then to get "someline" instead of "someline\r"?

In text mode, which is the default.

Originally Posted by CornedBee

In text mode, which is the default.

Won't that affect preformance as far as I know?

No. The scanning for LF already happens because your filter goes line by line. Kicking out a CR ought to be quite unnoticeable.

Opening a text file in binary mode can have undesireable sideeffects if there is a BOM (Byte Order Mark) present.

I have always thought that the right way on windows is to open a text file in binary mode because of preformance and other issues.

I've yet to see a narrow text mode that cares about the BOM in any way.

l2u: Well, you can always trim the thing yourself.

Which way do you think its better? Leave the text mode do the job or do it by yourself?

Whats the flag to open the file (file_stream) in text mode? I couldnt find it on the boost::iostreams documentation website (I tried ios::text).