read full file to string in one go

[YocR]

as a casual c++ user i recently stumbled upon a problem, which i didn't expect to be one at all. something as trivial as reading a file at once turned out to be not so trivial after all in c++.

in c i'd simply use fread and be done with it. for c++ however there doesn't seem to be such a thing. i had a look around here and searched for "read file" as well as across the net. i found a bunch of approaches and most of them are summarized there -> Insane Coding: How to read in a file in C++.

the interesting ones are:

std::string get_file_contents(const char *filename)
{
  std::ifstream in(filename, std::ios::in | std::ios::binary);
  if (in)
  {
    std::string contents;
    in.seekg(0, std::ios::end);
    contents.resize(in.tellg());
    in.seekg(0, std::ios::beg);
    in.read(&contents[0], contents.size());
    in.close();
    return(contents);
  }
  throw(errno);
}

and

std::string get_file_contents(const char *filename)
{
  std::ifstream in(filename, std::ios::in | std::ios::binary);
  if (in)
  {
    std::ostringstream contents;
    contents << in.rdbuf();
    in.close();
    return(contents.str());
  }
  throw(errno);
}

the first one looked good at first. "in.read(&contents[0], contents.size());" works, but only for the raw content. the counters of the string however are not updated and hence size() for example reports 0 and many member functions such as find() don't work. so that one was off the table.

the second one (and in fact all others i found) seems to require a copy to a string and that's where the catch is. my files are text but quite large, as in several gigabytes, so a copy, which doubles the ram usage, is simply not an option. and any line-by-line (getline and alike) based approaches neither of course.


and now i'm wondering, is there really no direct way of getting a text file into a string via the standard library? i remember even roguewave's RWCString had a readFile member function in the 90s.

[Salem]

Seems to work for me, running the code with it's own source file.

$ ls -l bar.cpp
-rw-rw-r-- 1 sc sc 577 Jun 10 06:08 bar.cpp
$ g++ -std=c++11 bar.cpp
$ ./a.out 
#include <iostream>
#include <fstream>
#include <string>
using namespace std;

std::string get_file_contents(const char *filename)
{
  std::ifstream in(filename, std::ios::in | std::ios::binary);
  if (in)
  {
    std::string contents;
    in.seekg(0, std::ios::end);
    contents.resize(in.tellg());
    in.seekg(0, std::ios::beg);
    in.read(&contents[0], contents.size());
    in.close();
    return(contents);
  }
}

int main()
{
  std::string r = get_file_contents("bar.cpp");
  std::cout << r << std::endl;
  std::cout << "Size=" << r.size() << std::endl;
  return 0;
}

Size=577

[YocR]

yeah as a separate function which then requires a copy again. hence the string has everything and size() works. but try it directly:

#include <cstdio>
#include <fstream>
#include <string>
#include <sys/stat.h>

#define LFILE "testfile"

using namespace std;


int main() {
	ifstream in;
	string str;
	struct stat s;

	if (stat(LFILE, &s)) {
		perror("ERROR: couldn't stat logfile");
		exit(1);
	}

	str.reserve(s.st_size + 1);
	in.open(LFILE);
	in.read(&str[0], s.st_size);
	in.close();
	printf("%lu\n", str.size());
	exit(0);
}

the crucial thing in this case is that i cannot afford a copy as mentioned in my first post already.

[Salem]

> my files are text but quite large, as in several gigabytes, so a copy, which doubles the ram usage
Why do you need the whole file in memory at the same time?

How much time is spent processing the data once the file is in memory?
If you're spending minutes in processing, saving seconds on the file I/O is a waste of effort.

[YocR]

so there's indeed no way to read a file into a string directly in c++ ?

[laserlight]

No, it can be done, but the issue is that based on what you wrote, you probably don't want to do that.

[YocR]

you probably don't want to do that.

oh i absolutely want to do that. this is such a basic task, i'm still buffled (pun!) that there's no obvious way to do it.
please do tell!

[laserlight]

Actually, you have already found a number of approaches. The copy issue is a non-issue: the copy will either be elided (the old named return value optimisation), or you can take steps to ensure a move or the use of a reference instead.

[YocR]

move sounds like the best option. how would i ensure a move?

[laserlight]

I would choose your second example (based on memory that it did well in timed tests some time back), and just compile with respect to C++11 or later. I'd suggest inverting the if statement though: use it to throw an exception derived from std::exception instead, then it's clear you do return a string at the end, though this shouldn't matter.

Nonetheless, Salem's point remains: are you really sure you cannot process your input in smaller chunks?

[YocR]

yes sorry, i didn't mention any requirements. it has to work everywhere so c++11 is not an option and exceptions are generally disabled via compiler option.

anyway, my example from post #3 does work after all but with one change: the problem was reserve(). if i use resize() instead it seems fine.

either way, thanks for all replies

[laserlight]

it has to work everywhere so c++11 is not an option

It's 2019: C++11 is likely to be available. If it isn't, then you need to be sure what exactly you're supporting: how can you be sure that C++98 is available?

Anyway, move semantics was introduced in C++11, so if C++11 is not an option, you'll have to do the same thing and depend on copy elision, which should happen since C++98.

exceptions are generally disabled via compiler option.

You're throwing an exception with throw(errno), except that it isn't a usual exception. If you cannot use exceptions, then you need some other way to indicate an error: return an empty string?

anyway, my example from post #3 does work after all but with one change: the problem was reserve(). if i use resize() instead it seems fine.

If you don't want to rely on copy elision, you can use this but with a reference parameter instead, then say, return a bool to indicate success/failure.

[YocR]

you need some other way to indicate an error: return an empty string?

If you don't want to rely on copy elision, you can use this but with a reference parameter instead, then say, return a bool to indicate success/failure.

thanks again but since i don't put this in a function (as post #3 shows) there is no return for me to come up with.
but if i wanna be sure, i can just compare str.size() against s.st_size.

EDIT: even better, with that "&str[0]" way, i can even keep using fread so i don't have to deal with streams at all