Thread: Writing std::string in binary mode

The problem:
Reading/Writing a binary data file that contains several variable length strings as well as binary data.

Code:

std::ofstream("CraftTest.dat",std::ios_base::binary);
...
...
//Write the string and its null terminator
outFile << m_CraftName.c_str();

//Skip past the null
outFile.seekp(1,std::ios_base::cur);

//Write binary data
outFile.write((char *)&m_CraftData,sizeof(BinaryCraftData));

What happens here is that the null terminator is written to disk, however, I must do the seekp because the stream position is not incremented via the terminating null. So without the seekp when the next write happens it blows away the null terminator.

However when reading the data file back in nothing works correctly except reading in the string. After that point none of the data following the string is correct even if I do a seekg() to get past the null.

How can I read in this variable length string without writing the length to the file? I would prefer to do this without having to manually read in each character until I encounter a null. My initial thoughts were that if I wrote the string to disk correctly with the null that this would work.

Code:

std::ifstream("CraftTest.dat",std::ios_base::binary);
...
...
//Read the string and it's null terminator
inFile >> m_CraftName;

//Read in binary data
inFile.read((char *)&m_CraftData,sizeof(BinaryCraftData));

This reads in the string correctly but now every read after this point is pure garbage as if my file position is off causing me to get some very weird numbers. Just in case the string read was not reading the null I tried to do a seekg() after it to skip the null and this still did not fix anything.

If I had just used C I/O I would have been done by now.

Originally Posted by Bubba

What happens here is that the null terminator is written to disk, however, I must do the seekp because the stream position is not incremented via the terminating null. So without the seekp when the next write happens it blows away the null terminator.

Are you sure the null terminator is written to disk? I had the impression that it was not, and a quick check with a test program and a hex editor appears to confirm that.

Originally Posted by Bubba

Only if you use c_str().

I do not think so. Try this test program:

Code:

#include <iostream>
#include <fstream>
#include <string>

int main()
{
    using namespace std;
    ofstream out("out.txt", ios_base::binary);
    std::string str = "hello world!";
    out << str.c_str();
    //out << '\0';
    out.close();

    ifstream in("out.txt", ios_base::binary);
    int ch;
    size_t count = 0;
    while ((ch = in.get()) != EOF)
    {
        ++count;
    }
    cout << count << endl;
}

I reason that if the null terminator is written to disk, the count should be 13. However, I get a count of 12. Uncommenting the line with out << '\0' gives a count of 13, so apparently the null terminator is not ignored, as expected. Therefore, I conclude that the null terminator was not written to disk contrary to what you stated.

Right, so my guess is that if you actually wrote out the null terminator in your earlier example, perhaps the problem would be fixed without having to write the size to disk. The use of c_str() would also be unnecessary in that case.

EDIT:
Actually, maybe not. The null terminator is not regarded as whitespace, so inFile >> m_CraftName would read it in as well. Perhaps a better solution is to write a space after writing m_CraftName, then when reading ignore that space and go on to read the binary data.

As usual it's not as simple as it sounds b/c there are a lot of gotchas. The only way I got it to work was from an idea off of gamedev which was to write 2 specific functions to read and write STL strings to/from disk. It works very well but requires the string size be written to disk. I made the functions part of a Serializeable base class so every object that needs serialization has the functions available to them. Next time I read/write binary data I think I will move away from streams since they are primarily for text files.

All this came about because I moved my data files over to binary since using XML became way to cumbersome, ugly, and did not make the loading/saving process any easier. IE: Using XML bought me nothing.

Originally Posted by Bubba

I moved my data files over to binary because using XML became way to cumbersome, ugly, and did not make the loading/saving process any easier. For this I just whipped up a console app in about 10 minutes and would have been done had I not ran into this string issue.

Just a thought, but have you considered using SQLite? One of its use cases is as a replacement for fopen(), i.e., to replace a custom file format with a database schema but without needing the use of a database server.

Don't mix formatted I/O with raw I/O. In other words, if on a stream you use write(), don't use << on the same stream.

To write a std::string to a binary stream, first write() the size, then write() the data(), in one block. For reading, read() the size, resize() the string and read() the data block to &s[0]. Note, however, that this is only guaranteed to work in C++0x; C++03 gives you no guarantees that the data of std::string is contiguous and writeable. If you want a proper C++03 method of doing it, read() the size, resize() a std::vector, read() to the vector and range-assign from the vector to the string.

Code:

std::string::size_type sz = s.size();
stream.write(reinterpret_cast<char*>(&sz), sizeof(std::string::size_type));
stream.write(s.data(), sz);

// Reading common:
stream.read(reinterpret_cast<char*>(&sz), sizeof(std::string::size_type));

// C++03 with C++0x semantics:
s.resize(sz);
stream.read(&s[0], sz);

// C++0x:
s.resize(sz);
stream.read(s.data(), sz);

// C++03:
std::vector<char> buf(sz);
stream.read(&buf[0], sz);
s.assign(buf.begin(), buf.end());

Would it work better if you used a std::vector<unsigned char> when reading/writing to disk?
std::string is meant for strings, not binary data...

This thread is nearly two years old. I think Bubba has taken care of it by now.