Best way to write a file

[afflictedd2]

Hi everyone,

I'm wondering if anyone knows the optimal way to write a very big tab-sepparated file with data that consists mostly of ints and floats. This ints and floats are really rows of data that will be input in a database. What I'm trying to do is to use a string to save like 500 rows and then I write to disk. It's rather slow and takes like 1.18 minutes to write all of this to a file. Anyone know a way to optimize better?

  
 std::string to_file = "";

  for (int i=0; i<numRecords; ++i)  {    
    
    for (ssize_t fldidx = 0; fldidx <  hdr.Length(); ++fldidx)  {
       
      switch (field_type) {
      case UNDEFINED:
        continue;
        
      case STRING:
        continue;
        
      case CHAR: {
        continue;
      }
      case INT: {
        const int field_addr = field.mNumVal.i;
        char result[100];
        sprintf( result, "%d", field_addr );   
        to_file += result;
        break;
      }
        
      case FLOAT: {
        const float field_addr = field.mNumVal.f;
        char result[100];
        sprintf( result, "%f", field_addr );
        to_file +=  result;
        break;
      }
        
      case DOUBLE: {
        const double field_addr = field.mNumVal.d;
        char result[100];
        sprintf( result, "%f", field_addr );
        to_file += result;
        break; 
      }
        
      case LONG: {
        const long field_addr = field.mNumVal.l;
        char result[100];
        sprintf( result, "%d", field_addr );
        to_file += result;
	break;	      
      }
      }
      
      if (fldidx != hdr.Length()-1)
        to_file += "\t";
      else 
        to_file += "\n";

     }
    if ( (i % bulk  == 0) || i == (numRecords-1) ) {
      fout << to_file;
      to_file = "";
    }

[matsp]

Why not use the normal file output directly? Like this:

...
      case INT: {
        fout << field.mNumVal.i;
        break;
      }
        
      case FLOAT: {
        fout << field.mNumVal.f;
        break;
      }
...

Or have you already tried that?

--
Mats

[elninio]

Use HDF (hierarchial data format) lib! I just finished using it, its quite rapid. It also compresses your file.

[Hunter2]

I remember someone posting benchmark results for various methods of file writing, at one point. Bottom line was:

ofstream slower than fwrite() *much* slower than direct API calls.

I'm no expert, but I believe if you're hardcore enough, you can use unbuffered file I/O (link here) to make a huge difference in speed, assuming you're careful and get it right.

Another thing to experiment with, would be to use SetEndOfFile() to allocate the entire file immediately, so that your successive write operations don't have to keep extending it. I haven't tried it, so I couldn't tell you if it actually works or not.

[Cactus_Hugger]

I remember someone posting benchmark results for various methods of file writing, at one point. Bottom line was:

ofstream slower than fwrite() *much* slower than direct API calls.

Hmm. Did my own tests:

C:\...\Documents\Code\cs_ex\filespeed>filespeed stdio
Doing stdio output...
Std. I/O took 561ms.

C:\...\Documents\Code\cs_ex\filespeed>filespeed ofstream
Doing ofstream output...
ofstream took 1014ms.

C:\...\Documents\Code\cs_ex\filespeed>filespeed win32
Doing Win32 output...
ofstream took 951ms.

All files have their handles closed before the time is stopped. Each test is run after the disk I/O from the previous settles, and each writes 100MB to disk. (Which is greater than most files that get written, and anything much bigger might run into disk I/O speeds, in which case whatever API you call will become irrelevant really quickly.) Listed times are the best out of several trials.

Oddly, Win32 did not win. Also, if I told the other too to fflush() and .flush(), it did not make a difference. If I called FlushFileBuffers() on the Win32 handle, Windows waited until the data was truely flushed, ie - on disk! (My harddisk cannot write 100 MB in <2 sec)

Code is attached, for those interested.

Now, for reading at least, I have found that requesting more data at a time works better. Don't fread() 1 byte at a time. Additionally, serializing the data before the actual I/O (such as a call to sprintf() or something) takes some time, and might be a bottleneck. Your profiler is your friend.

[Hunter2]

For interest's sake, I compiled it with FILE_FLAG_NO_BUFFERING on the win32 version. Maybe I did something funny, but it took roughly 30 times longer than any of the other tests So I guess disregard my previous post.

P.S. I also noticed that each test varied hugely; for example, the fwrite() test went from 2.3s to ~9s between two tests, and similar results with win32. On average though, fwrite() was faster than win32 and win32 was faster than fstream.

P.P.S. SetEndOfFile() does seem to improve performance somewhat:

SetFilePointer(hFile, OUTFILE_SIZE, NULL, FILE_BEGIN);
SetEndOfFile(hFile);
SetFilePointer(hFile, 0, NULL, FILE_BEGIN);

Note that this may only help if the file doesn't already exist.

[phantomotap]

Std. I/O took 561ms.
ofstream took 1014ms.
ofstream took 951ms.

O_o

Yea... you need a better C++ library implementation. The difference between the three should be negligible for such a small file.

(You also need to fix the bug in your code that prints 'ofstream' for the Windows API case.)

Soma