Thread: writing a file from buffer

Originally Posted by mariano_donati

2.
To divide buffer into smaller parts, of 1024 bytes each.
This, requires to copy 1024 bytes from original buffer to a temporal one, and then
copy it to the image like:

Code:

             int i;
             int copyStartPosition;           /* Position of original buffer where copy begins */
             for ( i = 0 ; i < 1024 ; i++ )
             {
                   tmpBuffer[i] = originalBuffer[i+copyStartPosition];
             }
             fwrite ( tmpBuffer , 1 , 1024 , filePointer );

Here i'm not counting on not all packages will have a regular size of 1024 bytes, the last
one could not have it.

Am i in the right way to solve this problem?

Well you probably need to move fwrite within the loop. Also you will need to remove the magic 1024, because at the very last frame of your buffer, if it is not 1024 then you will access not owned memory and write garbage to the file. You could do it as follows.
Have a variable bytesWritten, initialized to 0. Then before the fwrite you would have a variable bytesToWrite. Check whether bufferSize-bytesWritten>=1024 and if not then bytesToWrite is bufferSize-bytesWritten, else bytesToWrite is 1024. Do the fwrite with the custom size and update your variables appropriately. You can use pointer arithmetic with the bytesWritten variable on tmpBuffer or index it with that variable to find the next frame start position, whichever is your preferable style.

Thanks. This is what i've got so far. I haven't tested it yet, because i need to finish something else before i can do it properly.

Code:

void writeImageFromBuffer ( const char* filename , const char* buffer , unsigned long length )
{
    const int REGULARPACKAGE_SIZE = 1024;
    FILE *image;                                  /* Pointer to file */
    unsigned long bytesWritten;                   /* Bytes written so far in the buffer */
    int bytesToWrite;                             /* Bytes to write to file */                            
    char *tmpBuffer;                              /* Temporary buffer */
    
    bytesWritten = 0;

    image = fopen ( filename , "wb" );    

    // allocate tmpBuffer
    tmpBuffer = (char *) malloc ( REGULARPACKAGE_SIZE );
    
    while ( bytesWritten < length )
    {
        if ( length - bytesWritten >= REGULARPACKAGE_SIZE )  /* not last frame */
        {
            bytesToWrite = REGULARPACKAGE_SIZE;
        }
        else                                                 /* last frame */
        {
            bytesToWrite = length - bytesWritten;            
            // reallocate tmpBuffer to its adecuate size
            tmpBuffer = (char *) realloc ( tmpBuffer , bytesToWrite );
        }
        // copy original buffer <bytesToWrite> elements to tmpBuffer
        tmpBuffer = &buffer[bytesWritten];
        // write tmpBuffer to file
        fwrite ( tmpBuffer , 1 , bytesToWrite , image );
        // just upgrade the var
        bytesWritten += bytesToWrite;
    } 
   
    fclose ( image );   
}

tmpBuffer = &buffer[bytesWritten];

you just leaking allocated memory - remove all malloc/realloc stuff you do not use anyway

Originally Posted by esbo

I would just write the whole lot on in one go.
I currently write about 10 megabyes in that way, the time to do it is negligible.

I was writting some bit read/write routines and needed comparison on my machine of the throughput so i did some timing on the stdio.h routines. It was true that on WinXP it did matter which size i picked for a buffer splitter and i would top writting a 8MB file at 256k buffer, using fwrite. On Linux, i had constant speed of 110MB/sec as long as the buffer was smaller than the files size.
The fact is that it might be useful to read/write in chunks sometimes...

Mario: You don't need to do so complex things when writing your buffer. Think for a second that the variable bytesWritten, can be used to index the tmpBuffer and you need no realloc, or copies or nothing. Just send tmpBuffer[bytesWritten] to fwrite! It will work fine, and elegantly.

Originally Posted by mariano_donati

What do you mean with leaking? my english becomes too poor sometimes.
With malloc/realloc statements i thought that i was reserving the exact number of bytes i need. Is it unnecesary?.

leaking memory - you allocate some memory from OS, store the address of the beginning and then - loose this address, so you cannot access the memory and cannot free it, memory stays allocated, but you do not have access to it

your

tmpBuffer = &buffer[bytesWritten];

assignment does not copies the original buffer contents anywhere, you just move the temppointer along the buffer to access different parts of the original buffer (as you should) so no additional storage is required.

and then you write the part of the original buffer to disk once again - without unneded copy to the temp buffer

IF you already have a buffer, writing it in one go is usually most efficient. But you should be careful when reading a big file, since reading a GB file into memory will probably hurt performance on many machines.