writing a file from buffer

This is a discussion on writing a file from buffer within the C Programming forums, part of the General Programming Boards category; I want to know what would be an efficient way of do it. I haven't written any code yet, i ...

  1. #1
    Registered User
    Join Date
    Nov 2007
    Posts
    48

    Post writing a file from buffer

    I want to know what would be an efficient way of do it. I haven't written any code yet, i just have a couple of ideas. Here they go both (just an idea about how code would look like):

    1.
    To write entire buffer in one step like:
    Code:
      
               fwrite ( buffer , 1 , totalBufferSize , filePointer );
    2.
    To divide buffer into smaller parts, of 1024 bytes each.
    This, requires to copy 1024 bytes from original buffer to a temporal one, and then
    copy it to the image like:
    Code:
                 int i;
                 int copyStartPosition;           /* Position of original buffer where copy begins */
                 for ( i = 0 ; i < 1024 ; i++ )
                 {
                       tmpBuffer[i] = originalBuffer[i+copyStartPosition];
                 }
                 fwrite ( tmpBuffer , 1 , 1024 , filePointer );
    Here i'm not counting on not all packages will have a regular size of 1024 bytes, the last
    one could not have it.

    Am i in the right way to solve this problem?

  2. #2
    uint64_t...think positive xuftugulus's Avatar
    Join Date
    Feb 2008
    Location
    Pacem
    Posts
    355
    Quote Originally Posted by mariano_donati View Post
    2.
    To divide buffer into smaller parts, of 1024 bytes each.
    This, requires to copy 1024 bytes from original buffer to a temporal one, and then
    copy it to the image like:
    Code:
                 int i;
                 int copyStartPosition;           /* Position of original buffer where copy begins */
                 for ( i = 0 ; i < 1024 ; i++ )
                 {
                       tmpBuffer[i] = originalBuffer[i+copyStartPosition];
                 }
                 fwrite ( tmpBuffer , 1 , 1024 , filePointer );
    Here i'm not counting on not all packages will have a regular size of 1024 bytes, the last
    one could not have it.

    Am i in the right way to solve this problem?
    Well you probably need to move fwrite within the loop. Also you will need to remove the magic 1024, because at the very last frame of your buffer, if it is not 1024 then you will access not owned memory and write garbage to the file. You could do it as follows.
    Have a variable bytesWritten, initialized to 0. Then before the fwrite you would have a variable bytesToWrite. Check whether bufferSize-bytesWritten>=1024 and if not then bytesToWrite is bufferSize-bytesWritten, else bytesToWrite is 1024. Do the fwrite with the custom size and update your variables appropriately. You can use pointer arithmetic with the bytesWritten variable on tmpBuffer or index it with that variable to find the next frame start position, whichever is your preferable style.
    Last edited by xuftugulus; 02-24-2008 at 10:05 PM.
    Code:
    ...
        goto johny_walker_red_label;
    johny_walker_blue_label: exit(-149$);
    johny_walker_red_label : exit( -22$);
    A typical example of ...cheap programming practices.

  3. #3
    Registered User
    Join Date
    Nov 2007
    Posts
    48
    Thanks. This is what i've got so far. I haven't tested it yet, because i need to finish something else before i can do it properly.

    Code:
    void writeImageFromBuffer ( const char* filename , const char* buffer , unsigned long length )
    {
        const int REGULARPACKAGE_SIZE = 1024;
        FILE *image;                                  /* Pointer to file */
        unsigned long bytesWritten;                   /* Bytes written so far in the buffer */
        int bytesToWrite;                             /* Bytes to write to file */                            
        char *tmpBuffer;                              /* Temporary buffer */
        
        bytesWritten = 0;
    
        image = fopen ( filename , "wb" );    
    
        // allocate tmpBuffer
        tmpBuffer = (char *) malloc ( REGULARPACKAGE_SIZE );
        
        while ( bytesWritten < length )
        {
            if ( length - bytesWritten >= REGULARPACKAGE_SIZE )  /* not last frame */
            {
                bytesToWrite = REGULARPACKAGE_SIZE;
            }
            else                                                 /* last frame */
            {
                bytesToWrite = length - bytesWritten;            
                // reallocate tmpBuffer to its adecuate size
                tmpBuffer = (char *) realloc ( tmpBuffer , bytesToWrite );
            }
            // copy original buffer <bytesToWrite> elements to tmpBuffer
            tmpBuffer = &buffer[bytesWritten];
            // write tmpBuffer to file
            fwrite ( tmpBuffer , 1 , bytesToWrite , image );
            // just upgrade the var
            bytesWritten += bytesToWrite;
        } 
       
        fclose ( image );   
    }
    Last edited by mariano_donati; 02-24-2008 at 10:34 PM.

  4. #4
    Fountain of knowledge.
    Join Date
    May 2006
    Posts
    794
    Quote Originally Posted by mariano_donati View Post
    I want to know what would be an efficient way of do it. I haven't written any code yet, i just have a couple of ideas. Here they go both (just an idea about how code would look like):

    1.
    To write entire buffer in one step like:
    Code:
      
               fwrite ( buffer , 1 , totalBufferSize , filePointer );
    2.
    To divide buffer into smaller parts, of 1024 bytes each.
    This, requires to copy 1024 bytes from original buffer to a temporal one, and then
    copy it to the image like:
    Code:
                 int i;
                 int copyStartPosition;           /* Position of original buffer where copy begins */
                 for ( i = 0 ; i < 1024 ; i++ )
                 {
                       tmpBuffer[i] = originalBuffer[i+copyStartPosition];
                 }
                 fwrite ( tmpBuffer , 1 , 1024 , filePointer );
    Here i'm not counting on not all packages will have a regular size of 1024 bytes, the last
    one could not have it.

    Am i in the right way to solve this problem?
    I would just write the whole lot on in one go.
    I currently write about 10 megabyes in that way, the time to do it is negligible.
    1 kilobyte is absolutely nothing to a modern computer, if I did it your way it would
    take 10,000 iterations of the loop, you might not notice the difference but why make
    work for yourself? If you like work you can come around and wash my car .
    The processor will have significant block move operations which will shift 10 megabytes
    in a flash, putting it in a loop will slow it down, I will stake my reputaion on that!!
    Although I won't be losing much in that instance
    Anyhow the real actually delay will be the disk I/O and I doubt any disk can store data
    faster than your processor can throw it at it (correct me if I am wrong) and anyway
    it would be rather pointless if it could!!! (so don't bother correcting me anyway!!).

  5. #5
    CSharpener vart's Avatar
    Join Date
    Oct 2006
    Location
    Rishon LeZion, Israel
    Posts
    6,484
    tmpBuffer = &buffer[bytesWritten];
    you just leaking allocated memory - remove all malloc/realloc stuff you do not use anyway
    The first 90% of a project takes 90% of the time,
    the last 10% takes the other 90% of the time.

  6. #6
    Registered User
    Join Date
    Nov 2007
    Posts
    48
    Quote Originally Posted by vart View Post
    you just leaking allocated memory - remove all malloc/realloc stuff you do not use anyway
    What do you mean with leaking? my english becomes too poor sometimes.
    With malloc/realloc statements i thought that i was reserving the exact number of bytes i need. Is it unnecesary?.

  7. #7
    uint64_t...think positive xuftugulus's Avatar
    Join Date
    Feb 2008
    Location
    Pacem
    Posts
    355
    Quote Originally Posted by esbo View Post
    I would just write the whole lot on in one go.
    I currently write about 10 megabyes in that way, the time to do it is negligible.
    I was writting some bit read/write routines and needed comparison on my machine of the throughput so i did some timing on the stdio.h routines. It was true that on WinXP it did matter which size i picked for a buffer splitter and i would top writting a 8MB file at 256k buffer, using fwrite. On Linux, i had constant speed of 110MB/sec as long as the buffer was smaller than the files size.
    The fact is that it might be useful to read/write in chunks sometimes...

    Mario: You don't need to do so complex things when writing your buffer. Think for a second that the variable bytesWritten, can be used to index the tmpBuffer and you need no realloc, or copies or nothing. Just send tmpBuffer[bytesWritten] to fwrite! It will work fine, and elegantly.
    Code:
    ...
        goto johny_walker_red_label;
    johny_walker_blue_label: exit(-149$);
    johny_walker_red_label : exit( -22$);
    A typical example of ...cheap programming practices.

  8. #8
    CSharpener vart's Avatar
    Join Date
    Oct 2006
    Location
    Rishon LeZion, Israel
    Posts
    6,484
    Quote Originally Posted by mariano_donati View Post
    What do you mean with leaking? my english becomes too poor sometimes.
    With malloc/realloc statements i thought that i was reserving the exact number of bytes i need. Is it unnecesary?.

    leaking memory - you allocate some memory from OS, store the address of the beginning and then - loose this address, so you cannot access the memory and cannot free it, memory stays allocated, but you do not have access to it

    your

    tmpBuffer = &buffer[bytesWritten];

    assignment does not copies the original buffer contents anywhere, you just move the temppointer along the buffer to access different parts of the original buffer (as you should) so no additional storage is required.

    and then you write the part of the original buffer to disk once again - without unneded copy to the temp buffer
    The first 90% of a project takes 90% of the time,
    the last 10% takes the other 90% of the time.

  9. #9
    C++まいる!Cをこわせ! Elysia's Avatar
    Join Date
    Oct 2007
    Posts
    22,767
    IF you already have a buffer, writing it in one go is usually most efficient. But you should be careful when reading a big file, since reading a GB file into memory will probably hurt performance on many machines.
    Quote Originally Posted by Adak View Post
    io.h certainly IS included in some modern compilers. It is no longer part of the standard for C, but it is nevertheless, included in the very latest Pelles C versions.
    Quote Originally Posted by Salem View Post
    You mean it's included as a crutch to help ancient programmers limp along without them having to relearn too much.

    Outside of your DOS world, your header file is meaningless.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Trouble writing to file using fwrite()
    By yougene in forum C Programming
    Replies: 4
    Last Post: 12-30-2008, 04:13 PM
  2. writing a pack-style function, any advices?
    By isaac_s in forum C Programming
    Replies: 10
    Last Post: 07-08-2006, 08:09 PM
  3. buffer contents swapping
    By daluu in forum C++ Programming
    Replies: 7
    Last Post: 10-14-2004, 02:34 PM
  4. Simple File encryption
    By caroundw5h in forum C Programming
    Replies: 2
    Last Post: 10-13-2004, 10:51 PM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21