Thread: Copy any type of file to a memory buffer?

  1. #1
    Registered User
    Join Date
    Sep 2014
    Posts
    121

    Copy any type of file to a memory buffer?

    Hello, is there any way to copy any type of binary/text file to a memory buffer, and then use the buffer to make a copy of that file?
    Code:
    char* readFileToBuffer(const char* fname, const char* opts) {
        FILE* fp = fopen(fname, opts);
        if ( !fp ) return 0x00;
        fseek(fp, 0, SEEK_END);
        long fsize = ftell(fp);
        printf("readFileToBuffer(): %d\n", fsize);
        char *buff = (char*)malloc(sizeof(char)* fsize+1);
        fseek(fp, 0, SEEK_SET);
        char* begin = buff;
        fread((char*)buff,  fsize, 1,fp);
        fclose(fp);
        return buff;
    }
    This, actually can`t be used to copy to a new file from the return buffer.
    Last edited by heatblazer; 01-13-2015 at 01:43 PM.

  2. #2
    Registered User
    Join Date
    Oct 2008
    Location
    TX
    Posts
    2,059
    Not sure where your are going with this, since the *nix copy (cp) command does exactly that.
    It reads data from a source file into a buffer, followed by writing that buffer to a destination file.

  3. #3
    Registered User
    Join Date
    Sep 2014
    Posts
    121
    Experiment with manipulating the data.

  4. #4
    Lurker
    Join Date
    Dec 2004
    Posts
    296
    Quote Originally Posted by heatblazer View Post
    Experiment with manipulating the data.
    You have yet to ask a single question.

    So the answer is still "Yes it is possible to read in a file and then write it out again".

  5. #5
    Registered User rstanley's Avatar
    Join Date
    Jun 2014
    Location
    New York, NY
    Posts
    733
    Quote Originally Posted by heatblazer View Post
    Hello, is there any way to copy any type of binary/text file to a memory buffer, and then use the buffer to make a copy of that file?
    Code:
    char* readFileToBuffer(const char* fname, const char* opts) {
        FILE* fp = fopen(fname, opts);
        if ( !fp ) return 0x00;
        fseek(fp, 0, SEEK_END);
        long fsize = ftell(fp);
        printf("readFileToBuffer(): %d\n", fsize);
        char *buff = (char*)malloc(sizeof(char)* fsize+1);
        fseek(fp, 0, SEEK_SET);
        char* begin = buff;
        fread((char*)buff,  fsize, 1,fp);
        fclose(fp);
        return buff;
    }
    This, actually can`t be used to copy to a new file from the return buffer.
    I would have written this code differently, but yes, it is possible to read in the contents of any file to a buffer, and write it out to another file, with or without manipulating the data in between. Any file is just a collection of individual bytes. The file extension, and in many cases, the file's, "Magic Number" determine how the file should be used, or what application should handle the file.

  6. #6
    Registered User
    Join Date
    Sep 2014
    Posts
    121
    I`ve heard about that magic number you say, some refer to header, first 56 bytes? I know the first 3 are file header description like BMP, ELF, MPW etc. Can you be more specific how to approach it? THe code I`ve posted is from the net, it does not copy a file tho.

  7. #7
    C++ Witch laserlight's Avatar
    Join Date
    Oct 2003
    Location
    Singapore
    Posts
    27,854
    I suggest that you use a loop along with an array of say, BUFSIZ number of char. Loop until the end of the input file has been reached, reading into the array and then writing the contents of the array to the output file. The array would then be your buffer.

    Quote Originally Posted by heatblazer
    I`ve heard about that magic number you say, some refer to header, first 56 bytes? I know the first 3 are file header description like BMP, ELF, MPW etc. Can you be more specific how to approach it? THe code I`ve posted is from the net, it does not copy a file tho.
    This should be of no concern to you since you want to handle files without concern for their file format. If you are concerned with file format, then you need to be specific as to which file formats you have in mind in order to receive specific help concerning them.

    If you want to cater to all file formats specifically (sounds like an oxymoron, heh), then sorry, but that is impossible. You can only cater to those that you are aware of, and then handle more generically those that you are not aware of.
    Quote Originally Posted by Bjarne Stroustrup (2000-10-14)
    I get maybe two dozen requests for help with some sort of programming or design problem every day. Most have more sense than to send me hundreds of lines of code. If they do, I ask them to find the smallest example that exhibits the problem and send me that. Mostly, they then find the error themselves. "Finding the smallest program that demonstrates the error" is a powerful debugging tool.
    Look up a C++ Reference and learn How To Ask Questions The Smart Way

  8. #8
    Registered User
    Join Date
    Sep 2014
    Posts
    121
    Quote Originally Posted by laserlight View Post
    I suggest that you use a loop along with an array of say, BUFSIZ number of char. Loop until the end of the input file has been reached, reading into the array and then writing the contents of the array to the output file. The array would then be your buffer.


    This should be of no concern to you since you want to handle files without concern for their file format. If you are concerned with file format, then you need to be specific as to which file formats you have in mind in order to receive specific help concerning them.

    If you want to cater to all file formats specifically (sounds like an oxymoron, heh), then sorry, but that is impossible. You can only cater to those that you are aware of, and then handle more generically those that you are not aware of.
    I`ve been expecting your solution, I was thinking about that too. Just wanted to hear more options.

  9. #9
    Registered User
    Join Date
    Sep 2014
    Posts
    121
    Any good example? Can I put with
    Code:
    fprintf(myFhandle, "%s", myWholeBuffer);
    or to exchange the filesize reported by fseek as a global and pass it to a loop to both copy/paste file functions?

  10. #10
    Registered User
    Join Date
    Sep 2014
    Posts
    121
    Here is example:
    Code:
    static inline char* readFileToBuffer(const char* fname, const char* opts) {
        FILE* fp = fopen(fname, opts);
        if ( !fp ) return 0x00;
        fseek(fp, 0, SEEK_END);
        fsize = ftell(fp);
        char *buff = (char*) malloc(sizeof(char)*fsize);
        myHeap[hIterator++] =  buff;
        fseek(fp, 0, SEEK_SET);
    #ifdef READFILETOBUFF_VER1
        int i=0;
        for (; i < fsize; i++) buff[i] = fgetc(fp);
    #else
        fread(buff, fsize, 1, fp);
    #endif
        fclose(fp);
        return buff;
    }
    and the write to
    Code:
    static inline void* writeToFile(const char* string, const char* fname, const char* opts) {
        FILE* fp = fopen(fname, opts);
        if ( !fp ) return NULL;
        int i=0;
        while ( i < fsize ) {
            fputc(string[i++], fp);
        }
        fclose(fp);
    }
    WHere fsize is static int variable to exhcange between the 2 functions. Also I`ve add more checks if it`s 0 and so... For now it copies the file 1:1

  11. #11
    C++ Witch laserlight's Avatar
    Join Date
    Oct 2003
    Location
    Singapore
    Posts
    27,854
    Do you actually need the entire contents of the file to be in memory? My suggestion involves an array of fixed size that is reused on each iteration.

    As for your static int variable: what happens if I call readFileToBuffer twice in a row, and then call writeToFile twice in a row? Why not have the caller pass a pointer to a variable that readFileToBuffer can update, which is then passed to writeToFile?

  12. #12
    Registered User
    Join Date
    Sep 2014
    Posts
    121
    Yes, I `ve tought about that. I was thinkig of a call stack for the functions. Or some kind of priority. I am working it in my mind. So about your suggestion, can you scratch some idea in code? Array of fixed size? How?
    [EDIT] If you call it twice... you`ll overwrite the buffer I guess.... But isn`t that logical? This will fill my private heap I am using for the char pointers, but I havent worked any ideas so far.
    Last edited by heatblazer; 01-14-2015 at 07:56 AM.

  13. #13
    C++ Witch laserlight's Avatar
    Join Date
    Oct 2003
    Location
    Singapore
    Posts
    27,854
    Quote Originally Posted by heatblazer
    Yes, I `ve tought about that. I was thinkig of a call stack for the functions. Or some kind of priority. I am working it in my mind.
    A call stack for the functions?! I gave you a rather simple suggestion in post #11: "Why not have the caller pass a pointer to a variable that readFileToBuffer can update, which is then passed to writeToFile?" This is like telling you to use a screw for your screwdriver. Your idea is like building new workshop just because your screwdriver does not work with a nail.

    Quote Originally Posted by heatblazer
    [EDIT] If you call it twice... you`ll overwrite the buffer I guess.... But isn`t that logical?
    No, you will not overwrite the buffer: your code has a separate buffer for each call of readFileToBuffer since you call malloc. What you will overwrite is fsize since it is a "static int variable to exhcange between the 2 functions".

    Quote Originally Posted by heatblazer
    This will fill my private heap I am using for the char pointers, but I havent worked any ideas so far.
    When exactly do you free your private heap? If you only free it at the end of the program, then when running your program under a modern operating system, you're doing no better than having no private heap at all since the OS will most likely release the memory for you. Yes, I often gloss over this when insisting that help seekers here free what they malloc even when the free comes right at the end of the main function, but the idea is to create a good habit so that when the person writes a non-trivial program, he/she will remember to have a free to match the malloc.

    Quote Originally Posted by heatblazer
    So about your suggestion, can you scratch some idea in code? Array of fixed size? How?
    My idea runs along these lines:
    Code:
    #include <stdio.h>
    
    enum copy_result
    {
        COPY_SUCCESS,
        COPY_FAILURE,
        OPEN_FAILURE_SOURCE,
        OPEN_FAILURE_DESTINATION
    };
    
    enum copy_result copyFileContent(FILE *source, FILE *destination)
    {
        unsigned char buffer[BUFSIZ];
        size_t num_read;
        while ((num_read = fread(buffer, sizeof(buffer[0]), BUFSIZ, source)) > 0)
        {
            if (fwrite(buffer, sizeof(buffer[0]), num_read, destination) < num_read)
            {
                return COPY_FAILURE;
            }
        }
        return (feof(source) && !ferror(source)) ? COPY_SUCCESS : COPY_FAILURE;
    }
    
    enum copy_result copyFile(const char *source_name, const char *destination_name)
    {
        enum copy_result result;
        FILE *source;
        FILE *destination;
    
        source = fopen(source_name, "rb");
        if (!source)
        {
            return OPEN_FAILURE_SOURCE;
        }
    
        destination = fopen(destination_name, "wb");
        if (!destination)
        {
            fclose(source);
            return OPEN_FAILURE_DESTINATION;
        }
    
        result = copyFileContent(source, destination);
        fclose(destination);
        fclose(source);
        return result;
    }
    
    int main(int argc, char *argv[])
    {
        if (argc > 2)
        {
            const char * const source_name = argv[1];
            const char * const destination_name = argv[2];
            switch (copyFile(source_name, destination_name))
            {
            case COPY_SUCCESS:
                printf("Copied from '%s' to '%s'.\n", source_name, destination_name);
                break;
            case COPY_FAILURE:
                fprintf(stderr, "Could not copy from '%s' to '%s'.\n", source_name, destination_name);
                break;
            case OPEN_FAILURE_SOURCE:
                fprintf(stderr, "Could not open '%s' for reading.\n", source_name);
                break;
            case OPEN_FAILURE_DESTINATION:
                fprintf(stderr, "Could not open '%s' for writing.\n", destination_name);
                break;
            }
        }
        else
        {
            printf("Usage: %s source destination\n", argv[0]);
        }
        return 0;
    }
    Last edited by laserlight; 01-14-2015 at 10:32 AM.
    Quote Originally Posted by Bjarne Stroustrup (2000-10-14)
    I get maybe two dozen requests for help with some sort of programming or design problem every day. Most have more sense than to send me hundreds of lines of code. If they do, I ask them to find the smallest example that exhibits the problem and send me that. Mostly, they then find the error themselves. "Finding the smallest program that demonstrates the error" is a powerful debugging tool.
    Look up a C++ Reference and learn How To Ask Questions The Smart Way

  14. #14
    Registered User
    Join Date
    Sep 2014
    Posts
    121
    This is pretty code you wrote. Just wondering about the BUFSIZE. Yes, yes, I took the note, the stack idea was for something else, forgive me, I am working few things right now and I am in complete mess. So, about the BUFSIZE where to define it? Actually I was thinking for private static char pointer for the malloc of the copy, so I`ll overwrite it always.

    Code:
    if ( staticBuff != 0 ) free(staticBuff);
    else staticBuff = malloc(fsize);
    ....
    Just let me workit out a bit.... Actually I don`t want to copy and write to FILE, I really need to be explicitly in the buffer. Let`s say I want to copy 20 G file. I want to open the file, malloc some buffer, maybe fixed sized array, then write a block to the new file. Close both files, open to where SEEK pointer is, continue, repeat till copied. Let`s assume, this is copy per block. Actually such function could be recursive.

  15. #15
    C++ Witch laserlight's Avatar
    Join Date
    Oct 2003
    Location
    Singapore
    Posts
    27,854
    Quote Originally Posted by heatblazer
    So, about the BUFSIZE where to define it?
    BUFSIZ is already defined in <stdio.h>

    Quote Originally Posted by heatblazer
    Actually I was thinking for private static char pointer for the malloc of the copy, so I`ll overwrite it always.
    I see. I suggest that you think twice, very carefully, before reaching for a static local variable or a file scope variable that is not declared const.

    You're also reaching for malloc with such wild abandon. Do not use malloc unless you really need dynamic memory allocation, e.g., because the size cannot be set at compile time, or because you need to allocate something so large that you'll have problems placing it on the stack.

    Quote Originally Posted by heatblazer
    I am working few things right now and I am in complete mess.
    In a way, that's symptomatic of your code

    You have so much non-local state due to static variables here and there, file scope variables, pointers to dynamic memory stored in a "private heap", etc. Naturally, all this non-local state makes it more difficult to reason about your program, so it is more likely to end up as a "complete mess".

    Quote Originally Posted by heatblazer
    Actually I don`t want to copy and write to FILE, I really need to be explicitly in the buffer.
    Why?

    Quote Originally Posted by heatblazer
    Let`s say I want to copy 20 G file. I want to open the file, malloc some buffer, maybe fixed sized array, then write a block to the new file. Close both files, open to where SEEK pointer is, continue, repeat till copied. Let`s assume, this is copy per block. Actually such function could be recursive.
    Why would you do this when you can keep the files open and copy in a loop?

    From what I see, the main advantage that copying the entire file content to memory, closing the file, then opening the destination file and writing from memory to it has is that it will work correctly even if source and destination files are the same. Unfortunately, I do not think that one can avoid this for my example copyFile function: you can check the file names, but they might be aliased through say, symlinks, so you need lower level stuff to be sure that they are different.
    Last edited by laserlight; 01-14-2015 at 11:07 AM.
    Quote Originally Posted by Bjarne Stroustrup (2000-10-14)
    I get maybe two dozen requests for help with some sort of programming or design problem every day. Most have more sense than to send me hundreds of lines of code. If they do, I ask them to find the smallest example that exhibits the problem and send me that. Mostly, they then find the error themselves. "Finding the smallest program that demonstrates the error" is a powerful debugging tool.
    Look up a C++ Reference and learn How To Ask Questions The Smart Way

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Mixed-type contiguous memory buffer
    By Prandtl in forum C Programming
    Replies: 2
    Last Post: 03-02-2014, 01:10 AM
  2. Replies: 5
    Last Post: 12-20-2012, 07:10 PM
  3. Copy HWND into shared memory buffer
    By Opariti in forum Windows Programming
    Replies: 2
    Last Post: 12-26-2009, 01:08 PM
  4. Copy .dat file to dynamic memory...
    By IndioDoido in forum C Programming
    Replies: 5
    Last Post: 05-28-2007, 04:36 PM
  5. copy int to buffer??
    By Gugge in forum C Programming
    Replies: 4
    Last Post: 03-19-2002, 12:17 PM