Read (BIG) File into char array

This is a discussion on Read (BIG) File into char array within the C Programming forums, part of the General Programming Boards category; Hi! I have to read a huge data file (~180MB) of unkown size into a char array. I am really ...

  1. #1
    carpe diem
    Join Date
    Jan 2010
    Posts
    46

    Read (BIG) File into char array

    Hi!
    I have to read a huge data file (~180MB) of unkown size into a char array. I am really not sure how to go about doing this... This is what I have so far:

    Code:
        
                   FILE *input; 
                    int count; 
                    char *cFile; 
                    long filelength;          
                    /* file input */ 
     
         
                    if ( (input = fopen("inputfile", "rb" )) == NULL) 
                    { 
                   printf("file not found");
                   getchar();
                    exit(1); 
                    } 
     
                  // @ checking length of file 
     
                    fseek(input, 0L, SEEK_END); 
                    filelength = ftell(input);      
                    rewind(input);                  
     
                    // read the file into memory 
     
                    cFile = calloc(filelength + 1, sizeof(char));  //allocate memory for file 
     
                    if(cFile == NULL) 
                    { 
                            printf("\nInsufficient memory to read file\n");
                            getchar(); 
                            exit(1); 
                    } 
     
               //read file into allocated memory 
                    if (fread(cFile, filelength, 1, input) != 1) 
                    { 
                            printf("error reading file"); 
                            exit(1); 
                    } 
     
                    fclose(input);
    Does anyone have any better suggestions of how to better approach this issue? Right now I don't have the exact file I will be using (180MB of binary data) so I can only test with smaller samples. I am worried once I get the BIG file I won't have sufficient memory=(
    All comments/suggestions appreciated.

  2. #2
    Registered User
    Join Date
    Oct 2008
    Location
    TX
    Posts
    2,058
    Look into the mmap() call as it allows mapping portions of the file into memory.
    This can be of use on systems that don't have enough memory installed in 'em.

  3. #3
    Registered User
    Join Date
    Sep 2006
    Posts
    8,868
    Test your system and see how much memory you can allocate dynamically. If you can get enough allocated for your big files, then you don't need to do anything else. Otherwise, design your program to work with only parts of the data, at any one time.

    A little bit of testing and thinking about how you can work with smaller pieces of the data at a time, go a long, long, way.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Newbie homework help
    By fossage in forum C Programming
    Replies: 3
    Last Post: 04-30-2009, 05:27 PM
  2. Code review
    By Elysia in forum C++ Programming
    Replies: 71
    Last Post: 05-13-2008, 10:42 PM
  3. Personal Program that is making me go wtf?
    By Submeg in forum C Programming
    Replies: 20
    Last Post: 06-27-2006, 01:13 AM
  4. Post...
    By maxorator in forum C++ Programming
    Replies: 12
    Last Post: 10-11-2005, 09:39 AM
  5. read from .txt file & put into array?
    By slow brain in forum C Programming
    Replies: 6
    Last Post: 02-25-2003, 05:16 AM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21