Reading large files (20-110MB) into memory

[OOPboredom]

I was trying to think of efficient ways to read large files (20-110MB) into memory. I have a function where I must read an entire file, assign it to an unsigned char*, and then pass it to another function for evaluation. The char* must represent the contents of the file.

Is there a way to efficiently do this, or is the best method to simply read the entire file in one chunk? Obviously there will be a performance hit if many files have to be read. Consider doing this for data over the count of 10GB...

Any ideas are greatly appreciated.

[Prelude]

Well, if you must pass a pointer to char and that pointer to char must contain the entire contents of the file, there really isn't much you can do aside from rethink your design. Reading a 20-110MB file into memory should be done only as a last resort. If you can pass a FILE* to the function or make multiple calls with smaller blocks then you should do so.

Concerning performance, if you must have the file in memory, taking it in blocks as large as the system handles efficiently until you have everything is really the best you can do. At these sizes, and considering your restrictions, you're looking at a startup performance hit no matter what solution you choose.

[Sebastiani]

>> Consider doing this for data over the count of 10GB...

How many users do you think are going to have that much RAM to spare at any given moment? Besides that, most OS's place a limit on how much memory a single program can allocate. I agree with Prelude - just pass file handles around and process the data in smaller chunks. What sort of application is it, by the way?

[OOPboredom]

According to the design specification I do not have any flexibility on how I pass the data. The char* must represent the exact contents of the file.

The function that I pass the char* to only takes a char* and then computes a hash value from the value. I wish it were different or I would be able to cumulatively pass blocks of data, but I cannot. It simply looks like I'll have to read entire files in one chunk. This leads me to my next question.

If I allocate space for the unsigned char pointer, would it be best to get the size of the file and allocate accordingly or should I read 512byte blocks and keep adding what I read to the end of the previously read data. It might be easier to just allocate one large chunk of memory, but what happens when I have to read a 100MB file and represent that file by a char*?

[Hammer]

>>The function that I pass the char* to only takes a char*
Then how does it know how long the data is? You cannot assume a \0 terminated array in this scenario.

>> I do not have any flexibility on how I pass the data. The char* must represent the exact contents of the file.
Has anyone said you can't call this function more than once?

>>what happens when I have to read a 100MB file and represent that file by a char*
You're RAM starts to fill up, and the OS swaps out memory to disk, making the whole thing inefficient.

[OOPboredom]

Originally posted by Hammer
>>Then how does it know how long the data is? You cannot assume a \0 terminated array in this scenario.
I forgot to mention that I also pass in the size of the char* - in bytesand the struct to place the result in. The algorithm being used is the SHA-1 algorithm - standard hash algoritm.

>> Has anyone said you can't call this function more than once?
Under the guidelines of the SHA-1 algorithm I can only call it once.
Calling it multiple times will result in a different result (the result being a 160bit (40byte) string.

>>what happens when I have to read a 100MB file and represent that file by a char*
"You're RAM starts to fill up, and the OS swaps out memory to disk, making the whole thing inefficient."
Inefficient but still plausible - which is what I really need.

Thanks Hammer.

[OcTO_VIII]

OOPboredom: to read a large file, use memory mapping. If Linux is the case, use mmap(). I don't know the memory mapping function for Windows.