Doing a malloc from an ftell

I've reached this situation several times and just want to get some general advice.

I want to perform a call to malloc which allocates the exact number of bytes of a file on disk. One issue is that ftell() returns a long but malloc takes a size_t. I feel like the naive but obvious approach is to just do something like this:

Code:

---

long filesize = ftell(fp);

char *database = malloc((size_t)filesize);

---

The issue here is if ftell returns -1, this is not a good situation for malloc. Another issue is that the user can control directly how much is malloc'd by feeding various size files into the parser, potentially mallocing a HUGE amount of memory. Of course for this second problem, I could decide on a max # of bytes and create some logic to prevent a huge allocation. But I'm not sure on how to gather info to make this decision.

What are some better methods of accomplishing memory allocation based off of disk file size? I just need to map a database from disk to memory. I work as a security engineer professionally and I've seen some of the other engineers do a ftell output directly to a malloc but it created some security issues a few times, however, I've not yet discovered a better way of doing this aside from the aforementioned.

Last but not least, can someone explain some background on some of these C functions with different return types? For example, I would expect ftell to return a size_t but it returns a long. I've noticed this is the case with several C library functions and it makes programming in C tricky at times until of course one has learned how to deal with each of these specific conversion situations. I assume that the long may be due to the large number of bytes that could be output from ftell, but then again, why not unsigned long or long long?

Thank you.

First how are you opening the file binary or text?

Second what operating system are you using?

Third how are you calling fseek()?

Fourth do you realize that a file may be much larger than all of your available memory?

Quote:

---

The issue here is if ftell returns -1, this is not a good situation for malloc.

---

That's why you should always check the return value from this function. If it indicates an error, EOF, in this case you probably shouldn't use the value. And be careful, EOF can be any negative value.

Quote:

---

Another issue is that the user can control directly how much is malloc'd by feeding various size files into the parser, potentially mallocing a HUGE amount of memory.

---

Yes, see above. But realize that since ftell() returns an long, the largest amount of memory you will be able to allocate with this method is the largest value a long can hold. But beware of any negative values.

Quote:

---

Of course for this second problem, I could decide on a max # of bytes and create some logic to prevent a huge allocation. But I'm not sure on how to gather info to make this decision.

---

This is probably the best course, just insure your "max" is sized properly so as not to exceed the maximums of size_t and long.

Quote:

---

What are some better methods of accomplishing memory allocation based off of disk file size?

---

Normally trying to read a complete file into memory is a big mistake when dealing with large file sizes, stick with a fixed "buffer" and when you complete the processing of the buffer read in the next chunk.

Quote:

---

I just need to map a database from disk to memory.

---

This is normally accomplished by "indexes" into the file.

Quote:

---

Last but not least, can someone explain some background on some of these C functions with different return types?

---

Possibly, let's start by addressing some of your other issues in that paragraph?

Quote:

---

For example, I would expect ftell to return a size_t but it returns a long.

---

Why would you expect a size_t? Do you realize that a size_t is an implementation defined unsigned type? Do you realize that it is possible to seek backwards (negative)?

Quote:

---

I've noticed this is the case with several C library functions and it makes programming in C tricky at times until of course one has learned how to deal with each of these specific conversion situations.

---

Normally if you need to do "conversions" you're doing something wrong.

Quote:

---

I assume that the long may be due to the large number of bytes that could be output from ftell, but then again, why not unsigned long or long long?

---

You're assuming something that is probably wrong. First since you can access the file randomly you need negative values which are not supported by any unsigned type. Second it is possible that a long not be able to hold all positive values returned in a size_t type. For example on my system a long can hold positive values in the range of 0 to 9223372036854775807 while a size_t can hold values in the range of 0 to 18446744073709551615 so if you try to "convert" a size_t to a long you could invoke UB.