Thread: question about malloc

i notice that when my program outputs files that are >1GB in size, it always crashes with a 'segmentation fault'. i was told that this is because my computer has a RAM of 1GB, so if my program uses malloc to allocate memory of >1GB, then this exceeds the RAM and is the cause of the crash.

is that right? does that mean that the RAM actually limits the size of files that a c program can work with? if so, is there a way to overcome this RAM-imposed limit?

Originally Posted by bored_guy

i notice that when my program outputs files that are >1GB in size, it always crashes with a 'segmentation fault'. i was told that this is because my computer has a RAM of 1GB, so if my program uses malloc to allocate memory of >1GB, then this exceeds the RAM and is the cause of the crash.

is that right? does that mean that the RAM actually limits the size of files that a c program can work with? if so, is there a way to overcome this RAM-imposed limit?

Depends on the storage allocated inside your program to store the file contents. Use mmap() to load the file piece-meal; and post the code snippet.

the code looks something like this:

Code:

float *in1 = malloc(size * sizeof(float));
float *in2 = malloc(size * sizeof(float));
float *in3 = malloc(size * sizeof(float));
...

where size can be as large as 100million. i have to declare quite a number of such mallocs in my program.

i think doing it piece-meal for me might not work, because of the BSQ/BIL arrangement of my multi-band images.

So each malloc needs around 382 MiB of RAM...(32 bit float = 4 bytes, 400 million bytes). If you're doing all those mallocs at the same time, yeah you'll run out of memory.

As itCbitC said, RAM doesn't have to limit the size of the output file though, just depends on the program. If you're going to keep the entire contents of the file in memory, then yeah, RAM is going to limit how big of a file you can make.