Large Array Size

[smudge256]

Hi,

I need 2 dynamically create an array. occasionally this can mean my program is trying 2 create char arrays of 100+ million!!!

This causes segmentation faults. Is it possible to either prevent the segmentation faults by freeing the memory befor hand? or to capture and recover from the error?

Im pretty new to C so i appologise in advance if this a stupid question.

[Adak]

Welcome to the forum, Smudge.

What is your RAM (in bytes), what is your operating system, and what size of array are you able to dynamically allocate, right now?

Why do you need such huge arrays, may I ask? No way to work on some of the data, write it out to a file, and then work on another chunk of data?

[Salem]

It seems to me that there is something else going on if you're getting segmentation faults.

A segfault is a clear indication of a bug in the program. Typically, one of
- not allocating enough memory, then stepping off the end
- using it after it has been freed.
- freeing it more than once
- freeing something that wasn't allocated.
- not checking the allocation was successful
- for realloc, not taking into account that the block could have moved.

Post some code showing how you allocate and use some of these allocations.


A side note is you can run out of memory (a memory leak) by not freeing memory when you've finished with it.

Finally, this
Out of memory - Wikipedia, the free encyclopedia

[MK27]

- not checking the allocation was successful

Do that first, eg:

void *check_malloc(int size) {
       void *r = malloc(size);
       if (!r) puts("OUT OF MEMORY!!!");
       return r;
}

unless you are on linux, in which case it is meaningless (malloc never returns NULL there).

[smudge256]

Im creating a program that reads 2 files and concatenates them sideways. Im using fgets to read each line, one by one. some of the files im testing on have huge lines 100 million chars per line. I could use fgetc but i can imagine tht would be pretty slow and messy to write.

I would guess its saleems first point: not allocating enough memory, then stepping off the end.

All i am doing is.

char s1 [sizeOfLongestLine(argv[2]) + 2]; // seg fault here
fgets ( s1, sizeof s1, file1 )

The sizeOfLongestLine works fine and returns a long. when i hardedcoded array size to be 100 mill i also get the same error. Should i be doing some malloc prior to this?

Sorry i don't know the specs of the machine this will be running on. most likely 32 bit linux with 2Gb memory

[MK27]

You for sure cannot declare a stack variable that big. You will have to allocate heap space for it.

char *s1 = malloc(sizeOfLongestLine(argv[2]) + 2);

Remember to free() this appropriately.

[cpjust]

unless you are on linux, in which case it is meaningless (malloc never returns NULL there).

I find that highly unlikely. How can malloc() never fail on Linux?
Here it says it returns NULL: MALLOC

[laserlight]

I find that highly unlikely. How can malloc() never fail on Linux?
Here it says it returns NULL: MALLOC

"Never" is too strong, but if you read carefully you will see that by default, it may be the case that malloc() does not return a null pointer on Linux, and yet memory is not available.

[MK27]

The man page reflects the fact that it could return NULL, and if it does, it's because "the request failed", but unless you explicitly set your kernel in a non-standard way, the request will never fail. There's one explanation here:

OOM::Kill Me Not « Thermal Noise

Some people see it as a feature, others find this terrifying.

I believe the boot time switch is "overcommit" (on by default*), but I've never turned it off because I am too smart to run out of mem anyway

* maybe that started with 2.4

[cpjust]

The man page reflects the fact that it could return NULL, and if it does, it's because "the request failed", but unless you explicitly set your kernel in a non-standard way, the request will never fail. There's one explanation here:

OOM::Kill Me Not « Thermal Noise

Some people see it as a feature, others find this terrifying.

I believe the boot time switch is "overcommit" (on by default*), but I've never turned it off because I am too smart to run out of mem anyway

* maybe that started with 2.4

Wow! Now that is seriously F#'ed up! It makes sense for the 1-2GB initial process space, but that shouldn't be malloced the same way that a program malloc's memory; and a program shouldn't be mallocing more memory than it needs*.

* well maybe a bit more, but not ridiculously huge amounts.