How to may a REALLY big array?

Hi,

I want to make a BIG array with about 10M elements, but it seems like I'm overflowing the stack, if I do an array of 500 000 elements its ok, but when I go to 5 000 000 I get segmentation fault.

basically this is all I do:

Code:

---

#define SAMPLES 5000000;

int
main(void)
{
struct block_data bd[SAMPLES];
}

---

the I save a lot of data in bd, but the seg fault comes before that. tried valgrind and it points to where I initialize variables the lines under I defined bd, so I think I'm overflowing the stack ... i've also try to allocate memory like this:

Code:

---

    struct block_data *bd[SAMPLES];
    uint64_t i = 0;

    while(i<SAMPLES) {
        bd[i] = (struct block_data *)malloc(sizeof(struct block_data));
        ++i;
    }

---

but still segementationfault.


So how can I do this?

I doubt your first example even compiles, but you are trying to create a local array that is quite large, so you probably run out of stack-space.

Likewise if you create a local pointer array.

If you need really large amounts of data, global variable, or allocating the memory through a pointer will avoid stack overflow.

Most systems have a limit of around 2MB of stack-space.

--
Mats

Or I suppose you could use C++ to get around the problem, too. But I don't think that's an acceptable alternative?

Quote:

---

Originally Posted by matsp

I doubt your first example even compiles, but you are trying to create a local array that is quite large, so you probably run out of stack-space.

Likewise if you create a local pointer array.

If you need really large amounts of data, global variable, or allocating the memory through a pointer will avoid stack overflow.

Most systems have a limit of around 2MB of stack-space.

--
Mats

---

so how do I "allocating the memory through a pointer", thats what I'm trying to do in my second code section, what is wrong?

You're creating too many pointers. The stack still overflows.

Quote:

---

Originally Posted by Elysia

You're creating too many pointers. The stack still overflows.

---

Ahh, ok, but how to allocate memory the correct way then?

You are doing it correct. You're just using too much of it.
You can increase the stack size, make it global or avoid it by using lesser pointers. You could also, for example, group several pointers into a struct and allocate it on the heap, thus reducing the amount of pointers you use.
Or, you could allocate your big pointer array on the heap and access it with another pointer, so it becomes pointer-to-pointer, see mats's example below.

Something like this, perhaps?

Code:

---

    struct block_data **bd;
    uint64_t i = 0;

    bd = malloc(sizeof(struct block_data *) * SAMPLES);

    while(i<SAMPLES) {
        bd[i] = (struct block_data *)malloc(sizeof(struct block_data));
        ++i;
    }

---

--
Mats

Quote:

---

Originally Posted by Elysia

You are doing it correct. You're just using too much of it.
You can increase the stack size, make it global or avoid it by using lesser pointers. You can, for example, group several pointers into a struct and allocate it on the heap, thus reducing the amount of pointers you use.

---

Ok, thanks you for your answers, but I'm kind of new on the memory management in C, so how do I increase the stack size? how do I allocate memory on the heap?

Don't malloc use the heap? Are the pointers at the heap but the array of pointers at the stack which I'm overflowing? But how do I put the array of pointers at the heap? and how do I increase the size of the heap if I need to?

Quote:

---

Originally Posted by micke_b

Ok, thanks you for your answers, but I'm kind of new on the memory management in C, so how do I increase the stack size?

---

This is compiler and platform specific, so it should be avoided if possible.

Quote:

---

Don't malloc use the heap?

---

Yes, it does.

Quote:

---

Are the pointers at the heap but the array of pointers at the stack which I'm overflowing?

---

The stack is overflowing with a lot of pointers on it.

Quote:

---

But how do I put the array of pointers at the heap?

---

See mats's example.

Quote:

---

and how do I increase the size of the heap if I need to?

---

You don't. The heap is typically limited only to the amount of virtual memory left (which could be around 1.8 GB or so-ish).
Let's not get too complex. It's not 100% how it works, but it's close enough. Don't gobble up unnecessary memory though.

The code I posted allocates the array on the heap. And yes, malloc() and it's close relatives give you space on the heap. Since the heap can easily be 100-1000 times larger than the stack, you should be OK.

You should, however check if bd == NULL or bd[i] == NULL after the respective calls to malloc, and if so print something and exit the program, as this indicates that there wasn't enough memory.

Increasing the stack is something you need to tell the compiler as part of teh building of the application.

--
Mats

Quote:

---

Originally Posted by matsp

The code I posted allocates the array on the heap. And yes, malloc() and it's close relatives give you space on the heap. Since the heap can easily be 100-1000 times larger than the stack, you should be OK.

You should, however check if bd == NULL or bd[i] == NULL after the respective calls to malloc, and if so print something and exit the program, as this indicates that there wasn't enough memory.

Increasing the stack is something you need to tell the compiler as part of teh building of the application.

--
Mats

---

Ahh, ok thanks will try your code, must have been writing my post same time as you did. just saw it.

thanks matsp, your code works perfect, thanks, problem solved. :)