How to may a REALLY big array?

This is a discussion on How to may a REALLY big array? within the C Programming forums, part of the General Programming Boards category; Hi, I want to make a BIG array with about 10M elements, but it seems like I'm overflowing the stack, ...

  1. #1
    Registered User
    Join Date
    Oct 2007
    Posts
    47

    Unhappy How to may a REALLY big array?

    Hi,

    I want to make a BIG array with about 10M elements, but it seems like I'm overflowing the stack, if I do an array of 500 000 elements its ok, but when I go to 5 000 000 I get segmentation fault.

    basically this is all I do:

    Code:
    #define SAMPLES 5000000;
    
    int
    main(void)
    {
    struct block_data bd[SAMPLES];
    }


    the I save a lot of data in bd, but the seg fault comes before that. tried valgrind and it points to where I initialize variables the lines under I defined bd, so I think I'm overflowing the stack ... i've also try to allocate memory like this:


    Code:
        struct block_data *bd[SAMPLES];
        uint64_t i = 0;
    
        while(i<SAMPLES) {
            bd[i] = (struct block_data *)malloc(sizeof(struct block_data));
            ++i;
        }
    but still segementationfault.


    So how can I do this?

  2. #2
    Kernel hacker
    Join Date
    Jul 2007
    Location
    Farncombe, Surrey, England
    Posts
    15,677
    I doubt your first example even compiles, but you are trying to create a local array that is quite large, so you probably run out of stack-space.

    Likewise if you create a local pointer array.

    If you need really large amounts of data, global variable, or allocating the memory through a pointer will avoid stack overflow.

    Most systems have a limit of around 2MB of stack-space.

    --
    Mats
    Compilers can produce warnings - make the compiler programmers happy: Use them!
    Please don't PM me for help - and no, I don't do help over instant messengers.

  3. #3
    C++まいる!Cをこわせ! Elysia's Avatar
    Join Date
    Oct 2007
    Posts
    22,785
    Or I suppose you could use C++ to get around the problem, too. But I don't think that's an acceptable alternative?
    Quote Originally Posted by Adak View Post
    io.h certainly IS included in some modern compilers. It is no longer part of the standard for C, but it is nevertheless, included in the very latest Pelles C versions.
    Quote Originally Posted by Salem View Post
    You mean it's included as a crutch to help ancient programmers limp along without them having to relearn too much.

    Outside of your DOS world, your header file is meaningless.

  4. #4
    Registered User
    Join Date
    Oct 2007
    Posts
    47
    Quote Originally Posted by matsp View Post
    I doubt your first example even compiles, but you are trying to create a local array that is quite large, so you probably run out of stack-space.

    Likewise if you create a local pointer array.

    If you need really large amounts of data, global variable, or allocating the memory through a pointer will avoid stack overflow.

    Most systems have a limit of around 2MB of stack-space.

    --
    Mats
    so how do I "allocating the memory through a pointer", thats what I'm trying to do in my second code section, what is wrong?

  5. #5
    C++まいる!Cをこわせ! Elysia's Avatar
    Join Date
    Oct 2007
    Posts
    22,785
    You're creating too many pointers. The stack still overflows.
    Quote Originally Posted by Adak View Post
    io.h certainly IS included in some modern compilers. It is no longer part of the standard for C, but it is nevertheless, included in the very latest Pelles C versions.
    Quote Originally Posted by Salem View Post
    You mean it's included as a crutch to help ancient programmers limp along without them having to relearn too much.

    Outside of your DOS world, your header file is meaningless.

  6. #6
    Registered User
    Join Date
    Oct 2007
    Posts
    47
    Quote Originally Posted by Elysia View Post
    You're creating too many pointers. The stack still overflows.
    Ahh, ok, but how to allocate memory the correct way then?

  7. #7
    C++まいる!Cをこわせ! Elysia's Avatar
    Join Date
    Oct 2007
    Posts
    22,785
    You are doing it correct. You're just using too much of it.
    You can increase the stack size, make it global or avoid it by using lesser pointers. You could also, for example, group several pointers into a struct and allocate it on the heap, thus reducing the amount of pointers you use.
    Or, you could allocate your big pointer array on the heap and access it with another pointer, so it becomes pointer-to-pointer, see mats's example below.
    Last edited by Elysia; 01-31-2008 at 07:24 AM.
    Quote Originally Posted by Adak View Post
    io.h certainly IS included in some modern compilers. It is no longer part of the standard for C, but it is nevertheless, included in the very latest Pelles C versions.
    Quote Originally Posted by Salem View Post
    You mean it's included as a crutch to help ancient programmers limp along without them having to relearn too much.

    Outside of your DOS world, your header file is meaningless.

  8. #8
    Kernel hacker
    Join Date
    Jul 2007
    Location
    Farncombe, Surrey, England
    Posts
    15,677
    Something like this, perhaps?

    Code:
        struct block_data **bd;
        uint64_t i = 0;
    
        bd = malloc(sizeof(struct block_data *) * SAMPLES);
    
        while(i<SAMPLES) {
            bd[i] = (struct block_data *)malloc(sizeof(struct block_data));
            ++i;
        }
    --
    Mats
    Compilers can produce warnings - make the compiler programmers happy: Use them!
    Please don't PM me for help - and no, I don't do help over instant messengers.

  9. #9
    Registered User
    Join Date
    Oct 2007
    Posts
    47
    Quote Originally Posted by Elysia View Post
    You are doing it correct. You're just using too much of it.
    You can increase the stack size, make it global or avoid it by using lesser pointers. You can, for example, group several pointers into a struct and allocate it on the heap, thus reducing the amount of pointers you use.
    Ok, thanks you for your answers, but I'm kind of new on the memory management in C, so how do I increase the stack size? how do I allocate memory on the heap?

    Don't malloc use the heap? Are the pointers at the heap but the array of pointers at the stack which I'm overflowing? But how do I put the array of pointers at the heap? and how do I increase the size of the heap if I need to?

  10. #10
    C++まいる!Cをこわせ! Elysia's Avatar
    Join Date
    Oct 2007
    Posts
    22,785
    Quote Originally Posted by micke_b View Post
    Ok, thanks you for your answers, but I'm kind of new on the memory management in C, so how do I increase the stack size?
    This is compiler and platform specific, so it should be avoided if possible.

    Don't malloc use the heap?
    Yes, it does.

    Are the pointers at the heap but the array of pointers at the stack which I'm overflowing?
    The stack is overflowing with a lot of pointers on it.

    But how do I put the array of pointers at the heap?
    See mats's example.

    and how do I increase the size of the heap if I need to?
    You don't. The heap is typically limited only to the amount of virtual memory left (which could be around 1.8 GB or so-ish).
    Let's not get too complex. It's not 100% how it works, but it's close enough. Don't gobble up unnecessary memory though.
    Quote Originally Posted by Adak View Post
    io.h certainly IS included in some modern compilers. It is no longer part of the standard for C, but it is nevertheless, included in the very latest Pelles C versions.
    Quote Originally Posted by Salem View Post
    You mean it's included as a crutch to help ancient programmers limp along without them having to relearn too much.

    Outside of your DOS world, your header file is meaningless.

  11. #11
    Kernel hacker
    Join Date
    Jul 2007
    Location
    Farncombe, Surrey, England
    Posts
    15,677
    The code I posted allocates the array on the heap. And yes, malloc() and it's close relatives give you space on the heap. Since the heap can easily be 100-1000 times larger than the stack, you should be OK.

    You should, however check if bd == NULL or bd[i] == NULL after the respective calls to malloc, and if so print something and exit the program, as this indicates that there wasn't enough memory.

    Increasing the stack is something you need to tell the compiler as part of teh building of the application.

    --
    Mats
    Compilers can produce warnings - make the compiler programmers happy: Use them!
    Please don't PM me for help - and no, I don't do help over instant messengers.

  12. #12
    Registered User
    Join Date
    Oct 2007
    Posts
    47
    Quote Originally Posted by matsp View Post
    The code I posted allocates the array on the heap. And yes, malloc() and it's close relatives give you space on the heap. Since the heap can easily be 100-1000 times larger than the stack, you should be OK.

    You should, however check if bd == NULL or bd[i] == NULL after the respective calls to malloc, and if so print something and exit the program, as this indicates that there wasn't enough memory.

    Increasing the stack is something you need to tell the compiler as part of teh building of the application.

    --
    Mats
    Ahh, ok thanks will try your code, must have been writing my post same time as you did. just saw it.

  13. #13
    Registered User
    Join Date
    Oct 2007
    Posts
    47
    thanks matsp, your code works perfect, thanks, problem solved.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Replies: 8
    Last Post: 11-12-2008, 10:25 AM
  2. 2d array question
    By gmanUK in forum C Programming
    Replies: 2
    Last Post: 04-21-2006, 12:20 PM
  3. [question]Analyzing data in a two-dimensional array
    By burbose in forum C Programming
    Replies: 2
    Last Post: 06-13-2005, 07:31 AM
  4. Unknown Memory Leak in Init() Function
    By CodeHacker in forum Windows Programming
    Replies: 3
    Last Post: 07-09-2004, 09:54 AM
  5. Struct *** initialization
    By Saravanan in forum C Programming
    Replies: 20
    Last Post: 10-09-2003, 12:04 PM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21