Large Array Size

This is a discussion on Large Array Size within the C Programming forums, part of the General Programming Boards category; Hi, I need 2 dynamically create an array. occasionally this can mean my program is trying 2 create char arrays ...

  1. #1
    Registered User
    Join Date
    Mar 2010
    Posts
    2

    Large Array Size

    Hi,

    I need 2 dynamically create an array. occasionally this can mean my program is trying 2 create char arrays of 100+ million!!!

    This causes segmentation faults. Is it possible to either prevent the segmentation faults by freeing the memory befor hand? or to capture and recover from the error?

    Im pretty new to C so i appologise in advance if this a stupid question.

  2. #2
    Registered User
    Join Date
    Sep 2006
    Posts
    8,868
    Welcome to the forum, Smudge.

    What is your RAM (in bytes), what is your operating system, and what size of array are you able to dynamically allocate, right now?

    Why do you need such huge arrays, may I ask? No way to work on some of the data, write it out to a file, and then work on another chunk of data?

  3. #3
    and the hat of wrongness Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    32,546
    It seems to me that there is something else going on if you're getting segmentation faults.

    A segfault is a clear indication of a bug in the program. Typically, one of
    - not allocating enough memory, then stepping off the end
    - using it after it has been freed.
    - freeing it more than once
    - freeing something that wasn't allocated.
    - not checking the allocation was successful
    - for realloc, not taking into account that the block could have moved.

    Post some code showing how you allocate and use some of these allocations.


    A side note is you can run out of memory (a memory leak) by not freeing memory when you've finished with it.

    Finally, this
    Out of memory - Wikipedia, the free encyclopedia
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.
    I support http://www.ukip.org/ as the first necessary step to a free Europe.

  4. #4
    spurious conceit MK27's Avatar
    Join Date
    Jul 2008
    Location
    segmentation fault
    Posts
    8,300
    Quote Originally Posted by Salem View Post
    - not checking the allocation was successful
    Do that first, eg:
    Code:
    void *check_malloc(int size) {
           void *r = malloc(size);
           if (!r) puts("OUT OF MEMORY!!!");
           return r;
    }
    unless you are on linux, in which case it is meaningless (malloc never returns NULL there).
    C programming resources:
    GNU C Function and Macro Index -- glibc reference manual
    The C Book -- nice online learner guide
    Current ISO draft standard
    CCAN -- new CPAN like open source library repository
    3 (different) GNU debugger tutorials: #1 -- #2 -- #3
    cpwiki -- our wiki on sourceforge

  5. #5
    Registered User
    Join Date
    Mar 2010
    Posts
    2
    Im creating a program that reads 2 files and concatenates them sideways. Im using fgets to read each line, one by one. some of the files im testing on have huge lines 100 million chars per line. I could use fgetc but i can imagine tht would be pretty slow and messy to write.

    I would guess its saleems first point: not allocating enough memory, then stepping off the end.

    All i am doing is.

    char s1 [sizeOfLongestLine(argv[2]) + 2]; // seg fault here
    fgets ( s1, sizeof s1, file1 )

    The sizeOfLongestLine works fine and returns a long. when i hardedcoded array size to be 100 mill i also get the same error. Should i be doing some malloc prior to this?

    Sorry i don't know the specs of the machine this will be running on. most likely 32 bit linux with 2Gb memory

  6. #6
    spurious conceit MK27's Avatar
    Join Date
    Jul 2008
    Location
    segmentation fault
    Posts
    8,300
    You for sure cannot declare a stack variable that big. You will have to allocate heap space for it.
    Code:
    char *s1 = malloc(sizeOfLongestLine(argv[2]) + 2);
    Remember to free() this appropriately.
    C programming resources:
    GNU C Function and Macro Index -- glibc reference manual
    The C Book -- nice online learner guide
    Current ISO draft standard
    CCAN -- new CPAN like open source library repository
    3 (different) GNU debugger tutorials: #1 -- #2 -- #3
    cpwiki -- our wiki on sourceforge

  7. #7
    and the hat of sweating
    Join Date
    Aug 2007
    Location
    Toronto, ON
    Posts
    3,545
    Quote Originally Posted by MK27 View Post
    unless you are on linux, in which case it is meaningless (malloc never returns NULL there).
    I find that highly unlikely. How can malloc() never fail on Linux?
    Here it says it returns NULL: MALLOC
    "I am probably the laziest programmer on the planet, a fact with which anyone who has ever seen my code will agree." - esbo, 11/15/2008

    "the internet is a scary place to be thats why i dont use it much." - billet, 03/17/2010

  8. #8
    C++ Witch laserlight's Avatar
    Join Date
    Oct 2003
    Location
    Singapore
    Posts
    21,727
    Quote Originally Posted by cpjust
    I find that highly unlikely. How can malloc() never fail on Linux?
    Here it says it returns NULL: MALLOC
    "Never" is too strong, but if you read carefully you will see that by default, it may be the case that malloc() does not return a null pointer on Linux, and yet memory is not available.
    C + C++ Compiler: MinGW port of GCC
    Version Control System: Bazaar

    Look up a C++ Reference and learn How To Ask Questions The Smart Way

  9. #9
    spurious conceit MK27's Avatar
    Join Date
    Jul 2008
    Location
    segmentation fault
    Posts
    8,300
    The man page reflects the fact that it could return NULL, and if it does, it's because "the request failed", but unless you explicitly set your kernel in a non-standard way, the request will never fail. There's one explanation here:

    OOM::Kill Me Not Thermal Noise

    Some people see it as a feature, others find this terrifying.

    I believe the boot time switch is "overcommit" (on by default*), but I've never turned it off because I am too smart to run out of mem anyway

    * maybe that started with 2.4
    Last edited by MK27; 03-10-2010 at 12:32 PM.
    C programming resources:
    GNU C Function and Macro Index -- glibc reference manual
    The C Book -- nice online learner guide
    Current ISO draft standard
    CCAN -- new CPAN like open source library repository
    3 (different) GNU debugger tutorials: #1 -- #2 -- #3
    cpwiki -- our wiki on sourceforge

  10. #10
    and the hat of sweating
    Join Date
    Aug 2007
    Location
    Toronto, ON
    Posts
    3,545
    Quote Originally Posted by MK27 View Post
    The man page reflects the fact that it could return NULL, and if it does, it's because "the request failed", but unless you explicitly set your kernel in a non-standard way, the request will never fail. There's one explanation here:

    OOM::Kill Me Not Thermal Noise

    Some people see it as a feature, others find this terrifying.

    I believe the boot time switch is "overcommit" (on by default*), but I've never turned it off because I am too smart to run out of mem anyway

    * maybe that started with 2.4
    Wow! Now that is seriously F#'ed up! It makes sense for the 1-2GB initial process space, but that shouldn't be malloced the same way that a program malloc's memory; and a program shouldn't be mallocing more memory than it needs*.

    * well maybe a bit more, but not ridiculously huge amounts.
    "I am probably the laziest programmer on the planet, a fact with which anyone who has ever seen my code will agree." - esbo, 11/15/2008

    "the internet is a scary place to be thats why i dont use it much." - billet, 03/17/2010

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. How To Declare and Dynamically Size a Global 2D Array?
    By groberts1980 in forum C Programming
    Replies: 26
    Last Post: 11-15-2006, 08:07 AM
  2. Replies: 6
    Last Post: 11-09-2006, 02:28 AM
  3. An exercise in optimization
    By Prelude in forum Contests Board
    Replies: 10
    Last Post: 04-29-2005, 03:06 PM
  4. Replies: 42
    Last Post: 12-19-2004, 07:59 AM
  5. Quick question about SIGSEGV
    By Cikotic in forum C Programming
    Replies: 30
    Last Post: 07-01-2004, 07:48 PM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21