how to get and use large size array.

This is a discussion on how to get and use large size array. within the C++ Programming forums, part of the General Programming Boards category; Hi, I am goint to create a very large array but have this error when running this code Code: #include ...

  1. #1
    Registered User
    Join Date
    Jan 2009
    Posts
    159

    how to get and use large size array.

    Hi,
    I am goint to create a very large array but have this error when running this code
    Code:
    #include <iostream>
    #include <limits.h>
    #include <cstddef>
    #include <cmath>
    using namespace std;
    
    int main()
    {
        cout << UINT_MAX << " " << ULLONG_MAX << endl;
        cout << pow(pow(24,2),4) << endl;
    
        unsigned long long int n = pow(pow(24,2),4) ;
        cout << n << endl;
    
        double * p = new double (pow(pow(24,2),4));
        for (unsigned long long int i = 0; i < n; i++)
            p[i] = i;
        delete [] p;
    
        return 0;
    }
    The running stops at i = 16895 giving
    Debugger name and version: GNU gdb 6.8-debian
    Program received signal SIGSEGV, Segmentation fault.
    What's the ways to deal with large size array? I am not sure it C++ std library's various containers will work but I guess they might be too slow and not efficient, so it is slightly better to stick to normal array?

    Thanks for your help!

  2. #2
    Master Apprentice phantomotap's Avatar
    Join Date
    Jan 2008
    Posts
    4,359
    You are only allocating a single 'double'. You need to allocate an array.

    You also need a PC with a 64bit OS and a terabyte of RAM...

    Soma

  3. #3
    Registered User
    Join Date
    Jan 2009
    Posts
    159
    Thanks!
    Isn't this
    Code:
    double * p = new double (pow(pow(24,2),4));
    allocating a double array of size pow(pow(24,2),4)? If not, how to do it correctly?

  4. #4
    Kernel hacker
    Join Date
    Jul 2007
    Location
    Farncombe, Surrey, England
    Posts
    15,677
    Code:
    double * p = new double [pow(pow(24,2),4)];
    Just replace the outer parens with squre brackets.

    But as phantomotap says, this array is 576 * 576 * 576 * 576 bytes, which is around 102GB, so if you haven't at the very least got a very large hard-disk with large swap-partion/swap-file and a 64-bit OS, it's going to fail.

    --
    Mats
    Compilers can produce warnings - make the compiler programmers happy: Use them!
    Please don't PM me for help - and no, I don't do help over instant messengers.

  5. #5
    Registered User
    Join Date
    Jan 2009
    Posts
    159
    Thanks!

    Some quick question:

    1. Can the index to an array be any kind of integer besides unsigned int, eg unsigned long long int? Is the maximum size of an array specified as double p[size] and double *p = new double[size]?
    Can we set the max size to a higher value?

    2. Are there some clever ways sometimes being able to circumvent the case when index needs to be larger than the max size?

  6. #6
    Kernel hacker
    Join Date
    Jul 2007
    Location
    Farncombe, Surrey, England
    Posts
    15,677
    The index of an array can be any integer type, long, long long or short or char for that matter. Signed or unsigned.

    As for "the largest size you can allocate", it depends on a number of factors, but a 32-bit OS will not allow you to allocate more than 2^32 bytes, that's for sure. How much smaller you can get away with depends on the OS and system configuration settings (e.g. amount of memory and swap size).

    And the answer to question 2 is: "Sparse arrays" or "offset arrays". Sparse arrays are arrays where only some elements are actually present, and the remaining items are either not used, or given a constant value (e.g. 0.0) when calculations are done.

    Offset arrays are useful when you have an index that starts at a high number, but doesn't cover a huge range, e.g. you want to store telephone numbers from Austin, TX, whcih are all 512-XXX-YYYY, and you want to store that as an integer in the range 5120000000-5129999999 - which is 1 million entries, so doable, but if we start the array at zero, it will not work in a small-memory machine - but by subtracting 5120000000 from the index, it becomes useful.

    In your case, I guess sparse arrays is the best solution.

    In C++, if the number of entries is reasonable, you could create a sparse array from using std::map<int, double>. It will appear like an array of doubles with an integer index, but only those entries actually added to the map will exist in memory.

    --
    Mats
    Compilers can produce warnings - make the compiler programmers happy: Use them!
    Please don't PM me for help - and no, I don't do help over instant messengers.

  7. #7
    Registered User
    Join Date
    Jan 2009
    Posts
    159
    Thanks matsp! Very informative!
    I was trying to understand "those entries actually added to the map will exist in memory" while map implements a sparse array. If several entries that added share a common value, map will reduce to point all of them to the same single value in the memory?
    I guess std:: map could also fail if the array is large-size but not sparse. But in such case is std::map able to accommodate bigger size than a normal array?

  8. #8
    Kernel hacker
    Join Date
    Jul 2007
    Location
    Farncombe, Surrey, England
    Posts
    15,677
    std::map will not merge entries with the same value, but it will allow you to have values in a greater range than a linear array - each entry will take somewhat MORE space than the individual value.

    For example, if we have 16KB of memory, a double array would be limited to 2048 entries (2048 * 8 = 16KB), but we could use:
    Code:
    std::map<int, double> mymap;
    
    mymap[0] = 3.0;
    mymap[2000] = 6.0;
    mymap[3000] = 8.0;
    mymap[180000] = 9.0;
    It is of course still not able to use more than the 16KB of memory, and each entry within the array would actually take a bit more space than the basic array.

    --
    Mats
    Compilers can produce warnings - make the compiler programmers happy: Use them!
    Please don't PM me for help - and no, I don't do help over instant messengers.

Popular pages Recent additions subscribe to a feed

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21