how to get and use large size array.

**lehe** · 03-13-2009

Hi,
I am goint to create a very large array but have this error when running this code

Code:

#include <iostream>
#include <limits.h>
#include <cstddef>
#include <cmath>
using namespace std;

int main()
{
    cout << UINT_MAX << " " << ULLONG_MAX << endl;
    cout << pow(pow(24,2),4) << endl;

    unsigned long long int n = pow(pow(24,2),4) ;
    cout << n << endl;

    double * p = new double (pow(pow(24,2),4));
    for (unsigned long long int i = 0; i < n; i++)
        p[i] = i;
    delete [] p;

    return 0;
}

The running stops at i = 16895 giving

Debugger name and version: GNU gdb 6.8-debian
Program received signal SIGSEGV, Segmentation fault.

What's the ways to deal with large size array? I am not sure it C++ std library's various containers will work but I guess they might be too slow and not efficient, so it is slightly better to stick to normal array?

Thanks for your help!

**phantomotap** · 03-13-2009

You are only allocating a single 'double'. You need to allocate an array.

You also need a PC with a 64bit OS and a terabyte of RAM...

Soma

**lehe** · 03-13-2009

Thanks!
Isn't this

Code:

double * p = new double (pow(pow(24,2),4));

allocating a double array of size pow(pow(24,2),4)? If not, how to do it correctly?

**matsp** · 03-13-2009

Code:

double * p = new double [pow(pow(24,2),4)];

Just replace the outer parens with squre brackets.

But as phantomotap says, this array is 576 * 576 * 576 * 576 bytes, which is around 102GB, so if you haven't at the very least got a very large hard-disk with large swap-partion/swap-file and a 64-bit OS, it's going to fail.

--
Mats

**lehe** · 03-13-2009

Thanks!

Some quick question:

1. Can the index to an array be any kind of integer besides unsigned int, eg unsigned long long int? Is the maximum size of an array specified as double p[size] and double *p = new double[size]?
Can we set the max size to a higher value?

2. Are there some clever ways sometimes being able to circumvent the case when index needs to be larger than the max size?

**matsp** · 03-13-2009

The index of an array can be any integer type, long, long long or short or char for that matter. Signed or unsigned.

As for "the largest size you can allocate", it depends on a number of factors, but a 32-bit OS will not allow you to allocate more than 2^32 bytes, that's for sure. How much smaller you can get away with depends on the OS and system configuration settings (e.g. amount of memory and swap size).

And the answer to question 2 is: "Sparse arrays" or "offset arrays". Sparse arrays are arrays where only some elements are actually present, and the remaining items are either not used, or given a constant value (e.g. 0.0) when calculations are done.

Offset arrays are useful when you have an index that starts at a high number, but doesn't cover a huge range, e.g. you want to store telephone numbers from Austin, TX, whcih are all 512-XXX-YYYY, and you want to store that as an integer in the range 5120000000-5129999999 - which is 1 million entries, so doable, but if we start the array at zero, it will not work in a small-memory machine - but by subtracting 5120000000 from the index, it becomes useful.

In your case, I guess sparse arrays is the best solution.

In C++, if the number of entries is reasonable, you could create a sparse array from using std::map<int, double>. It will appear like an array of doubles with an integer index, but only those entries actually added to the map will exist in memory.

--
Mats

**lehe** · 03-13-2009

Thanks matsp! Very informative!
I was trying to understand "those entries actually added to the map will exist in memory" while map implements a sparse array. If several entries that added share a common value, map will reduce to point all of them to the same single value in the memory?
I guess std:: map could also fail if the array is large-size but not sparse. But in such case is std::map able to accommodate bigger size than a normal array?

**matsp** · 03-13-2009

std::map will not merge entries with the same value, but it will allow you to have values in a greater range than a linear array - each entry will take somewhat MORE space than the individual value.

For example, if we have 16KB of memory, a double array would be limited to 2048 entries (2048 * 8 = 16KB), but we could use:

Code:

std::map<int, double> mymap;

mymap[0] = 3.0;
mymap[2000] = 6.0;
mymap[3000] = 8.0;
mymap[180000] = 9.0;

It is of course still not able to use more than the 16KB of memory, and each entry within the array would actually take a bit more space than the basic array.

--
Mats

Thread: how to get and use large size array.

Thread Tools

Search Thread

Display

how to get and use large size array.