Big 2D array

**chico1st** · 08-01-2007

I need to make a 1x128,000,000 element array (it looks like)

Code:

[1]
[2]
[3]
[4]
...
[128000000]

but i cant seem to declare 128,000,000 elements my C program seems to crash. As soon as I try.

Here is the declaring line of code (i dont have code for this array any where else yet.)

Code:

int TransposeData[1][128];

I do get this error when I compile but i ignored it

Code:

C:\...cpp(256) : warning C4101: 'TransposeData' : unreferenced local variable

can i use an array this big?

**robatino** · 08-01-2007

It's very unlikely you can allocate that much memory statically (normally there's a limit of a few meg for that). Try allocating it dynamically instead, using malloc().

**matsp** · 08-01-2007

That is awfully close to 2GB, so if your OS + compiler aren't both 64-bit, I'd expect that to fail due to memory limitiations for the data segment.

--
Mats

**MacGyver** · 08-01-2007

If you need that big of an array for a simple project, you're probably doing something wrong with your design.

**matsp** · 08-01-2007

Maybe you can descibe a bit about what you are trying to do. It is most likely that such a large array wouldn't be OK in most computers, even if the application runs OK on your machine. I have "only" got 2GB in my machine, and most machines have less than that. If you use arrays larger than the amount of memory available in the machine, you get "swapping", which is terrible for performance - 10000x slower or so than using "real memory".

So you probably should consider a different solution. Can you explain why you think you need such a big array?

--
Mats

**robatino** · 08-01-2007

If each element is 4 bytes, say, that's only 512 MB. When my 32-bit Linux box had 512 MB RAM and 1 GB swap, I could malloc about 1.3 GB (I just increased the memory to 2 GB and my maximum malloc() only increased to about 1.9 GB, don't know why it's not higher).

**chico1st** · 08-01-2007

ok the thread is now over here
http://cboard.cprogramming.com/showthread.php?t=92289

**matsp** · 08-01-2007

Originally Posted by robatino

If each element is 4 bytes, say, that's only 512 MB. When my 32-bit Linux box had 512 MB RAM and 1 GB swap, I could malloc about 1.3 GB (I just increased the memory to 2 GB and my maximum malloc() only increased to about 1.9 GB, don't know why it's not higher).

Yes, of course - I must have multiplied by 4 twice. .. :-(

I guess that you have a kernel that is configured to 2GB user and 2GB kernel-space, perhaps?

--
Mats

**robatino** · 08-01-2007

> I guess that you have a kernel that is configured to 2GB user and 2GB kernel-space, perhaps?
I don't know - didn't realize that userspace was typically limited that way, thought it could go all the way up to 4 GB (my swap is still 1 GB, my next clean install in a few months will probably result in a larger swap partition when it sees the extra memory).

**chico1st** · 08-01-2007

the reason that I think i need this giant array is that i have a special card in my computer which takes 250Million measurements and puts them into an array. (im not using all the measurements.)

that part is automated.
I need to take that array and Transpose it:

x,x,x,x goes to
x,x,x,x

x,x
x,x
x,x
x,x

Then read it into a binary file, i might be able to avoid this giant array if reading 1 large array into a binary file is the same as reading a couple smaller ones. I dont know if the output is the same though... nor do i know how to check because i cant read it

*thats the basics.. my program isnt that lame*

**matsp** · 08-01-2007

So, you have some sort of measurement card in the machine, and you want to get the data out of that onto a file on disk, correct?

How long does it take to get those measurements? A few milliseconds, seconds, minutes, hour? Once the data is captured, are you needing to get MORE data in, or is it "that's all folks!" at that point?

It will quite obviously take roughly the same time to save one 128M entry table as it takes to save 128 1M entry tables - not EXACTLY, but within a few percent. The only difference is the amount of memory needed to complete such a transaction.

By the way, are you trying to have this array inside a function or as a global? If you try to create it locally in a function, I can understand that it crashes (because no 32-bit OS I know of supports a stack of 512MB - not sure about 64-bit Linux or Windows - it may support that large a stack). I just created a global variable of 128000000 integers, with no problem - I can even fill it with zero's with no problem (except it takes a little bit of time from the program starts until the print after the clearing of the array - a few tenths of a second or so).

--
Mats

**chico1st** · 08-02-2007

The card takes the data and sticks it into an array in 1s. This issue is that I have to transpose all the data, so i have to use my normal computer to do that which is where the giant array comes from.

Actually the array will be 20x128M eventually but that might just be too crazy. This computer is a beast though so it just might fly.

Using a global array is fine. So if that makes it easier I will do that, thanks

I didnt get to work on it today, so i will report back once i try all this out.

**matsp** · 08-02-2007

Reading and processing 512M in a second shouldn't be too hard, as long as you're not doing MUCH calculation on each step.

20 * 512M is 10G, that's a lot - most "normal" single processor machines have at most 4 slots for memory, which given that 2GB stick is about as large as you can get (4GB sticks exist, but they are HIDEOUSLY expensive, and quite hard to find), that gives 8GB. Using 10GB of memory on a 8GB machine is a bad idea.

Machines with multiple sockets can usually take more memory sticks, 8 or 16 sticks are common numbers.

You also need a 64-bit OS to be able to perform operations on such large arrays - nothing else will work, 32-bit OS's are commonly supporting 2GB or 3GB - theoretically up to 4GB, but all OS's need "kernel space" for the OS itself, drivers and such.

--
Mats

**dwks** · 08-02-2007

Originally Posted by chico1st

the reason that I think i need this giant array is that i have a special card in my computer which takes 250Million measurements and puts them into an array. (im not using all the measurements.)

that part is automated.
I need to take that array and Transpose it:

x,x,x,x goes to
x,x,x,x

x,x
x,x
x,x
x,x

Then read it into a binary file, i might be able to avoid this giant array if reading 1 large array into a binary file is the same as reading a couple smaller ones. I dont know if the output is the same though... nor do i know how to check because i cant read it

*thats the basics.. my program isnt that lame*

Reading an array from a binary file is just like reading one variable of the array's type over and over again. This

Code:

int array[10];

fread(array, sizeof(*array), sizeof(array)/sizeof(*array), fp);
/* or, more readably but less good-practicability */
fread(array, sizeof(int), 10, fp);

is the same as

Code:

int array[10];

fread(array, sizeof(int), 5, fp);
fread(&array[5], sizeof(int), 5, fp);

which is the same as

Code:

int array[10], x;

for(x = 0; x < 10; x ++) {
    fread(&array[x], sizeof(int), 1, fp);
}

See this reference for fread(). http://www.cplusplus.com/reference/c...dio/fread.html

But if all you're doing is . . . well, transposing some numbers, then you don't even need to store the data in an array. Just read a couple of integers and repeat until EOF.

**iMalc** · 08-03-2007

Originally Posted by chico1st

The card takes the data and sticks it into an array in 1s.

Is it connected to the sensors of a particle accelerator or something?!?!

Thread: Big 2D array

Thread Tools

Search Thread

Display

Big 2D array

Similar Threads

Dynamic array of structures containing yet another dynamic array of structures

question about multidimensional arrays

malloced 2d array passed to qsort

Text file to 2D Array PLEASE HELP!

2d Array access by other classes