Allocating a number of large 3d arrays in C

**rasta_freak** · 08-18-2008

Originally Posted by matsp

It would be interesting to try out, but I expect I know the result - swapping is faster than file unless you know where you should read next.

--
Mats

Yes, i expect swapping to be far more optimized too. But it does interest me, i must confess. I tryed alloc()ing 3GB on linux with 2GB, and it halted pretty strong for one minute before it crashed (I gave it 1 GB swap as a test). I bet sysrq keys were working, but i didn't want to test that - i was just curios how linux will handle such big malloc() request and how it will handle crash. Anyway, i'm disappointed a bit...
EDIT: program crashed, not linux! But i expected more (some!) interactivity during swapping...
EDIT2: don't get me wrong, i believe windows would "handle" it even worse.

**Salem** · 08-18-2008

> It's a 'leapfrog' algorithm, so each iteration of my code relies on the previous one and all arrays are required.
So in the innermost loops, you always touch all 30 arrays?
If not, then some of them could be swapped.

If you're accessing [0][0][0], do you ever go anywhere near [299][299][299]?
If not, consider alternative allocations which make use of such "locality".
Say for example only storing the current [10][300][300] slice of each cube.

**Surrender** · 08-19-2008

Originally Posted by matsp

With the difference that you don't have to go through fread + fseek to get there, yes. Those are, however, in themselves not the fastest functions in the system.

A lot would of course depend on the actual processing done, how sequential it is, etc.

--
Mats

I just compared running a simple simulation with arrays allocated for 40*40*40 in Matlab and C and the difference is astounding
Matlab: 15 minutes
C: 19 seconds

I think I'll roll with the file idea and see what kind of lag I get, but I'd be shocked if it was that bad.

Thanks again folks for all the help, I don't normally code in C and I was struggling to find advice on this problem. I'll let you know how it goes.

**rasta_freak** · 08-19-2008

If you need to do this calculations often (or few times), you could also benefit from async I/O, or using another thread to do loading/saving. That way you would do math and I/O at a same time (probably math part will wait on I/O, but at least whole process wouldn't take as long as sum of I/O & math times).

**matsp** · 08-19-2008

Before we can advice on the correct actions for solving large memory allocations, we need to understand more of the problem.

For example, what order are the values traversed. In some cases, Salem's suggestion may be the best solution. In other cases, perhaps slicing the array in different ways would solve the problem.

Without understanding the patterns in which the code accesses the different arrays, then we can not give good advice as to what the correct/best solution would be.

--
Mats

**Surrender** · 08-19-2008

Code:

for (t = 1;t<=maxSamples;t++)
	{

		//Dx Field
		for(x=1;x<xa;x++)
		{
			for(y=1;y<ySize;y++)
			{
				for (z=1;z<zSize;z++)
				{
					curl_h = ( hz[x][y][z] - hz[x][y-1][z] - hy[x][y][z] + hy[x][y][z-1]);
					idxl[x][y][z] = idxl[x][y][z] + curl_h;
					dx[x][y][z] = gy3[y]*gz3[z]*dx[x][y][z]  + gy2[y]*gz2[z]*.5*  (curl_h + gx1[x]*idxl[x][y][z]);
				}
			}
		}

		for(x=xa;x<=xb;x++)
		{
			for(y=1;y<ySize;y++)
			{
				for (z=1;z<zSize;z++)
				{

					curl_h = ( hz[x][y][z] - hz[x][y-1][z] - hy[x][y][z] + hy[x][y][z-1]);
					dx[x][y][z] = gy3[y]*gz3[z]*dx[x][y][z]   + gy2[y]*gz2[z]*.5*curl_h ;

				}

			}

		}

		for(x=xb+1;x<xSize;x++)
		{
			ixh = x - xb - 1;
			for(y=1;y<ySize;y++)
			{
				for (z=1;z<zSize;z++)
				{

					curl_h = ( hz[x][y][z] - hz[x][y-1][z] - hy[x][y][z] + hy[x][y][z-1]);
					idxh[ixh][y][z] = idxh[ixh][y][z]  + curl_h;
					dx[x][y][z]= gy3[y]*gz3[z]*dx[x][y][z] + gy2[y]*gz2[z]*.5*(curl_h + gx1[x]*idxh[ixh][y][z] );

				}

			}

		}

}

Here is a time iteration of a snippet of my code where dx is a 300^3 array
idxl and idxh are 7*300*300

This is one magnetic flux calculation, there are another 7 calculations (each with 3 loop structures). Reading and writing to a file here could be problematic and may slow up the code to a point where it makes no difference if I run it in Matlab

**matsp** · 08-19-2008

So, I take it that this is in a function that is called by something else, where you are passing in one of the 30 arrays?

Yes, reading and writing file(s) within the code will most likely reduce the efficiency of the program.

You said earlier that you had tried this on a Linux system with a 64-bit OS. Are you using gcc to compile? If so, can you do "gcc -v" and post the output?

--
Mats

**rasta_freak** · 08-19-2008

I might suggest (sorry if i'm borring) that if partial swapping of array data (between disk and ram) in-between loops around [x] might be too ineffiecient, maybe there's a USB flash disk, memory card, or solid state disk.. with better speeds so you might perhaps use that as storage... Or you could use 2,3 or more disks (or at least partitions) to put data across devices (or some combination of all that)...
But of course, RAM would be the best option.

EDIT: in linux, it could be possible to easily (or at least a LOT easier) use virtual filesystem that will hide this "fragmentation" and division od data parts across devices, and combine it into single coherent filesystem, so you don't have to do it in your code (you just use a single file of 3GB (or hold in RAM as much as possible, rest on "disk")). Something like custom, "wild", homemade raid.

EDIT2:
And just one more suggestion

You can boot linux in text mode, no GUI, no most of services/daemons, no nothing, just text mode, shell, and gcc. You should then have more than 3GB really free and do calculations. You can strip down memory usage of "clean" linux to ~30MB after startup, and rest is free, no need for swap.

Thread: Allocating a number of large 3d arrays in C

Thread Tools

Search Thread

Display

Similar Threads

memory issue

Looking for constructive criticism

Allocating Arrays on the Free Store

3D or 4D arrays

Array of boolean