Allocating a number of large 3d arrays in C

This is a discussion on Allocating a number of large 3d arrays in C within the C Programming forums, part of the General Programming Boards category; Originally Posted by matsp It would be interesting to try out, but I expect I know the result - swapping ...

  1. #16
    Registered User
    Join Date
    Jul 2008
    Posts
    133
    Quote Originally Posted by matsp View Post
    It would be interesting to try out, but I expect I know the result - swapping is faster than file unless you know where you should read next.

    --
    Mats
    Yes, i expect swapping to be far more optimized too. But it does interest me, i must confess. I tryed alloc()ing 3GB on linux with 2GB, and it halted pretty strong for one minute before it crashed (I gave it 1 GB swap as a test). I bet sysrq keys were working, but i didn't want to test that - i was just curios how linux will handle such big malloc() request and how it will handle crash. Anyway, i'm disappointed a bit...
    EDIT: program crashed, not linux! But i expected more (some!) interactivity during swapping...
    EDIT2: don't get me wrong, i believe windows would "handle" it even worse.
    Last edited by rasta_freak; 08-18-2008 at 06:04 PM.

  2. #17
    and the hat of wrongness Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    32,484
    > It's a 'leapfrog' algorithm, so each iteration of my code relies on the previous one and all arrays are required.
    So in the innermost loops, you always touch all 30 arrays?
    If not, then some of them could be swapped.

    If you're accessing [0][0][0], do you ever go anywhere near [299][299][299]?
    If not, consider alternative allocations which make use of such "locality".
    Say for example only storing the current [10][300][300] slice of each cube.
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.
    I support http://www.ukip.org/ as the first necessary step to a free Europe.

  3. #18
    Registered User
    Join Date
    Aug 2008
    Posts
    5
    Quote Originally Posted by matsp View Post
    With the difference that you don't have to go through fread + fseek to get there, yes. Those are, however, in themselves not the fastest functions in the system.

    A lot would of course depend on the actual processing done, how sequential it is, etc.

    --
    Mats
    I just compared running a simple simulation with arrays allocated for 40*40*40 in Matlab and C and the difference is astounding
    Matlab: 15 minutes
    C: 19 seconds

    I think I'll roll with the file idea and see what kind of lag I get, but I'd be shocked if it was that bad.

    Thanks again folks for all the help, I don't normally code in C and I was struggling to find advice on this problem. I'll let you know how it goes.

  4. #19
    Registered User
    Join Date
    Jul 2008
    Posts
    133
    If you need to do this calculations often (or few times), you could also benefit from async I/O, or using another thread to do loading/saving. That way you would do math and I/O at a same time (probably math part will wait on I/O, but at least whole process wouldn't take as long as sum of I/O & math times).

  5. #20
    Kernel hacker
    Join Date
    Jul 2007
    Location
    Farncombe, Surrey, England
    Posts
    15,677
    Before we can advice on the correct actions for solving large memory allocations, we need to understand more of the problem.

    For example, what order are the values traversed. In some cases, Salem's suggestion may be the best solution. In other cases, perhaps slicing the array in different ways would solve the problem.

    Without understanding the patterns in which the code accesses the different arrays, then we can not give good advice as to what the correct/best solution would be.

    --
    Mats
    Compilers can produce warnings - make the compiler programmers happy: Use them!
    Please don't PM me for help - and no, I don't do help over instant messengers.

  6. #21
    Registered User
    Join Date
    Aug 2008
    Posts
    5
    Code:
    for (t = 1;t<=maxSamples;t++)
    	{
    
    		//Dx Field
    		for(x=1;x<xa;x++)
    		{
    			for(y=1;y<ySize;y++)
    			{
    				for (z=1;z<zSize;z++)
    				{
    					curl_h = ( hz[x][y][z] - hz[x][y-1][z] - hy[x][y][z] + hy[x][y][z-1]);
    					idxl[x][y][z] = idxl[x][y][z] + curl_h;
    					dx[x][y][z] = gy3[y]*gz3[z]*dx[x][y][z]  + gy2[y]*gz2[z]*.5*  (curl_h + gx1[x]*idxl[x][y][z]);
    				}
    			}
    		}
    
    		for(x=xa;x<=xb;x++)
    		{
    			for(y=1;y<ySize;y++)
    			{
    				for (z=1;z<zSize;z++)
    				{
    
    					curl_h = ( hz[x][y][z] - hz[x][y-1][z] - hy[x][y][z] + hy[x][y][z-1]);
    					dx[x][y][z] = gy3[y]*gz3[z]*dx[x][y][z]   + gy2[y]*gz2[z]*.5*curl_h ;
    
    				}
    
    			}
    
    		}
    
    		for(x=xb+1;x<xSize;x++)
    		{
    			ixh = x - xb - 1;
    			for(y=1;y<ySize;y++)
    			{
    				for (z=1;z<zSize;z++)
    				{
    
    					curl_h = ( hz[x][y][z] - hz[x][y-1][z] - hy[x][y][z] + hy[x][y][z-1]);
    					idxh[ixh][y][z] = idxh[ixh][y][z]  + curl_h;
    					dx[x][y][z]= gy3[y]*gz3[z]*dx[x][y][z] + gy2[y]*gz2[z]*.5*(curl_h + gx1[x]*idxh[ixh][y][z] );
    
    				}
    
    			}
    
    		}
    
    }
    Here is a time iteration of a snippet of my code where dx is a 300^3 array
    idxl and idxh are 7*300*300

    This is one magnetic flux calculation, there are another 7 calculations (each with 3 loop structures). Reading and writing to a file here could be problematic and may slow up the code to a point where it makes no difference if I run it in Matlab

  7. #22
    Kernel hacker
    Join Date
    Jul 2007
    Location
    Farncombe, Surrey, England
    Posts
    15,677
    So, I take it that this is in a function that is called by something else, where you are passing in one of the 30 arrays?

    Yes, reading and writing file(s) within the code will most likely reduce the efficiency of the program.

    You said earlier that you had tried this on a Linux system with a 64-bit OS. Are you using gcc to compile? If so, can you do "gcc -v" and post the output?

    --
    Mats
    Compilers can produce warnings - make the compiler programmers happy: Use them!
    Please don't PM me for help - and no, I don't do help over instant messengers.

  8. #23
    Registered User
    Join Date
    Jul 2008
    Posts
    133
    I might suggest (sorry if i'm borring) that if partial swapping of array data (between disk and ram) in-between loops around [x] might be too ineffiecient, maybe there's a USB flash disk, memory card, or solid state disk.. with better speeds so you might perhaps use that as storage... Or you could use 2,3 or more disks (or at least partitions) to put data across devices (or some combination of all that)...
    But of course, RAM would be the best option.

    EDIT: in linux, it could be possible to easily (or at least a LOT easier) use virtual filesystem that will hide this "fragmentation" and division od data parts across devices, and combine it into single coherent filesystem, so you don't have to do it in your code (you just use a single file of 3GB (or hold in RAM as much as possible, rest on "disk")). Something like custom, "wild", homemade raid.

    EDIT2:
    And just one more suggestion You can boot linux in text mode, no GUI, no most of services/daemons, no nothing, just text mode, shell, and gcc. You should then have more than 3GB really free and do calculations. You can strip down memory usage of "clean" linux to ~30MB after startup, and rest is free, no need for swap.
    Last edited by rasta_freak; 08-19-2008 at 06:28 PM.

Page 2 of 2 FirstFirst 12
Popular pages Recent additions subscribe to a feed

Similar Threads

  1. memory issue
    By t014y in forum C Programming
    Replies: 2
    Last Post: 02-20-2009, 11:37 PM
  2. Looking for constructive criticism
    By wd_kendrick in forum C Programming
    Replies: 16
    Last Post: 05-28-2008, 09:42 AM
  3. Allocating Arrays on the Free Store
    By evilkillerfiggi in forum C++ Programming
    Replies: 2
    Last Post: 12-05-2005, 03:02 PM
  4. 3D or 4D arrays
    By Unregistered in forum C Programming
    Replies: 3
    Last Post: 02-25-2002, 05:02 PM
  5. Array of boolean
    By DMaxJ in forum C++ Programming
    Replies: 11
    Last Post: 10-25-2001, 11:45 PM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21