Thread: Large 2D Array's Problem

  1. #1
    Registered User
    Join Date
    May 2002
    Posts
    208

    Large 2D Array's Problem

    Hello all,

    So I have a program for which is is necessary for me to have many (7) large arrays (order of 8000x8000) of the type long double. Obviously, this has proven difficult for a novice programmer such as myself. My pointers are global... I suppose it does not have to be this way but it made the development of the program significantly easier and I use this function to allocate the desired memory.

    Code:
    void init(){
    
    	int i =0;
    	
    	r  = malloc(sizeof(long double)*NUM_PARTICLES); rx = malloc(sizeof(long double)*NUM_PARTICLES);
    	ry = malloc(sizeof(long double)*NUM_PARTICLES); rz = malloc(sizeof(long double)*NUM_PARTICLES);
    	x  = malloc(sizeof(long double)*NUM_PARTICLES);
    	y  = malloc(sizeof(long double)*NUM_PARTICLES); z = malloc(sizeof(long double)*NUM_PARTICLES);
    	
    	for(i=0;i<NUM_PARTICLES;++i) {
    		r[i]  = malloc(sizeof(long double)*NUM_PARTICLES); rx[i] = malloc(sizeof(long double)*NUM_PARTICLES); 
    		ry[i] = malloc(sizeof(long double)*NUM_PARTICLES); rz[i] = malloc(sizeof(long double)*NUM_PARTICLES); 
    
    	}
    	
    	rAA  = malloc(sizeof(long double)*NUM_A); rAB = malloc(sizeof(long double)*NUM_A);
    	rBB = malloc(sizeof(long double)*NUM_B);
    	
    	for(i=0;i<NUM_A;++i) {
    		rAA[i]  = malloc(sizeof(long double)*NUM_A);		
    	}
    	for(i=0;i<NUM_B;++i) {
    		rAB[i]  = malloc(sizeof(long double)*NUM_B);
    		rBB[i]  = malloc(sizeof(long double)*NUM_B);
    	}
    	
    }
    The memory is freed with
    Code:
    void delete(){
    	
    		int i =0;
    		
    
    		
    		for(i=0;i<NUM_PARTICLES;++i) {
    			free(r[i]);    free(rx[i]); 
    			free(ry[i]);   free(rz[i]);
    			
    		}
    	
    	free(r); free(rx);	free(ry); free(rz);
    	free(x); free(y); free(z); 
    				
    		for(i=0;i<NUM_A;++i) {
    			free(rAA[i]);		
    		}
    		for(i=0;i<NUM_B;++i) {
    			free(rAB[i]);
    			free(rBB[i]);
    		}
    	
    	free(rAA); free(rAB); free(rBB);
    }
    Ideally I would then run through the rest of my program which does some calculations on particle positions as generated by a computer simulation and outputs the results to files. That all works fine as I tested it on very small systems. When I attempt to implement the large system, however, I was forced to switch to malloc to allocate for large data sets and I have become confused. I am receiving the following errors after my program runs for a few seconds and I must suspend my program from there.

    Code:
    *** error: can't allocate region
    *** set a breakpoint in malloc_error_break to debug
    a.out(3088) malloc: *** mmap(size=143360) failed (error code=12)
    *** error: can't allocate region
    *** set a breakpoint in malloc_error_break to debug
    a.out(3088) malloc: *** mmap(size=143360) failed (error code=12)
    *** error: can't allocate region
    *** set a breakpoint in malloc_error_break to debug
    a.out(3088) malloc: *** mmap(size=143360) failed (error code=12)
    *** error: can't allocate region
    *** set a breakpoint in malloc_error_break to debug
    a.out(3088) malloc: *** mmap(size=143360) failed (error code=12)
    *** error: can't allocate region
    *** set a breakpoint in malloc_error_break to debug
    a.out(3088) malloc: *** mmap(size=143360) failed (error code=12)
    *** error: can't allocate region
    *** set a breakpoint in malloc_error_break to debug
    a.out(3088) malloc: *** mmap(size=143360) failed (error code=12)
    *** error: can't allocate region
    *** set a breakpoint in malloc_error_break to debug
    a.out(3088) malloc: *** mmap(size=143360) failed (error code=12)
    *** error: can't allocate region
    *** set a breakpoint in malloc_error_break to debug
    a.out(3088) malloc: *** mmap(size=143360) failed (error code=12)
    *** error: can't allocate region
    *** set a breakpoint in malloc_error_break to debug
    I have no clue what's going on. Any help would be hugely appreciated.

    Cheers,
    Paddon
    Jeff Paddon
    Undergraduate Research Assistant
    Physics Department
    St. Francis Xavier University

  2. #2
    and the hat of int overfl Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    39,656
    > So I have a program for which is is necessary for me to have many (7) large arrays (order of 8000x8000) of the type long double
    Long double is 10 bytes IIRC
    10 * 8000 * 8000 * 7 = 4480000000
    That's 4.8GB, which is well past ALL the memory you could ever have on a 32-bit system.

    > r = malloc(sizeof(long double)*NUM_PARTICLES);
    Also, this should be (note, sizeof a pointer, not a long double).
    r = malloc(sizeof(long double*)*NUM_PARTICLES);
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.

  3. #3
    Registered User
    Join Date
    May 2002
    Posts
    208
    So...

    Judging by that fact. I may need to rethink how I am performing these calculations.
    Jeff Paddon
    Undergraduate Research Assistant
    Physics Department
    St. Francis Xavier University

  4. #4
    Registered User
    Join Date
    May 2007
    Posts
    147
    You may need to move into 64 bit computing here.

    Unless you can trim the precision requirements, you may have to use files - perhaps memory mapped files. Speed differences are vast, though.

    Perhaps you can calculate the work on one array at a time?

  5. #5
    Registered User
    Join Date
    May 2002
    Posts
    208
    I think I can accomplish the same thing by performing the calculations on one array at a time and then dumping the results of it to a histogram.

    Which means I would only be working with a single array of (N-1, N~8000).

    Thanks for all your help! I would have spent hours looking for errors in my code had no one enlightened me to the fact I was trying to use such a ridiculous amount of resources.
    Jeff Paddon
    Undergraduate Research Assistant
    Physics Department
    St. Francis Xavier University

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Replies: 26
    Last Post: 06-11-2009, 11:27 AM
  2. Large arrays, malloc, and strcmp
    By k2712 in forum C Programming
    Replies: 1
    Last Post: 09-24-2007, 08:22 PM
  3. returning 2D arrays
    By ... in forum C++ Programming
    Replies: 2
    Last Post: 09-02-2003, 12:28 PM
  4. Large Arrays
    By xurumin in forum C++ Programming
    Replies: 7
    Last Post: 08-26-2003, 05:17 PM
  5. 2D Arrays: Column/Row Order
    By Ashes999 in forum C++ Programming
    Replies: 1
    Last Post: 07-01-2003, 08:03 PM