Not enough room to store in arrays

This is a discussion on Not enough room to store in arrays within the C++ Programming forums, part of the General Programming Boards category; Hey guys, I'm doing a project where I have to take five very large arrays and sort them. But, I'm ...

  1. #1
    Registered User
    Join Date
    Jun 2004
    Posts
    10

    Not enough room to store in arrays

    Hey guys,

    I'm doing a project where I have to take five very large arrays and sort them.
    But, I'm getting stuck here.

    Here's the part that's bombing out.
    Code:
    #include <stdlib.h>
    #include <stdio.h>
    
    #define BIG 1000000000
    
    int main()
    {
    	int    a[BIG][7];
    	float  b[BIG];
    	float  x[BIG];
    	float  y[BIG];
    	double c[BIG];
    
    	return 0;
    }
    This results in
    Code:
    array_test.cpp: In function `int main()':
    array_test.cpp:8: error: size of array `a' is too large
    array_test.cpp:12: error: size of array `c' is too large
    array_test.cpp:9: error: size of variable `float b[1000000000]' is too large
    array_test.cpp:10: error: size of variable `float x[1000000000]' is too large
    array_test.cpp:11: error: size of variable `float y[1000000000]' is too large
    In my full project, the only error I get is:
    segmentation fault

    I've compiled using gcc and g++ on Red Hat Fedora 1.

    Is there any way for the arrays to accept and store this? Thanks for the help!

  2. #2
    End Of Line Hammer's Avatar
    Join Date
    Apr 2002
    Posts
    6,231
    Use dynamic memory (new/delete). But even then, that's a pretty large number of elements. Maybe allocate only the ammount you're actually using would be better. Or even better still, use the standard containers to do that for you (eg vector or something like that).
    When all else fails, read the instructions.
    If you're posting code, use code tags: [code] /* insert code here */ [/code]

  3. #3
    Registered User
    Join Date
    Jun 2004
    Posts
    10
    Quote Originally Posted by Hammer
    Use dynamic memory (new/delete). But even then, that's a pretty large number of elements. Maybe allocate only the ammount you're actually using would be better. Or even better still, use the standard containers to do that for you (eg vector or something like that).
    I'm afraid I don't understand what you mean by standard containers.

    Anyways, here's my updated snippet using the freestore to store the array in.
    Code:
    #include <stdlib.h>
    #include <stdio.h>
    
    int main()
    {
    
    	int BIG = 275000000;
    	int *pointer;
    	pointer = new int[BIG];
    
    	delete pointer;
    
    	return 0;
    }
    This works, but it doesn't like anything above 275 million.
    A problem arose when trying to put a two dimensional array into the freestore.

    I tried something like this:
    Code:
    #include <stdlib.h>
    #include <stdio.h>
    
    int main()
    {
    
    	int BIG = 1000000;
    	int *pointer;
    	pointer = new int[BIG][7];
    
    	delete pointer;
    
    	return 0;
    }
    and the error message I get is:
    Code:
    array_test.cpp: In function `int main()':
    array_test.cpp:9: error: cannot convert `int (*)[7]' to `int*' in assignment
    I tried to make the second line in main:
    int **pointer;
    ...but that resulted in a more confusing error message.

    What's the correct syntax to do this? Thanks!

  4. #4
    Registered User
    Join Date
    Jul 2004
    Posts
    19
    I think when Hammer talks of standard containers, he is talking about the STL. Maybe you can use vectors or sets to fullfill your task. You must not worry about reallocating memory, and sort can be used instead of writing your own algorithm.

  5. #5
    Registered User
    Join Date
    Jul 2004
    Posts
    101
    >>I'm afraid I don't understand what you mean by standard containers
    Classes that manage their own memory and that change their size as required. You have no doubt heard talk of the vector class? That is a standard container.

    >>but it doesn't like anything above 275 million
    Memory is finite. Assuming that integers on your machine are four bytes in length then you are already pushing the the edges of a gigabyte. Virtual memory helps, but not for long at the rate you are going. A better solution would be to work with small manageable pieces rather than the whole.

    >>What's the correct syntax to do this?
    Code:
    int (*pointer)[7];
    >>delete pointer;
    This is undefined behavior. When you use new [] to allocate memory you must use delete [] to release it.

  6. #6
    Registered User hk_mp5kpdw's Avatar
    Join Date
    Jan 2002
    Location
    Northern Virginia/Washington DC Metropolitan Area
    Posts
    3,802
    Definitly to large. 1 Billion doubles equal 8 Billion bytes (assuming 8 byte doubles) which is 7.45 Gb... far to much to fit in anybody's RAM that I know of and that's just for your c array. All three float arrays take up up 11 Gb combined and the int array takes up about 26 Gb. Add all of that up and your array allocations alone total around 44.45 Gigs of space. What are you trying to do here? Do you have any idea of how long all of that would take to sort even if you could get it to fit?

    On my work computer I tried to see what the max_size of a single vector<double> would be:

    Code:
    #include <vector>
    #include <iostream>
    using std::vector;
    using std::cout;
    using std::endl;
    
    int main()
    {
        vector<double> dVect;
        cout << "Max size is: " << dVect.max_size() << endl;
        return 0;
    }
    My results:
    Code:
    Max size is: 536870911
    This is about half of what you would need for just your c array. Your results may vary according to available RAM but I doubt even the STL containers are going to be able to help you much here. Even trying to use dynamic allocation for such large collections isn't going to work.
    Last edited by hk_mp5kpdw; 08-10-2004 at 01:09 PM.
    "Owners of dogs will have noticed that, if you provide them with food and water and shelter and affection, they will think you are god. Whereas owners of cats are compelled to realize that, if you provide them with food and water and shelter and affection, they draw the conclusion that they are gods."
    -Christopher Hitchens

  7. #7
    End Of Line Hammer's Avatar
    Join Date
    Apr 2002
    Posts
    6,231
    @Bri Rock:
    I think that a redesign is in order, is it really necessary to have all data loaded into memory at the same time? Maybe if you explain about what you're up to we can advise you better.
    When all else fails, read the instructions.
    If you're posting code, use code tags: [code] /* insert code here */ [/code]

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Searching Multiple Arrays?
    By Cell in forum C Programming
    Replies: 3
    Last Post: 03-19-2009, 03:45 PM
  2. Help with arrays and pointers please...
    By crazyeyesz28 in forum C++ Programming
    Replies: 8
    Last Post: 03-17-2005, 12:48 PM
  3. pointers
    By InvariantLoop in forum C Programming
    Replies: 13
    Last Post: 02-04-2005, 08:32 AM
  4. Pathfinding AI? (Not the A* Algorithim)
    By harryP in forum C++ Programming
    Replies: 22
    Last Post: 08-01-2003, 02:32 PM
  5. how to make arrays s->name store string ?
    By Adisonz in forum C Programming
    Replies: 8
    Last Post: 12-24-2001, 11:08 AM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21