Big 2D array

This is a discussion on Big 2D array within the C Programming forums, part of the General Programming Boards category; I need to make a 1x128,000,000 element array (it looks like) Code: [1] [2] [3] [4] ... [128000000] but i ...

  1. #1
    Wanabe Laser Engineer chico1st's Avatar
    Join Date
    Jul 2007
    Posts
    168

    Big 2D array

    I need to make a 1x128,000,000 element array (it looks like)

    Code:
    [1]
    [2]
    [3]
    [4]
    ...
    [128000000]
    but i cant seem to declare 128,000,000 elements my C program seems to crash. As soon as I try.

    Here is the declaring line of code (i dont have code for this array any where else yet.)

    Code:
    int TransposeData[1][128];
    I do get this error when I compile but i ignored it

    Code:
    C:\...cpp(256) : warning C4101: 'TransposeData' : unreferenced local variable
    can i use an array this big?

  2. #2
    Registered User
    Join Date
    Sep 2006
    Posts
    835
    It's very unlikely you can allocate that much memory statically (normally there's a limit of a few meg for that). Try allocating it dynamically instead, using malloc().

  3. #3
    Kernel hacker
    Join Date
    Jul 2007
    Location
    Farncombe, Surrey, England
    Posts
    15,677
    That is awfully close to 2GB, so if your OS + compiler aren't both 64-bit, I'd expect that to fail due to memory limitiations for the data segment.

    --
    Mats

  4. #4
    Deathray Engineer MacGyver's Avatar
    Join Date
    Mar 2007
    Posts
    3,211
    If you need that big of an array for a simple project, you're probably doing something wrong with your design.

  5. #5
    Kernel hacker
    Join Date
    Jul 2007
    Location
    Farncombe, Surrey, England
    Posts
    15,677
    Maybe you can descibe a bit about what you are trying to do. It is most likely that such a large array wouldn't be OK in most computers, even if the application runs OK on your machine. I have "only" got 2GB in my machine, and most machines have less than that. If you use arrays larger than the amount of memory available in the machine, you get "swapping", which is terrible for performance - 10000x slower or so than using "real memory".

    So you probably should consider a different solution. Can you explain why you think you need such a big array?

    --
    Mats

  6. #6
    Registered User
    Join Date
    Sep 2006
    Posts
    835
    If each element is 4 bytes, say, that's only 512 MB. When my 32-bit Linux box had 512 MB RAM and 1 GB swap, I could malloc about 1.3 GB (I just increased the memory to 2 GB and my maximum malloc() only increased to about 1.9 GB, don't know why it's not higher).

  7. #7
    Wanabe Laser Engineer chico1st's Avatar
    Join Date
    Jul 2007
    Posts
    168
    ok the thread is now over here
    Big 2D array

  8. #8
    Kernel hacker
    Join Date
    Jul 2007
    Location
    Farncombe, Surrey, England
    Posts
    15,677
    Quote Originally Posted by robatino View Post
    If each element is 4 bytes, say, that's only 512 MB. When my 32-bit Linux box had 512 MB RAM and 1 GB swap, I could malloc about 1.3 GB (I just increased the memory to 2 GB and my maximum malloc() only increased to about 1.9 GB, don't know why it's not higher).
    Yes, of course - I must have multiplied by 4 twice. .. :-(

    I guess that you have a kernel that is configured to 2GB user and 2GB kernel-space, perhaps?

    --
    Mats

  9. #9
    Registered User
    Join Date
    Sep 2006
    Posts
    835
    > I guess that you have a kernel that is configured to 2GB user and 2GB kernel-space, perhaps?
    I don't know - didn't realize that userspace was typically limited that way, thought it could go all the way up to 4 GB (my swap is still 1 GB, my next clean install in a few months will probably result in a larger swap partition when it sees the extra memory).

  10. #10
    Wanabe Laser Engineer chico1st's Avatar
    Join Date
    Jul 2007
    Posts
    168
    the reason that I think i need this giant array is that i have a special card in my computer which takes 250Million measurements and puts them into an array. (im not using all the measurements.)

    that part is automated.
    I need to take that array and Transpose it:

    x,x,x,x goes to
    x,x,x,x

    x,x
    x,x
    x,x
    x,x


    Then read it into a binary file, i might be able to avoid this giant array if reading 1 large array into a binary file is the same as reading a couple smaller ones. I dont know if the output is the same though... nor do i know how to check because i cant read it

    *thats the basics.. my program isnt that lame*

  11. #11
    Kernel hacker
    Join Date
    Jul 2007
    Location
    Farncombe, Surrey, England
    Posts
    15,677
    So, you have some sort of measurement card in the machine, and you want to get the data out of that onto a file on disk, correct?

    How long does it take to get those measurements? A few milliseconds, seconds, minutes, hour? Once the data is captured, are you needing to get MORE data in, or is it "that's all folks!" at that point?

    It will quite obviously take roughly the same time to save one 128M entry table as it takes to save 128 1M entry tables - not EXACTLY, but within a few percent. The only difference is the amount of memory needed to complete such a transaction.

    By the way, are you trying to have this array inside a function or as a global? If you try to create it locally in a function, I can understand that it crashes (because no 32-bit OS I know of supports a stack of 512MB - not sure about 64-bit Linux or Windows - it may support that large a stack). I just created a global variable of 128000000 integers, with no problem - I can even fill it with zero's with no problem (except it takes a little bit of time from the program starts until the print after the clearing of the array - a few tenths of a second or so).

    --
    Mats

  12. #12
    Wanabe Laser Engineer chico1st's Avatar
    Join Date
    Jul 2007
    Posts
    168
    The card takes the data and sticks it into an array in 1s. This issue is that I have to transpose all the data, so i have to use my normal computer to do that which is where the giant array comes from.

    Actually the array will be 20x128M eventually but that might just be too crazy. This computer is a beast though so it just might fly.

    Using a global array is fine. So if that makes it easier I will do that, thanks

    I didnt get to work on it today, so i will report back once i try all this out.

  13. #13
    Kernel hacker
    Join Date
    Jul 2007
    Location
    Farncombe, Surrey, England
    Posts
    15,677
    Reading and processing 512M in a second shouldn't be too hard, as long as you're not doing MUCH calculation on each step.

    20 * 512M is 10G, that's a lot - most "normal" single processor machines have at most 4 slots for memory, which given that 2GB stick is about as large as you can get (4GB sticks exist, but they are HIDEOUSLY expensive, and quite hard to find), that gives 8GB. Using 10GB of memory on a 8GB machine is a bad idea.

    Machines with multiple sockets can usually take more memory sticks, 8 or 16 sticks are common numbers.

    You also need a 64-bit OS to be able to perform operations on such large arrays - nothing else will work, 32-bit OS's are commonly supporting 2GB or 3GB - theoretically up to 4GB, but all OS's need "kernel space" for the OS itself, drivers and such.

    --
    Mats

  14. #14
    Frequently Quite Prolix dwks's Avatar
    Join Date
    Apr 2005
    Location
    Canada
    Posts
    8,046
    Quote Originally Posted by chico1st View Post
    the reason that I think i need this giant array is that i have a special card in my computer which takes 250Million measurements and puts them into an array. (im not using all the measurements.)

    that part is automated.
    I need to take that array and Transpose it:

    x,x,x,x goes to
    x,x,x,x

    x,x
    x,x
    x,x
    x,x


    Then read it into a binary file, i might be able to avoid this giant array if reading 1 large array into a binary file is the same as reading a couple smaller ones. I dont know if the output is the same though... nor do i know how to check because i cant read it

    *thats the basics.. my program isnt that lame*
    Reading an array from a binary file is just like reading one variable of the array's type over and over again. This
    Code:
    int array[10];
    
    fread(array, sizeof(*array), sizeof(array)/sizeof(*array), fp);
    /* or, more readably but less good-practicability */
    fread(array, sizeof(int), 10, fp);
    is the same as
    Code:
    int array[10];
    
    fread(array, sizeof(int), 5, fp);
    fread(&array[5], sizeof(int), 5, fp);
    which is the same as
    Code:
    int array[10], x;
    
    for(x = 0; x < 10; x ++) {
        fread(&array[x], sizeof(int), 1, fp);
    }
    See this reference for fread(). http://www.cplusplus.com/reference/c...dio/fread.html

    But if all you're doing is . . . well, transposing some numbers, then you don't even need to store the data in an array. Just read a couple of integers and repeat until EOF.
    dwk

    Seek and ye shall find. quaere et invenies.

    "Simplicity does not precede complexity, but follows it." -- Alan Perlis
    "Testing can only prove the presence of bugs, not their absence." -- Edsger Dijkstra
    "The only real mistake is the one from which we learn nothing." -- John Powell


    Other boards: DaniWeb, TPS
    Unofficial Wiki FAQ: cpwiki.sf.net

    My website: http://dwks.theprogrammingsite.com/
    Projects: codeform, xuni, atlantis, nort, etc.

  15. #15
    Algorithm Dissector iMalc's Avatar
    Join Date
    Dec 2005
    Location
    New Zealand
    Posts
    6,304
    Quote Originally Posted by chico1st View Post
    The card takes the data and sticks it into an array in 1s.
    Is it connected to the sensors of a particle accelerator or something?!?!
    My homepage
    Advice: Take only as directed - If symptoms persist, please see your debugger

    Linus Torvalds: "But it clearly is the only right way. The fact that everybody else does it some other way only means that they are wrong"

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Replies: 2
    Last Post: 07-11-2008, 07:39 AM
  2. question about multidimensional arrays
    By richdb in forum C Programming
    Replies: 22
    Last Post: 02-26-2006, 08:51 AM
  3. malloced 2d array passed to qsort
    By jamie85 in forum C Programming
    Replies: 7
    Last Post: 11-25-2005, 12:55 PM
  4. Text file to 2D Array PLEASE HELP!
    By lostboy101 in forum C Programming
    Replies: 0
    Last Post: 03-26-2002, 09:51 AM
  5. 2d Array access by other classes
    By deaths_seraphim in forum C++ Programming
    Replies: 1
    Last Post: 10-02-2001, 08:05 AM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21