giant arrays

This is a discussion on giant arrays within the C++ Programming forums, part of the General Programming Boards category; When creating arrays I can't define any larger than 100 million with double. Is there a way to get even ...

  1. #1
    Registered User
    Join Date
    Jun 2008
    Posts
    6

    giant arrays

    When creating arrays I can't define any larger than 100 million with double. Is there a way to get even larger arrays or would this be unstable and I would be better off using several arrays rather than one large one?

  2. #2
    Deathray Engineer MacGyver's Avatar
    Join Date
    Mar 2007
    Posts
    3,211
    Um, assuming a double is 64-bits in size, you're trying to allocate 800,000,000 bytes on the stack. That's about 762.939453125 MB of data. In general, your stack size is very limited, from 1 MB to something about 8 MB.

    I would recommend you use new (or malloc() in C) to allocate the memory dynamically from the heap, however, if you are going to allocate that much memory..... well, rethink what you're planning on doing.

  3. #3
    int x = *((int *) NULL); Cactus_Hugger's Avatar
    Join Date
    Jul 2003
    Location
    Banks of the River Styx
    Posts
    902
    That's about 762.939453125 MB of data.
    About. You know, roughly speaking, plus or minus a few kilobytes, in the general ballpark of 762.939453125 MB.
    Might've been a copy/paste from calc. (Couldn't resist - probably time I go to sleep.)
    long time; /* know C? */
    Unprecedented performance: Nothing ever ran this slow before.
    Any sufficiently advanced bug is indistinguishable from a feature.
    Real Programmers confuse Halloween and Christmas, because dec 25 == oct 31.
    The best way to accelerate an IBM is at 9.8 m/s/s.
    recursion (re - cur' - zhun) n. 1. (see recursion)

  4. #4
    Registered User
    Join Date
    Jan 2005
    Posts
    7,317
    It sounds like Jarwulf is already using dynamic memory and running out of that.

    It doesn't matter if there are several smaller arrays or one larger one, if you need 100 million doubles then you need that much memory.

    You have to figure out if you really need over 100 million at one time and if they really need to be doubles.

  5. #5
    Algorithm Dissector iMalc's Avatar
    Join Date
    Dec 2005
    Location
    New Zealand
    Posts
    6,304
    In general when you need more than perhaps about 512MB of something you need to change your architecture to primarily store the data on disk, and only use RAM as a cache for some of the most recently needed part of that data. Like idtech6
    My homepage
    Advice: Take only as directed - If symptoms persist, please see your debugger

    Linus Torvalds: "But it clearly is the only right way. The fact that everybody else does it some other way only means that they are wrong"

  6. #6
    Registered User
    Join Date
    Jun 2008
    Posts
    6
    Okay, maybe if I describe what I'm doing we can come up with a good solution. I'm doing a population simulation of sorts. Say it consists of a billion steps. But I have to do this billion steps 100 times. After doing the billion steps 100 times I average it and come up with one billion step simulation.

    Since for every particular step I need all 100 instances before I average I can't do it piecemeal. The only solution I can think of would be to use File IO to write down every single value and then once the first cycle is finished read in every single value again, do an addition and then right it down again repeating this for every single pass. I think this would slow down the process too much.

  7. #7
    Registered User
    Join Date
    Jan 2005
    Posts
    7,317
    If you have to do all billion steps in order and can't figure any of them out from others, then I'm guessing you'll have to do something like what you described. If you use a binary file and fixed size pieces of data then it might not be too slow to read and write certain values chunks at a time. For example, you could run the first 1000 steps of the simulation, then read in the first 1000 values and add the steps you ran to them and write them back out, then continue with the next 1000 and do the same until you've done all 1 billion. You might be able to experiment with different sized chunks to see which performs fastest.
    Last edited by Daved; 06-28-2008 at 11:37 PM.

  8. #8
    Ex scientia vera
    Join Date
    Sep 2007
    Posts
    478
    It sounds like you'd want File I/O with a worker thread to do the reading for you. That'd be the fastest solution. That is, worker thread locks file, reads x from file, unlocks and waits for main thread to be done using the file handle to read more. Perhaps an auxiliary array would be best, so your worker thread supply your main thread with values all the time.

    Also, instead of copying all data from array A to B, just switch the pointers like they do in DirectX. Much faster.
    "What's up, Doc?"
    "'Up' is a relative concept. It has no intrinsic value."

  9. #9
    C++まいる!Cをこわせ! Elysia's Avatar
    Join Date
    Oct 2007
    Posts
    22,587
    The problem is that the main thread would be lightning fast compared to the read, so the read will lag behind quickly, thus eliminating the whole advantage of a second thread. It may even be slower.
    Using caching when reading/writing files, however, will give you a fair speedup. At least on Windows.

    And besides, it's easier to make it easy instead of complex.
    Quote Originally Posted by Adak View Post
    io.h certainly IS included in some modern compilers. It is no longer part of the standard for C, but it is nevertheless, included in the very latest Pelles C versions.
    Quote Originally Posted by Salem View Post
    You mean it's included as a crutch to help ancient programmers limp along without them having to relearn too much.

    Outside of your DOS world, your header file is meaningless.

  10. #10
    Registered User
    Join Date
    Jun 2008
    Posts
    106
    use a map and sparse vectors. your data probably is full of null chars, a sparse vector is thus best suited.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. pointers & arrays and realloc!
    By zesty in forum C Programming
    Replies: 14
    Last Post: 01-19-2008, 03:24 PM
  2. Replies: 16
    Last Post: 01-01-2008, 03:07 PM
  3. Dynamic (Numeric) Arrays
    By DavidB in forum C++ Programming
    Replies: 5
    Last Post: 05-03-2006, 07:34 PM
  4. Need Help With 3 Parallel Arrays Selction Sort
    By slickwilly440 in forum C++ Programming
    Replies: 4
    Last Post: 11-19-2005, 09:47 PM
  5. Crazy memory problem with arrays
    By fusikon in forum C++ Programming
    Replies: 9
    Last Post: 01-15-2003, 08:24 PM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21