Printable View
I have a question about an error in a C program.
I need to define a float array as follow:
float rad[1000][1000000];
But this error is occured:
size of variable 'rad' is too large
I need to define an array with those sizes. So, what should I do in order to solve the problem? Please help me if anyone knows the solution.
Thanks so much
I'm not surprised. You want 1000 * 1000000 * sizeof(float) bytes. That is about 4GB, which for a 32-bit application is "too large".
If you can use a 64-bit OS and a 64-bit compiler, you could probably solve the problem that way.
However, I would seriously consider your needs - because even if you manage to build an application that "works" for this size of an array, you will probably find that it's quite slow unless your machine also has a huge amount of memory [at least 5GB].
For most problems like this [that is, where you have a "need" for a large array], there are alternative solutions, where you only actually store "useful" data, and "standard values are ignored".
If you can't use a 64-bit OS for some reason, you will DEFINITELY have to rethink the design of your software.
--
Mats
You'll need to acquire the memory from the heap (general processor RAM) instead of using the stack. Look to malloc().
Todd
In a 32-bit machine, that won't make ANY difference.Quote:
Originally Posted by Todd Burch
Of course, using a approx 4GB stack based variable would be complete lunacy - even in a 64-bit system.
--
Mats
1,000 * 1,000,000 * 4 (length of float)= 4,000,000,000 = 0xEE6B 2800 (ala, < 4GB.)Quote:
Originally Posted by matsp
Why won't it work? Are lengths treated as signed values?
Todd
Could you please tell me what is a malloc() code program which is equivalent to rad[1000][1000000]
p.s. And yes, it's lunacy to request this much storage.
It is less than 4GB by a bit more than 256MB. But for any 32-bit OS that has any memory management at all, the > 3GB is sufficient to make it not work.Quote:
Originally Posted by Todd Burch
Quote:
Originally Posted by yazdan59
But that will not work any better if you don't have a machine that supports more than 3GB for application address spaces, so you need a 64-bit OS [or a custom built OS].Code:rad = malloc(sizeof(float) * 1000 * 1000000)
--
Mats
you can do one thing....
just create a memory space when it is needed usin maloc and cear of the space usig free if you r done with it....
why do you want to allocate so much of space at a time...
allocate it before you take an input.......
more details read dma
I believe each application is limited to 2 GB of (virtual) memory, aren't they (on x86)?Quote:
Originally Posted by Todd Burch
But even so, good luck allocating much more than 1 GB of memory - it just won't work, because you'll find that there's no such thing as 1 GB of contiguous virtual memory available! It's all fragmented a little bit here and there (why it doesn't allocate in a contiguous manner beats me).
No, all 32-bit operating systems can handle up to 4gb of memory(Eg, 2^32). 64-bit operating systems are thus limited to 2^64 bytes, which I think is something like 16-18 exabytes.Quote:
Originally Posted by Elysia
Please differentiate between can handle and limited/reserved. Yes, 32-bit can address 4 GB of memory, but a lot of virtual memory for each process is reserved (I believe it's the lower half of the memory space) plus Windows reserves a bit of memory space for itself, so it can't address all 4 GB.
EDIT:
Each application only has access to 2 GB virtual memory.Quote:
Operating systems based on Microsoft Windows NT technologies have always provided applications with a flat 32-bit virtual address space that describes 4 gigabytes (GB) of virtual memory. The address space is usually split so that 2 GB of address space is directly accessible to the application and the other 2 GB is only accessible to the Windows executive software.
The limitation has nothing to do with the architecture. The OS is implemented with a certain chunk of virtual address space reserved for its own purposes while in kernel mode. If all 4 gigabytes were reserved for user space, how would the kernel be able to access its internal data structures and user application data at the same time? To make things easier the kernel just says that the top gig (or two gig) of virtual address space is reserved for the kernel, period.Quote:
Originally Posted by Elysia
An OS could be written or modified to allow the user application access to more memory, but at that point it seems preferable to go to a 64-bit architecture anyway.
It's reserved because the OS says it is. There's no reason it couldn't be made available, it just isn't.Quote:
Please differentiate between can handle and limited/reserved. Yes, 32-bit can address 4 GB of memory, but a lot of virtual memory for each process is reserved (I believe it's the lower half of the memory space)
Yeah, point being, just because the OS is 32-bit doesn't mean you can access and use all that 4 GB of virtual memory. On Windows (NT), each application only has 2 GB to play around with.
So all calls with > 2GB automatically fails.
On IBM mainframes (my day job), prior to the 64 bit OS, we had (actually still have) 31 (not 32) bit addressing. An address space (aka, a logged on user for simplicity sake) was 2GB-1 bytes long. Reducing out the system resident code and the common areas of memory available to all address spaces, the rest is available for the user (or application). If you needed more memory than 2GB, you simply told the operating system you needed another address space, and thus you got another "2GB", and so on and so on. In theory, you can have 32,767 address spaces (minus the address spaces required to keep the system running.
Now with the 64-bit OS, you get 16 exabytes (10**18) virtual in each address space. It's obscene.
Todd
But lol, NEED an array of such size? For what?!Quote:
Originally Posted by yazdan59
What could possibly eat 4 GB of memory?
Perhaps you should elaborate.
Yeah, the 32nd bit is just a flag telling the OS if the address is representing a 31-bit address or a 24-bit address. The thing I'm wondering though is, shouldn't the 64-bit z/OS actually be 62-bit? Does it use some other method of knowing whether to interpret the address as 24, 31, or 64 bit?Quote:
Originally Posted by Todd Burch
No, IBM went to great lengths to make sure it was actually 64 bits. It's my understanding IBM caught a lot sneers when all the PC guys implemented 32 bits. IBM marketing didn't want that to happen again - so 64 it was.Quote:
Originally Posted by cpjust
In 64-bit, the PSW (program status word) now has a new bit (previously unused) that indicates 64 bit mode. And, the instruction address portion of the PSW is now odd as well. And, there are new instructions:
TAM - test addressing mode. It sets a cond code for what amode you are in.
SAM24 - set 24 bit amode
SAM31 - set 31 bit amode
SAM64 - set 64 bit amode.
Todd
Yes, but it's not so simple. There are two considerations. It matters both how a program is linked, and, what address mode, or modes, it chooses to operate in.Quote:
Originally Posted by Elysia
If a program is linked for 24 bit mode, it can't be loaded (resident) in any storage higher than what can be addressed by 24 bits. If linked in 31 bit mode, it can be loaded in either 24 or 31 bit storage. At this point, programs cannot execute in storage that is not addressable by 31 bits, so being resident in 64-bit storage is a moot point. Not sure if that will change in the future or not. For now, it's not a problem.
If a program is linked in 24, 31 or 64 bit mode, it can still choose to switch addressing modes if it wants or needs to. However, if it chooses to run it 24 bit mode while residing in 31 bit storage, it better not reference any storage higher than 24 bits. (not recommended protocol)
Todd
If you've got the physical memory, in the right machine, with the right OS.
http://en.wikipedia.org/wiki/Physical_Address_Extension
Yes, but for any given process, you are still limited to 32-bit addressable space, which commonly also have further limitations to allow simple comparisons to recognize kernel vs. user mode addresses, e.g. 2+2GB or 3+1GB user+kernel address mapping. [It is not only a matter of comparing addresses, but also when mapping kernel mode memory, this is shared across the entire system at all times, whilst user-mode memory is obviously "per process", and thus needs to be swapped for each process. To manage this, the kernel will want to lump all the kernel space memory in one "bucket" so that the sharing can be done simply by using the same "bucket" for all different processes - the kernel actually uses the exact same "bucket", not just a copy of the "bucket".]Quote:
Originally Posted by Salem
There are ways to use more memory, but it involves some address mapping tricks, and whilst it's probably better than "not being able to do what you want to do", it's not going to allow you to just write code with huge arrays directly in the code [like the OP wants to do], you need some sort of interface that takes a 64-bit value and maps the relevant section of memory into the "visible 32 bits".
By the way, on x86_64, in the current implementation, the address space is "limited" to 48 bits. This means that 2^48 is the limit for virtual memory [physical memory in early K8 processors is 40 bits, so "only" 1024GB]. Other 64-bit architectures may or may not be able to actually allow you to use a full 64-bit address space.
Also, since a 64-bit OS has a bigger address space, 32-bit apps may well get "more space" even in 32-bit, since the kernel addresses can now be put clearly outside the first 32 bits of address spaces, so only a small amount of "shim" is needed to translate the 32-bit OS functions into 64-bit OS - but that's some tens or hundreds of kilobytes, and nowhere near the amounts needed for the entire OS.
--
Mats
That's nice and all, but arrays need contiguous memory too.
So even if you can extend the amount of memory you can address, you won't be able to get too much memory at once in a 32-bit app, period.
> you won't be able to get too much memory at once in a 32-bit app, period.
Never fear, dun dun dun --
Don't seriously use that :\Code:size_t i = -1;
do {
ptr = malloc(i);
--i;
} while(ptr == NULL);
Yes, of course. But each individual allocation from heap memory can be in a separate block of memory. The stack and it's size needs to be determined when it is being allocated (as part of thread or process creation). It doesn't need to be committed at that time, so any "unused area of stack" doesn't necessarily occupy real memory, but the space must be reserved for the stack at start of day for that thread/process.Quote:
Originally Posted by Elysia
--
Mats
Individual allocation is fine, but trying to allocate a 2 or 4GB memory region at once is bound to fail.Quote:
Originally Posted by matsp
Don't seriously try it indeed. It will take forever to find the right size to allocate even. This code is sloooooooooooooooooooooooooooooooooooooooooow.
But if I do [although I stepped it with 4096 to not go through many millions of malloc calls all unnecessarily], I get i=2004410464 at the end of the loop. So that is just under 2GB.Quote:
Originally Posted by zacs7
--
Mats
I was hoping the 'dun dun dun' was enough sarcasm. But seriously, did you actually try it matsp? :\
How long did it take?
Hehehe... the code is still looping for me, though I had to reduce it to start from 0x7FFDEFE0, otherwise I'd get an ASSERT inside malloc or an invalid heap allocation size from HeapAlloc.
Sure, but since I reduced the loop time by 4096 times, using THIS:Quote:
Originally Posted by zacs7
It still takes a few seconds - didn't time it...Code:size_t i = -4095;
do {
ptr = malloc(i);
i -= 4096;
} while(ptr == NULL);
free(ptr);
printf("i=%d\n", i);
I also fixed the memory leak and added a printf for the result, of course. As I did this in release mode (-O2 in gcc) it shouldn't cause any asserts to fire, etc.
[Don't ever post code that says "Don't do this" ;) ]
--
Mats
Bored and lonely -- I decided to try, without cheating ;)
Or 1.3GB roughly. Long time just to find that out :(Code:zac@neux:~/devel/C/sandbox> time ./sandbox
i = 1345761301
real 15m42.793s
user 7m12.687s
sys 8m21.467s
* That gives me an idea, that'd make a pretty awesome library, libGimmeAllTheMemory -- useful for those new comers who think they need all the memory they can get.[1]
[1] Please, I'm not serious :)