Available memory size

Hi,
I'm building an application which needs to dynamicall create memory entities - each of 10MB in size. In the heap, not using virtual memory.
I'd like to evaluate, before the memory objects creation starts, what would be the max (just evaluation) number that can be created, thus I need to get the process' available heap at the instant.
How can I get such info, if possible, for Windows environments (XP, Vista, Server 2008 ...). I have some hints but I'd like to have your expert opinion.
Many thanks!

System Information Functions (Windows)

AFAIK theres no way to get the amount of physical ram that is 'free' vs the amount that will be virtual, because any number returned would be immediately stale, the OS pages memory on an as needed basis, so there is no way of determining how much s allocated to your application right at that moment short of a kernel hack.

Then again I could be wrong, the API is pretty big.

http://msdn.microsoft.com/en-us/libr...89(VS.85).aspx

Code:

---

//  Sample output:
//  There is      51 percent of memory in use.
//  There are 2029968 total Kbytes of physical memory.
//  There are  987388 free Kbytes of physical memory.
//  There are 3884620 total Kbytes of paging file.
//  There are 2799776 free Kbytes of paging file.
//  There are 2097024 total Kbytes of virtual memory.
//  There are 2084876 free Kbytes of virtual memory.
//  There are      0 free Kbytes of extended memory.

#include <windows.h>
#include <stdio.h>

// Use to convert bytes to KB
#define DIV 1024

// Specify the width of the field in which to print the numbers.
// The asterisk in the format specifier "%*I64d" takes an integer
// argument and uses it to pad and right justify the number.
#define WIDTH 7

void main(int argc, char *argv[])
{
  MEMORYSTATUSEX statex;

  statex.dwLength = sizeof (statex);

  GlobalMemoryStatusEx (&statex);

  printf ("There is  %*ld percent of memory in use.\n",
          WIDTH, statex.dwMemoryLoad);
  printf ("There are %*I64d total Kbytes of physical memory.\n",
          WIDTH, statex.ullTotalPhys/DIV);
  printf ("There are %*I64d free Kbytes of physical memory.\n",
          WIDTH, statex.ullAvailPhys/DIV);
  printf ("There are %*I64d total Kbytes of paging file.\n",
          WIDTH, statex.ullTotalPageFile/DIV);
  printf ("There are %*I64d free Kbytes of paging file.\n",
          WIDTH, statex.ullAvailPageFile/DIV);
  printf ("There are %*I64d total Kbytes of virtual memory.\n",
          WIDTH, statex.ullTotalVirtual/DIV);
  printf ("There are %*I64d free Kbytes of virtual memory.\n",
          WIDTH, statex.ullAvailVirtual/DIV);

  // Show the amount of extended memory available.

  printf ("There are %*I64d free Kbytes of extended memory.\n",
          WIDTH, statex.ullAvailExtendedVirtual/DIV);
}

---

Yeah, that's system one.
What I need is info about the process I'm calling from.
I found a "way around" i.e. looping with malloc increasing the size and exiting when getting out of memory (NULL), but I wonder there's no process info structure available. The system should know it so that to manage the process exec space. So where is it ;-) ?

the OS uses the global descriptor table and the local descriptor tables to manage physical and virtual memory. Applications do not have access to those structures as they are only available in ring 0. The OS doesn't know how much of your specific program is in memory or not as such, it just knows globally which memory locations are in physical memory and which are on disk, It would have to specifically do a page walk to collect that information, which would be of no use to it, so its not something the kernel would do, but it is something only the kernel can do.

There is no reason you would need to know this information. The kernel handles paging transparently. Whatever it reports as free system wide physical memory is what is available to your process as physical memory. Each process has access to all available physical memory up to its per process limit, which is 2GB normally or 3GB if you use the /3gb switch. However, just because it reports, e.g. 2GB free physical memory, doesn't mean if you allocate a 2GB object that the entire object will be in physical ram, the OS will handle what is and is not paged to optimize system wide performance. I believe there is a way of marking heap memory as being non-paged though, check out the heap functions.

Ok fine now, your thorough comments confirms what I found on msdn. FYI, I just want to anticipate the apps limits before the crash test, so that's the reason.
Thanks!

Quote:

---

Originally Posted by abachler

the OS uses the global descriptor table and the local descriptor tables to manage physical and virtual memory. Applications do not have access to those structures as they are only available in ring 0. The OS doesn't know how much of your specific program is in memory or not as such, it just knows globally which memory locations are in physical memory and which are on disk, It would have to specifically do a page walk to collect that information, which would be of no use to it, so its not something the kernel would do, but it is something only the kernel can do.

There is no reason you would need to know this information. The kernel handles paging transparently. Whatever it reports as free system wide physical memory is what is available to your process as physical memory. Each process has access to all available physical memory up to its per process limit, which is 2GB normally or 3GB if you use the /3gb switch. However, just because it reports, e.g. 2GB free physical memory, doesn't mean if you allocate a 2GB object that the entire object will be in physical ram, the OS will handle what is and is not paged to optimize system wide performance. I believe there is a way of marking heap memory as being non-paged though, check out the heap functions.

---

2 GB for an x86 OS, but that is not required for an x64 OS.

Quote:

---

Originally Posted by Elysia

2 GB for an x86 OS, but that is not required for an x64 OS.

---

No theres a switch you can add to system.ini that lets you use 3GB per process on an x86.

That was beside the point. The point was that a process can use a maximum of 2/3 GB in an x86 OS, but in an x64 OS, such a limit does not exist. It may not be unlimited, but it is way more than 2/3 GB.

Quote:

---

Originally Posted by Elysia

That was beside the point. The point was that a process can use a maximum of 2/3 GB in an x86 OS, but in an x64 OS, such a limit does not exist. It may not be unlimited, but it is way more than 2/3 GB.

---

its 64GB, thanks to Microsoft.

64 GB is plenty for today. No doubt it will be increased in the future.

I thought most supercomputers pack A LOT more than 64GB RAM already. But then they don't run Windows anyways.

Even some consumer enthusiast systems already have 16GB RAM (4x4GB, 4GB sticks are widely available now).

64GB will run out in no time (3 years if we assume it doubles every 1.5 year).

But of course, Windows 8 will support 128GB RAM... so if you need that, you'll have to upgrade. Then Windows 9 will support 256GB RAM :).

As for the original question, I don't think malloc till failure will work. It will always give you close to the amount of virtual memory you have.

Quote:

---

Originally Posted by cyberfish

As for the original question, I don't think malloc till failure will work. It will always give you close to the amount of virtual memory you have.

---

While that's true on Linux, I don't think it is true on Windows. I believe that Windows will always check to ensure there is heap space available before returning a valid pointer from malloc().

Ah ok. Stand corrected.

I have never done much Windows programming.

Memory limits in Windows releases

Summary:

If program isn't linked with LargeAddressAware flag, no more than 2GB address space regardless of program bittage or machine configuration.

32-bit program = max of 4 GB of user address space on 64-bit machines, max of 3GB with 4GT on 32-bit.

32-bit machine = up to 4 GB physical memory in workstation editions, no more than 64GB for server versions

64-bit program = max of 7TB (IPF) / 8TB (x64) user address space.

64-bit machine = up to 192 GB physical memory in workstation editions, no more than 2TB for server versions.

Quote:

---

Originally Posted by Elysia

64 GB is plenty for today. No doubt it will be increased in the future.

---

Not really. Video editing a Bluray disk eats 50GB just for the image data, and thats assuming you aren't editing multiple sources at once.

I have neural networks that would easily use all the virtual memory windows would let me have. On the order of 1000 TB.

Nothing says you have to store all that data in memory at the same time. You have to be somewhat conservative. This is a problem that will probably always exist.
Stuff that requires 1000s of TBs of memory is most likely not designed to run on your average computer, thus it is not a really valid comparison to the 64 GB is plenty comparison.

4 GB is the usual standard today. I am looking at 8 GB myself, but anything beyond that is something I have not even considered. I think 64 GB is plenty.

You have to be conservative if you don't have that much memory.

If you do, but an arbitrarily set OS limit won't let you use it, that's another story.