Realloc and Linux

I am running the below code on Debian Linux kernel 2.6.26. I was watching the resource manager in Linux and noticed that this program does not free memory properly until its execution ends. I assumed it would return the memory after the free() and before the sleep() but that is not what I am seeing when I run this code. If you run it with parameters that exhaust your system's memory it even begins to use swap space which it also holds onto until the sleep ends and the program terminates. Does anyone know why Linux doesn't show this memory as being freed?

Code:

---


 int block_size = 262144;
 char* stuff = malloc(block_size);
 int current = block_size;
 int x;

 printf("Growing..... \n");
 for(x=0; x<100; x++)
 {
  stuff = realloc(stuff, current + block_size);
  current += block_size;
  memset(stuff, 'a', current);
 }
 
 printf("Shrinking..... \n");
 for(x=0; x<100; x++)
 {
  stuff = realloc(stuff, current - block_size);
  current -= block_size;
  memset(stuff, 'b', current);
 }
 
 free(stuff);
 
printf("I am sleeping now.  Did the memory free?\n");
sleep(10);

---

This is not just a linux thing.

The basic reason is that malloc(), realloc(), calloc() and free() typically manage memory on behalf of the application, not the operating system. There is nothing requiring realloc() or free() to return memory to the operating system and, in practice, they often simply allow memory previously used by the application to be recycled back to the application. The reason for that is performance: reaching back to the operating system every time memory is allocated or released by the application is relatively expensive compared. Keeping track of things internally - within the implementation of malloc()/calloc()/realloc()/free() - avoids the expense of going back to the operating system every time.

However, a side effect is that memory usage of the application - as observed from the operating system - will not immediately reduce when the application releases memory through a call to free() or realloc(). It also means that memory usage measured by the operating system does not increase immediately with every call of malloc() as well.

These effects are most noticeable with code that regularly allocates, reallocates, and deallocates memory dynamically.

The memory will, however, be reclaimed when the program exits - either because the data structures used internally by malloc/realloc/calloc/free are cleaned up by the library code as the program terminates or, as a fallback with modern operating systems, the operating system itself cleans up as the application terminates.

So what if the process doing this type of allocation doesn't end? a daemon for example? I know that the memory is not immediately given back to the OS, but I was assuming it would be reused by the program as time went on. I definitely didn't think it would ever force the OS into using swap. My program eventually crashes because it does not appear that the memory is being recycled.

The issues here are myriad, but here are the more important ones:

1. The implementation of malloc(), realloc(), and free() is not required to obtain or release system memory in any particular way. Their only purpose is to provide usable memory blocks upon request, then cause those blocks to become invalidated when freed.

2. Even if free() called to the operating system to request a release of memory, there is no guarantee that the system will do anything with that request. On UNIX in particular, small allocations are usually drawn from the dynamic data segment and the underlying memory is allocated with sbrk(). Once allocated by sbrk(), the memory is never actually released by the operating system. This is okay, because a correctly functioning malloc() implementation will reuse memory if it is available rather than asking the OS for more. Also, the OS can swap any dormant pages to disk.

3. The resource usage reported by the OS toolkit is unreliable at best. I wouldn't use those numbers for anything important. It's actually quite hard to define how much memory a process is actually using (though RSS comes close, it doesn't account for things like shared dynamic code which doesn't really "count" against a process since it's mapped in multiple processes simultaneously).

If your program is doing reasonable things with memory, I am almost completely certain that any crashes you are experiencing are due to bugs in your own code.

The above example is my code. If I loop it to execute several times the system runs out of memory. I cannot figure out why this is happening if everything is freed with each iteration. I even ran valgrind against it and it found nothing.

Quote:

---

Originally Posted by fguy817817

So what if the process doing this type of allocation doesn't end? a daemon for example? I know that the memory is not immediately given back to the OS, but I was assuming it would be reused by the program as time went on. I definitely didn't think it would ever force the OS into using swap.

---

A deamon needs to be smart in how it allocates memory.

Any program (daemon or otherwise) that allocates small chunks of memory and progressively allocates bigger chunks will eventually cause memory fragmentation - old released small chunks cannot always be reused when the program requests a large chunk.

One strategy is that, every time memory is allocated, the buffer sizes are constant. That way, when one function in the buffer calls free(), that buffer can definitely be recycled back through a subsequent call to malloc().

Daemons, generally, do memory allocation very rarely (eg only in program startup, after which the buffers are treated as "fixed"). They very rarely rely on any form of recycling of memory, unless all allocated and deallocated chunks are the same size.

Either way, it is strongly inadvisable to use any form of "growing buffer" in code that runs continuously. That is true whether you use malloc()/free() or some other low level mechanism supported by the operating system. The principle is the same - the only thing that changes is where the fragmentation is experienced (in the application or by the operating system) as memory resources are always finite.

Quote:

---

Originally Posted by fguy817817

My program eventually crashes because it does not appear that the memory is being recycled.

---

While a lack of "recycling" can eventually cause a program crash, my bet is that your program is crashing because of some other problem in your code. For example, you have some code using the address returned by realloc() without testing it first - realloc() returns NULL if an error occurs.

Your program being forced to use swap will simply make things slow down (which is inconvenient, but does not usually cause a program crash). It can increase the likelihood of a call to realloc() failing when increasing a buffer size but, realistically, a daemon should avoid doing that in the first place.

I guess my real question is this: If the above code is iterated several times without the program terminating, should it ever crash? This is assuming your system has enough memory to run one iteration before the free() occurs. So if I wrapped the whole thing in

Code:

---

for(a=0; a<100; a++){}

---

with the above conditions should your program ever crash?

Should crash? No. Can crash? Possibly. Will definitely crash? No.

Your code is repeatedly doing

Code:

---

  stuff = realloc(stuff, current + block_size);
  current += block_size;
  memset(stuff, 'a', current);

---

If the realloc() call fails for any reason, it will return NULL. You are not testing that. However, if realloc() fails, the call of memset() gives undefined (as that term is defined in the C standard) behaviour. Anything can happen at that point, including a program crash.

Just because your code runs correctly once, does not guarantee it will run correctly 100 times. In practice, it often will. Unless you implement an appropriate guarantee it may crash occasionally.

Put it this way: imagine that you are tied in a chair with a loaded rifle pointed at your head. You are unable to move your head in any way and any attempt to touch the rifle causes it to fire. You do not wish to die. A guaranteed consequence, if your program crashes, is that the rifle will fire. You know that malloc(), realloc() and free() are implemented in a such a way that memory is not consistently recycled, so the realloc() call can sometimes fail. The computer on which your code will be run has been configured by someone who wants you to die, and is aware of what your code does. Would you execute your code?

I am questioning this example because I am going to try to write some methods that will manage files in memory i.e. mseek, mtell, mwrite.. etc. The prob is that the memory for this type of approach has to be designed to grow to hold files as they grow similar to how they can grow on disk. I can't think of a way of doing this without constantly realoccing the memory which according to what we have just discussed will be prone to failure. Even if I initialize each file to a 256k chunk there is a good chance it will need to grow and shrink. What would be the proper methodolgy for doing this? It will occur in a process that is constantly running and can run for an indeterminate amount of time.

You're trying to tell me you want to be able to allocate an arbitrarily large amount of memory and be guaranteed never to encounter a failure???? Pull my other leg: it plays jingle bells.

Let's be clear: the concepts of "program that never fails to allocate memory" and "program with memory usage that grows without any upper bound" are mutually exclusive, unless you assume a computer with infinite memory resources. Such a computer is physically impossible.

The "proper methodology" would start with defining a requirement that can actually be met on a real-world computer.

Rather than trying to work out how to grow your allocated memory to hold an arbitrarily large file, try working out some strategy for managing a large file within a memory pool that rarely (preferably never) grows beyond some defined limit. That will mean only storing parts of the file in memory that are actually needed, flushing data from memory to larger storage device (eg hard drive) if it is no longer needed or reading data from a storage device in a timely manner when it is needed. As to how you work out what parts of a file need to be in memory .... that will depend on what you want to do with the file.

> What would be the proper methodolgy for doing this?
Write a file system.

Consider that your disk has a finite amount of space as well, and that sooner or later, you will get a file system full message.

Allocate whatever large block of memory you like to be your 'disk', which you then 'format', and then you create 'directories' and 'files'

I understand everything you have told me. I have had this discussion with numerous other people. To answer your question, no I am not trying to "allocate an arbitrarily large amount of memory and be guaranteed never to encounter a failure". So I'll pass on pulling your other leg.

I understand there is no guaruntee that memory allocation will happen properly 100% of the time. The code I displayed ALWAYS fails if iterated because the memory is not reused properly. That is my problem.

I understand that computers have finite resources available. As far as disk IO is concerned, it will not be possible. The speed requirement I am going to have to meet will exceed what can be swapped to disk within an acceptable amount of time. This code will eventually analyze all files transmitted over a network connection in realtime (well.. as close as possible). It will have to handle a throughput of over 1gbps of data that will have to be managed in memory. Unfortunately, each these files will also need checksums generated so it will be essential to capture each bit. I have implemented all of the above but the growing and shrinking file is a problem that I did not consider. This is the first time I have ever worked with this much memory allocation. My current code can burn through 64GB of memory in less than half an hour with the test data I am using.

This is why I wrote the sample piece of code that demonstrates the same problem I am having, but in a much more isolated scenario. I was fully aware (before I started this project) that the OS would "hold onto" memory blocks that had already been allocated by the program and that Linux memory usage is inaccurate at best, but I was not aware that it would consistantly fail once memory usage peaked. It was my understanding that at some point it would release that memory back (even if there is performance degredation) rather than hold onto it until it died.

Quote:

---

Originally Posted by fguy817817

Does anyone know why Linux doesn't show this memory as being freed?

---

I don't know which value of "free memory" reads your resource manager but if it's exactly MemFree in /proc/meminfo then in 2.6.26 kernel you will not see desired value.
Other memory will be freed too but it will grow Cached value in same file.

If you want to move Cached value to MemFree you must do three things:
1) Install at least 2.6.35 kernel
2) disable overcommit in your memory manager. See this for details.
3) run malloc(so_facking_big_value_i_ever_see). Pointer returns NULL, of course, but system at least will do that movement because it firstly will revise how much memory really free to satisfy your requirement.

Quote:

---

Originally Posted by grumpy

If the realloc() call fails for any reason, it will return NULL. You are not testing that. However, if realloc() fails, the call of memset() gives undefined (as that term is defined in the C standard) behaviour. Anything can happen at that point, including a program crash.

---

Of course, code is incorrect in this way. But anyway why it crash fo very first time... It's question. I can only guess that with memory on machine was smth wrong before.
fguy817817, have you run memtest?

Aha, maybe one more reason. Your memory could be well defragmented to not even satisfy such realloc for very first time.

Well I think I found the problem. I had been running these test cases in Valgrind. Apparently valgrind holds onto a tremendous amount of memory until the program terminates... I ran this outside of Valgrind and the memory appears to come back down within the execution of the program as it iterates. I guess this is the point where I admit that I goofed. I will run some additional tests to verify.

The program was not failing on the first realloc. It was tying up memory as the program repeated itself and eventually exhausting all memory. This is when it was crashing.

Valgrind has to hold onto the memory so that it (and you) can spot any "use-after-free" memory problems you may have.

Aside from Valgrind I am still leaking memory due to constant reallocation. I guess this is just not a valid approach for a large amount of memory reallocation. I think my strategy will be to just malloc a very large cache of 256k blocks at the start of the daemon and then link them together as needed for files rather than realloc the individual blocks. I would imagine this would solve the problem.