Thread: How robust is malloc() in practise?

Hi, everyone. I have a Wikipedia bot that has the capability to download a number of items from Wikipedia - the HTML version of a page, the wikitext version, XML information about the page's author and timestamp, a past version of the page, and so on.

What I have been doing up till now is estimating an amount of RAM which can hold even the largest page (4 megs) and allocating 6 such buffers, then passing the buffer addresses to various routines as part of a handle. However, as the number of functions in my bot increases, the number of potential buffers increases. At the moment I'm allocating 24 megs, most of which will never be used. My original idea was that if I only do one malloc() at the start of the program, it will decrease the amount of paper-shuffling needed to keep track of the different buffers, and reduce the possibility of memory fragmentation.

However, I'd like to at least have the option of dynamic allocation. When downloading a page of any type, the function would malloc(1) (0 doesn't work on some OS's), and then every time stuff is added to it, realloc() and memcpy() until done. Then the function must free() the buffer.

My bot will potentially go through this routine 10 times to do the simplest Wikipedia task. Yet, I feel as though this is the 'right' way to do it. Otherwise I need to keep on adding 4-meg buffers to my handle for each possible function.

Realistically, am I creating any headaches by using malloc()/realloc()/memcpy()/free() rather than working with fixed buffers? Could it ever lead to unrecoverable run-time memory fragmentation or memory leaks? How robust is the malloc() mechanism in practise? Note that my code is very portable and designed to work with everything from the Amiga to Windows to Slackware to VxWorks.

Originally Posted by Salem

>
But the amount of fragmentation is down to each specific implementation. If you want a totally robust embedded system say, then don't use malloc.

I don't like the sound of that. I would like the bot to run on embedded/small processors. Having said that, in order for the bot to do anything useful, it needs to be able to cope with at least several megabytes of text data and HTTP transactions. So running it on a microwave oven is not realistic.

Richard

Originally Posted by Salem

Code:

struct pageFragment {
    struct pageFragment *next;
    size_t usedSize;
    char fragment[10240];
};
struct page {
  struct pageFragment *head;
  struct pageFragment *tail;
  size_t totalSize;
};

Sure there are more allocations (and more frees later on), but the pool manager is going to have a lot more chance of joining adjacent blocks together into larger free blocks to satisfy future requests.

What you never end up doing is trying to realloc xMB into yMB and finding that it doesn't fit in any free block.

Salem,

I see what you're proposing, but I worry that it's not in the spirit of C. Surely C allows me to store a string in a flat memory space and operate on it directly using <string.h>. If I use your suggestion, then I'll have to wrap all my string-handling functions.

I figure that any modern operating system that can handle a TCP/IP stack is going to have virtual memory plus other trickery going on. I know that Linux doesn't actually truly allocate the memory until it's actually used, for example. If you malloc() and then free() memory without using it at all, it does the paperwork but doesn't carve off the actual RAM.

When I was a little boy I had an Amiga, and any time memory was allocated using the OS, it would permanently fragment the memory map until the next reboot. So avoidable allocation/deallocation was a very bad thing, especially when you start off with 512 k. I hope that all modern OS's, including AmigaOS, have moved on from those days.

Richard

Richard... why do you not centralize all incoming data. One function that does all your "getting"... give it a big buffer; malloc 10mb if needed. Now when something comes in it goes into 1 (one) place and guess what, you know the size! So when your getter returns or signals it can say "327677 bytes ready" and you can then use malloc to create a memory block the right size and memcpy or memmove to scoop up the data. No more guesswork.

In my experience centralizing stuff like this ultimately leads to simpler, more robust code.

Tater,

The HTTP requests return data in pieces, each up to 4 kbytes long. A write callback function is used to put it together. The write callback can do anything it wants - serialize it to disk, print it to screen, checksum it, whatever.

I guess you could have a persistent buffer for the write callback and put the returned data together sequentially, then malloc(strlen()+1) && strcpy(). But it's just as easy to put the received data directly into the new buffer. I figure that you would code it so that the initial malloc() allocates some fixed value, and when your buffer runs out of memory, you'll allocate up to the next multiple of the fixed value, where the fixed value is something like 4 kbytes. That way you'll realloc() every 4 kbytes but you won't have to realloc() every 20 bytes if your HTTP data comes in 20 byte chunks.

Richard

You can use either scheme... but I would favor a getter function that returns the required buffer size... The caller is responsible for creating it's own memory space, fetching the data, doing whatever, then disposing of it when done. The reason I favor that is that you eliminate a lot of scoping problems with your temporary buffer pointers; the data is housed in the function operating on it. In a single tasking environment, you could even make the getter's buffer global (yes, sometimes it is appropriate) and operate directly on that one buffer, elminating all other buffers. In a multitasking environment you would use interprocess signalling to stop the getter until you copy the data out...

You can use the realloc trick inside the getter but with 4 and 8 gb machines being popular these days I would hardly think a fixed buffer size of 10mb is much of an issue. (it's not like you're trying to shoe-horn this into some dodgy old 128mb antique, is it?)