Complicated Memory Management

In my HTTP Server I have to read in the HTTP request (pretty much just like any other string) using the recv() function, which requires me to specify the maximum size of the buffer. I am however, for obvious reasons, wary of setting a size limit. That, is the essence of my problem.

I thought about it most of yesterday, and realized that I don't have to read in the entire request in one call to recv() - I can specify a small buffer, read in the data, process that, then read in some more, etc... That gives me a bit more freedom, but at some point along the road I'm going to have to store the whole message, regardless of size.

Can anyone think of a solution? Open-ended arrays would be perfect. Although I do have to specify the buffer size, I can just read in the message in parts. Unfortunately, I don't think you have those in C. So any ideas would be greatly appreciated.

A linked list of fixed-length buffers to read in the data.
Once you know the final size, you can easily sum the size of all the buffers on the list and allocate a single buffer for the whole thing.

Thanks Salem! It took me a while to figure out exactly how to work the whole system, as I've never done a linked list before, but I'm pretty conifident with what I have now.

Hate to say this on the C board but vectors might be easier.

What happens if a couple of users decide to upload 100MB files at the same time?

>> Hate to say this on the C board but vectors might be easier. <<

As you know, a vector is also simple to implement in C.

Quote:

---

What happens if a couple of users decide to upload 100MB files at the same time?

---

100000 loop iterations each...

Quote:

---

As you know, a vector is also simple to implement in C.

---

What would be the advantage of a vector over a linked list? Bearing in mind I've already coded most of the linked-list, so ease of implementation is not a big concern.

>> >> What happens if a couple of users decide to upload 100MB files at the same time?
>> 100000 loop iterations each...

But you'll run out of memory.

>> What would be the advantage of a vector over a linked list? Bearing in mind I've already coded most of the linked-list, so ease of implementation is not a big concern. <<

Ideally, you halve the memory required and remove the need for a memory copy. Realistically, that depends on the implementation of realloc(). A poor implementation of realloc() or a fragmented address space may result in several memory copies.

Why do you need the entire HTTP request in memory? Ideally, you would only put the HTTP headers in memory and the following POST or PUT data would be written directly to the CGI program.

Quote:

---

But you'll run out of memory

---

Oh ye of little faith. I've already done the error handling. If malloc can't allocate space for the next packet, the details are logged on the server, an Internal Server Error is dispatched to the clients (500), and the memory already on the heap is free()'ed.

Quote:

---

Why do you need the entire HTTP request in memory? Ideally, you would only put the HTTP headers in memory and the following POST or PUT data would be written directly to the CGI program.

---

At some point it will all have to be in memory (the whole rquest will be written to the log too), and for simplicity, all at the same time. The current set up meets my demands, but I'm still open for suggestions to make it better.

Couldn't you read the Content-Length field of the HTTP request, and then use that value to allocate the memory?

Quote:

---

14.13 Content-Length
The Content-Length entity-header field indicates the size of the entity-body, in decimal number of OCTETs, sent to the recipient or, in the case of the HEAD method, the size of the entity-body that would have been sent had the request been a GET.

Content-Length = "Content-Length" ":" 1*DIGIT
An example is

Content-Length: 3495
Applications SHOULD use this field to indicate the transfer-length of the message-body, unless this is prohibited by the rules in section 4.4.

Any Content-Length greater than or equal to zero is a valid value. Section 4.4 describes how to determine the length of a message-body if a Content-Length is not given.

Note that the meaning of this field is significantly different from the corresponding definition in MIME, where it is an optional field used within the "message/external-body" content-type. In HTTP, it SHOULD be sent whenever the message's length can be determined prior to being transferred, unless this is prohibited by the rules in section 4.4.

---