Complicated Memory Management

This is a discussion on Complicated Memory Management within the C Programming forums, part of the General Programming Boards category; In my HTTP Server I have to read in the HTTP request (pretty much just like any other string) using ...

  1. #1
    Super Moderator
    Join Date
    Sep 2001
    Posts
    4,913

    Complicated Memory Management

    In my HTTP Server I have to read in the HTTP request (pretty much just like any other string) using the recv() function, which requires me to specify the maximum size of the buffer. I am however, for obvious reasons, wary of setting a size limit. That, is the essence of my problem.

    I thought about it most of yesterday, and realized that I don't have to read in the entire request in one call to recv() - I can specify a small buffer, read in the data, process that, then read in some more, etc... That gives me a bit more freedom, but at some point along the road I'm going to have to store the whole message, regardless of size.

    Can anyone think of a solution? Open-ended arrays would be perfect. Although I do have to specify the buffer size, I can just read in the message in parts. Unfortunately, I don't think you have those in C. So any ideas would be greatly appreciated.

  2. #2
    and the hat of wrongness Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    32,590
    A linked list of fixed-length buffers to read in the data.
    Once you know the final size, you can easily sum the size of all the buffers on the list and allocate a single buffer for the whole thing.

  3. #3
    Super Moderator
    Join Date
    Sep 2001
    Posts
    4,913
    Thanks Salem! It took me a while to figure out exactly how to work the whole system, as I've never done a linked list before, but I'm pretty conifident with what I have now.

  4. #4
    Super Moderator VirtualAce's Avatar
    Join Date
    Aug 2001
    Posts
    9,596
    Hate to say this on the C board but vectors might be easier.

  5. #5
    Yes, my avatar is stolen anonytmouse's Avatar
    Join Date
    Dec 2002
    Posts
    2,544
    What happens if a couple of users decide to upload 100MB files at the same time?

    >> Hate to say this on the C board but vectors might be easier. <<

    As you know, a vector is also simple to implement in C.

  6. #6
    Super Moderator
    Join Date
    Sep 2001
    Posts
    4,913
    What happens if a couple of users decide to upload 100MB files at the same time?
    100000 loop iterations each...

    As you know, a vector is also simple to implement in C.
    What would be the advantage of a vector over a linked list? Bearing in mind I've already coded most of the linked-list, so ease of implementation is not a big concern.

  7. #7
    Yes, my avatar is stolen anonytmouse's Avatar
    Join Date
    Dec 2002
    Posts
    2,544
    >> >> What happens if a couple of users decide to upload 100MB files at the same time?
    >> 100000 loop iterations each...

    But you'll run out of memory.

    >> What would be the advantage of a vector over a linked list? Bearing in mind I've already coded most of the linked-list, so ease of implementation is not a big concern. <<

    Ideally, you halve the memory required and remove the need for a memory copy. Realistically, that depends on the implementation of realloc(). A poor implementation of realloc() or a fragmented address space may result in several memory copies.

    Why do you need the entire HTTP request in memory? Ideally, you would only put the HTTP headers in memory and the following POST or PUT data would be written directly to the CGI program.

  8. #8
    Super Moderator
    Join Date
    Sep 2001
    Posts
    4,913
    But you'll run out of memory
    Oh ye of little faith. I've already done the error handling. If malloc can't allocate space for the next packet, the details are logged on the server, an Internal Server Error is dispatched to the clients (500), and the memory already on the heap is free()'ed.

    Why do you need the entire HTTP request in memory? Ideally, you would only put the HTTP headers in memory and the following POST or PUT data would be written directly to the CGI program.
    At some point it will all have to be in memory (the whole rquest will be written to the log too), and for simplicity, all at the same time. The current set up meets my demands, but I'm still open for suggestions to make it better.
    Last edited by sean; 11-25-2004 at 07:34 PM.

  9. #9
    Registered User
    Join Date
    Sep 2004
    Location
    California
    Posts
    3,246
    Couldn't you read the Content-Length field of the HTTP request, and then use that value to allocate the memory?

    14.13 Content-Length
    The Content-Length entity-header field indicates the size of the entity-body, in decimal number of OCTETs, sent to the recipient or, in the case of the HEAD method, the size of the entity-body that would have been sent had the request been a GET.

    Content-Length = "Content-Length" ":" 1*DIGIT
    An example is

    Content-Length: 3495
    Applications SHOULD use this field to indicate the transfer-length of the message-body, unless this is prohibited by the rules in section 4.4.

    Any Content-Length greater than or equal to zero is a valid value. Section 4.4 describes how to determine the length of a message-body if a Content-Length is not given.

    Note that the meaning of this field is significantly different from the corresponding definition in MIME, where it is an optional field used within the "message/external-body" content-type. In HTTP, it SHOULD be sent whenever the message's length can be determined prior to being transferred, unless this is prohibited by the rules in section 4.4.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. tools for finding memory leaks
    By stanlvw in forum C++ Programming
    Replies: 4
    Last Post: 04-03-2009, 11:41 AM
  2. Valgrind says I suck at memory management :)
    By carrotcake1029 in forum C Programming
    Replies: 6
    Last Post: 02-01-2009, 07:10 PM
  3. Memory Management
    By black_spot1984 in forum C++ Programming
    Replies: 9
    Last Post: 10-08-2008, 11:25 AM
  4. Accessing Video Memory Information...need help
    By KneeLess in forum C++ Programming
    Replies: 8
    Last Post: 08-24-2003, 03:53 PM
  5. Replies: 4
    Last Post: 10-11-2002, 06:58 AM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21