Thread: Read HTTP headers

Hello,

I'm writing a small app which uses some of the HTTP protocol. I've read the HTTP 1.1 RFC, looked at examples, and it doesn't mention anything about a max header size (eg how large the HTTP headers can be in bytes) - and a lot of the examples allocate different amounts of size for the headers (from 512b to 2k), So I was wondering, what is the best way to read say HTTP headers from a client? Read until you encounter CRLFCRLF? If so how should you read from the socket to efficiently achieve that?

Reading char-by-char from a socket seems rather "inefficient".

Thanks

> Reading char-by-char from a socket seems rather "inefficient".
Perhaps, but then again trying something else might be premature optimisation disease.

I suppose you could create a wrapper class which does block recv() calls and splits the data into consecutive CRLF lines, and maintains any residual data at the end (which would be the first part of the content) in an internal (to the class) buffer.

Also remember that the network driver already does buffering, so it's not like it's receiving the data byte by byte.

I see, thanks everyone - very helpful. The fact is I'm only really interested in the header, one part even "Host: "

I've decided to read it into a 2K buffer on the stack, find CR-LF-CR-LF and lop the end off. Is that wise?

Or I was thinking of something a little more complex,

Code:

int r = 0;
char buf[128];
char * header = NULL;

while((r = recv(sock, buf, sizeof(buf), 0)) == sizeof(buf))
{
    /* add buf to header (realloc and strcat) */
    /* search through 'header' (from the last buf addon) if we find \r\n\r\n stop */
}

Good or crappy way?

Seems more trouble than necessary. Read in a buffer of any size, then parse it for headers. If you reach the end of the buffer but not the end of the headers, copy the unprocessed parts to the beginning of the buffer and fill the rest of it with new data. Continue until you reach the end of the headers.

Ahhh, thanks

The only problem with this approach is when a single header is larger than the buffer. If you're not interested, you can just discard it, otherwise you must collect (and thus introduce state into your loop).

I don't really see how that is more trouble than my suggestion, is it faster or less resource intensive?

Less resource intensive. You need less memory because you're not collecting all headers at once.

Hmm, thanks for that.

Why didn't they implement some sort of payload into the HTTP protocol? They do for content-length yet not for headers? Bah

And if it can't, there's the chunked transfer encoding.

Headers are even more complicated. Proxies can insert, change and remove headers. Not that that matters that much - if it's already parsing, it might as well insert the new length.

But I think, in essence, it wasn't considered necessary. It's not that complicated. If you use Boost.Asio, for example, there's the read_until call that implements my pattern pretty much exactly.