help with implementing timeout in sockets

Hi everyone,

I have made a program that will connect to a server and download the webpage from it using the GET method. I am using this program to constantly monitor updates on some specific webpages, and I will probably be using thread to handle multiple sites.

Anyway, the problem I am having is, for some websites, when I try to recv() the html, it will receive all of it but then at the end just hang for a while (maybe 7-10 seconds).

For example, on the website Netgear.com, I will receive the last line "</body>" and then it will pause there for about 5-6 seconds then exit. Actually this website just hangs forever!

On other hand, if I try something like Linux.com | The source for Linux information, it will work (the last two lines are "</html>\n" and a "0\n") with no hanging at all.

I want to have it such that it never waits for more than 1-2 seconds.

So I tried following:

=============================
insert

Code:

---

struct timeval tv;
        tv.tv_sec = 3;
        tv.tv_usec = 0;
        setsockopt(sd,SOL_SOCKET,SO_RCVTIMEO,&tv,sizeof(tv));

while (recv(sd, &str_buffer, 1, 0))
{
...copy webpage
}

---

==============================

but it didnt work. the same hanging still occured I got that above example of a timeout from the unix network programming book by stevens.

Does anyone know whats wrong with my code? Or any other method besides SELECT()?

I don't want to use select() since i feel it will make the handling of threads more difficult for me. I have searched on the web for ages and its always SELECT(), alarm with signal (not good for threads i hear) or this setsockopt option.

Thanks guys, your help is much appreciated!

With HTTP requests you should use the Content-Length field to loop your read until you have read the number of bytes indicated.

This:

Code:

---

while (recv(sd, &str_buffer, 1, 0))

---

has no reason at all to stop waiting for (infinite amounts of) data.

but what if we are in a situation where we cant get all the content? I have noticed some websites give the last line or two like 10 seconds later. iwouldnt mind just closing it off by that time