Internet Socket Returning Extra data

Hi,

I have a basic spider that goes and downloads a webpage. It works fine except it returns the complete webpage and then an extra bit.

I enclosed recv inside of a while statement:

Code:

---

while(numbytes = recv(sockfd, buf, MAXDATASIZE - 1, 0))
      {
      printf(buf);
      }

---

Is this the right way to do this?

I believe you should be checking for SOCKET_ERROR if you're on Windows and -1 if you're on *nix to signify errors. Zero is an indication of a closed connection, so that part you have correct.

http://www.opengroup.org/onlinepubs/...ions/recv.html

http://msdn.microsoft.com/en-us/libr...21(VS.85).aspx

Shouldn't the while loop look like this:

Code:

---

while((numbytes = recv(sockfd, buf, MAXDATASIZE - 1, 0)))
      {
      printf(buf);
      }

---

It returns the page fine so that all seems to work. Been reading all weekend on Internet Sockets and I think the problem is with the charset of the web page I am accessing. Do I need to convert the charset of the page I am down loading?

It is unix I am developing on if that helps.

Wait, is it still printing extra stuff? If it is and you havn't tried my solution, then you should try it because i think it might be the problem. It seems to go through the loop one too many times.

39ster:

It works using your loop also but having same problem. Perhaps it's in issue with the way printf is printing out to my console?

it prints out the page no problem and them after the closing html it echos out another two lines... these lines are not on the original webpage.

You havn't null terminated the string.

Code:

---

while((numbytes = recv(sockfd, buf, MAXDATASIZE - 1, 0)))
{
      buf[numbytes] = 0;
      printf(buf);
}

---

Still same problem.

you are right. The last loop prints about 20 - 50 characters of the previous loop then it exits and this is the extra data. I am thinking maybe I have to detect an EOF

You have to look at the HTTP-Header, if Content-Length size or Transfer-Encoding:chunked is present and handle these correctly.

I guess you are downloading a site that uses chunked transfer (e.g. google).

yes, encoding is chunked. Least now I can start work in the right direction.

http://tools.ietf.org/html/rfc2068#section-3.6

It's a bit tricky...

Looks like I will learn a lot more about http which prob isnt a bad thing.

Code:

---

while((numbytes = recv(sockfd, buf, MAXDATASIZE - 1, 0)) > 0)
{
      buf[numbytes] = 0;
      printf(buf);
}

---

Only 0 is false. Anything else, including negatives would pass

Code:

---

printf(buf);

---

to

Code:

---

printf("%s", buf);

---

or

Code:

---

fputs(buf, stdout);

---

See this.