HTTP Protocol :: Winsock

Hi.

I posted a thread about adding a feature to a program that searches a websites via Google. I have the basic design and implementaton done. Now I am debugging it.

I have successfully design and implement a chat program using TCP/IP and non-blocking mode via WAsynSelect(). I am using that same design with this website algorithm. I am running into a problem in this early debugging stage.

The program is I am not sure if TCP/IP and HTTP are the same or are similar. With TCP/IP and using WAsynSelect() mode, the non-blocking model relies on the Windows messaging system. With HTTP protocol, will the server, i.e. Google, send messages about the read, write, and/or close status of a socket (the socket connects the user computer to Google server via an Internet).

Thanks,
Kuphryn

It should'nt matter what protocol you're using, the FD_READ message should catch any incoming data whatever protocol is being used. I don't see any reason for it to be any different if you are using the Google API.

Thanks.

I am not using Google API.

Kuphryn

Quote:

---

I am not using Google API.

---

Funny... :p

I don't understand you're problem? Can you explain it better?
Personally, I have no current need for any of the advanced Winsock functions, so my insight may be irelevant to your non-blocking routines. Still, I am curious to know what you mean...

Success! Thanks wrathgame of GameDev for the nice HTTP GET code and the other modifying HTTP handshake codes.

The program I am working on can now do a search on Google and filter out sites. The only problem I face now is determine when to stop sending GET for addition pages. Google has a limit on the number of pages in a search. For example, if you do a search for "car," I believe Google returns about 800 or so pages. Google sends ten hits per GET command, so you send the GET command 80 times. With programming, I am not sure how to determine the last page Google will send back.

Please post if there is some HTTP code that Google sends back indication that it is the last page for a particular searchkey.

Thanks,
Kuphryn

Run the program and ask for more pages than allowed, each iteration printing the recieved page to file. Look in the file and note the message recieved when you max out on requests. Hard-code that message into the proggie :)

Yeah. That is the same technique I am trying. The problem is Google continue sending new pages. I have no way of knowing when the last page comes around so I can stop the GET command and analyze the output data.

I will keep track of all sites from the previous GET command and compare the new data. If the two are the same or there are no hits, then I will stop the GET command.

Kuphryn

Sounds reasonable...hey BTW, when you're done, why not post the EXE here?

Sure. I have some minor tweaks I want to finish. I will release the first beta on my website. I should finish the beta sometime today.

Kuphryn

Okay. I have just compiled the first beta release. Please tell me if there is any bug especially in the keyword you use to search.

http://www22.brinkster.com/rakryn/software.html

Thanks,
Kuphryn