Search Engines :: Winsock

Hi.

I will begin work on a feature in a program to do a search on search engines such as Google and Yahoo. For example, the user will type in whatever phrase he/she wants such as "race & cars," and the program will do a search via Google and print the responses.

I am not sure what message I need to send and where to send it if I want to search from, for example, Google. Do I send this:

// "http://www.google.com/search?hl=en&lr=&ie=UTF8&oe=UTF8&q=race+%26+cars//

I pasted the code above directly from the URL when I did a search at Google.

Secondly, what data will the search engine return? For example, the search above came back with more than ten pages at Google. Will Google send back all pages or one page at a time?

Please add if you have any experience dealing with search engines and/or communicating with websites in general.

Thanks,
Kuphryn

Using Telnet (in Windows) connect to google.com like this:

telnet google.com 80

Do that from the console prompt. Then you would use the HTTP command "GET" which retrieves stuff from the server. Like so:

GET /search?hl=en&lr=&ie=UTF8&oe=UTF8&q=race+%26+cars

This will get the the first page of the results. To get the next lot of results, you'd have to GET the data from page 2 of the results like so:

GET /search?q=race+%26+cars&hl=en&lr=&ie=UTF8&oe=UTF8&s tart=10&sa=N

(and so on...)

The search only returns one page at a time so you can't receive all the data at once. You'd probably have to issue multiple GET commands. You should read up on the HTTP protocol.

Okay. Thanks.

Kuphryn

I found out Google offers Google API that will help with the search and make the programming process much, much easier and quicker. The only drawback is the fact that under the license agreement, you can only make 1,000 query. I think I have end up having to use the Google API because of the simplicity.

I really want full control over the search via programming skill. In other words, I would like to do any searches via programming Win32 API instead of going through Google.

Kuphryn