Thread: http get requests etc

I want to write a program that reads webpages and so on, I want to be able to enter a URL and have some code that goes out and gets the page, so like if I entered http://www.google.com/ I'd get basically what you would get by going to http://www.google.com/ and clicking view source.
I've seen it done in Python relatively simply, just wondering how I'd do it in C++?

Oh okay.
I thought it might be something like that. Just hoping there might have a been a simple way.
i guess I'll go refresh my knowledge of python and use that instead.
The win32 API? I don't see what http has to do with windows, I mean, you can use the internet from *nix and macs too...

Thanks salem.
I've just been off learning WinSock stuff, and now I know how to send a recieve data through a socket, all I need to know now is how to structure a GET request and send it through a socket.
But I'll have a look at that link now too...

I want to do it from within my own program, so the command line cURL isn't any use.
And the libcurl doens't seem to have much documentation, see http://curl.haxx.se/libcurl/cplusplus/
I think I'll stick with the WinSock and HTTP GET request, I've almost got that worked out now...

I need help with this too. I an currently trying to get DEV back up and working, so i haven't finished.

Here is what i have gleaned from the RFCs:
GET REQUESTS:
GET *path* HTTP/*version*
ex.:
GET / HTTP/1.0

But, i also believe that you have to include both \n and \r at the end of the line to make it work. This is after you have connected the socket to port 80 on the remote machine.

Let me know if you find out how to do this!

~Inquirer

Yeah, I'm up to about where you are Inq. Problem is now, at Uni we have a proxy or some such so I can't work on it becuase my socket cannot connect to anything. Gonna have to wait 'till I get home.
Maybe I'll just go home now. There's only a formal logic tutorial and Lecture left today, 4 hours from now. Yeah, I think I will just go home now.

I found a nice little program that does almost what I want, it has source code included and also when you run the program it shows you what it sends and recieves.
Have a look here: http://www.programmersheaven.com/art...t/article.html

face, there isn't winsock on mac. It is opentrasport.

~Inquirer

spirit, that code tere is WAY out of my league... THanks for the help anyway! The program itself has helped.

~Inquirer

Yeah, I looked at the source only after I'd posted the link.
Only the program itself is of much use to me as well.
I think I've got sending going, but I keep getting blank replies...

** opens eyes wide **

Umm.. Me too. I think, however, that i have got it figured out. The thing i had to do is send the following:

Code:

GET / HTTP/1.0
Accept: text/html, text/plain, */*\r\n
Accept-Language: en-us\r\n
Accept-Encoding: gzip, deflate\r\n
Host: [SERVER_NAME_HERE]\r\n
\r\n

An example of the Host: line would be "Host: google.com\r\n"

The "\r\n" apparently is what servers look for that i was missing. I also forgot to send the final "blank" line to terminate the request. I found this by having IE query one of my servers, and outputting all the text, and the nonprintables as their escaped strings. Hope this helps! (Hope this works... )

My only problem, is that i kep getting the contents of other cariables, so i think i have an array problem somewhere.

~Inquirer

My problem is for my outgoing socket, recv() always returns -1. It works fine for incoming sockets, bit not for outgoing. HEre is basically what my program is doing. Would someone tell me if i am missing something?

1) Get the server from the user. Ex.: www.google.com
2) Get the IP address of the server. Ex.: 216.239.53.101
3) Get the port number from the user. Ex. 80
4) Make the connection.
5) Use recv() to try to get 1024 bytes of daya from the server at a time. // ERRORS HERE, so show the returned value and perror("recv")
6) Output the read data
7) Quit

Help?

~Inquirer

What does perror("recv") tell you?

Can you post your code, or at least some of it?