http get requests etc

This is a discussion on http get requests etc within the C++ Programming forums, part of the General Programming Boards category; I want to write a program that reads webpages and so on, I want to be able to enter a ...

  1. #1
    Registered User SPiRiToFCaT's Avatar
    Join Date
    May 2002
    Posts
    35

    Question http get requests etc

    I want to write a program that reads webpages and so on, I want to be able to enter a URL and have some code that goes out and gets the page, so like if I entered http://www.google.com/ I'd get basically what you would get by going to http://www.google.com/ and clicking view source.
    I've seen it done in Python relatively simply, just wondering how I'd do it in C++?
    - Well that's my 4c Australian.

  2. #2
    Refugee face_master's Avatar
    Join Date
    Aug 2001
    Posts
    2,052
    just wondering how I'd do it in C++?
    With in depth knowledge of C++, and the Win32bitAPI and Wisock 1.1 / 2.0 APIs.
    Please direct all complaints regarding this post to the nearest brick wall Have a nice day.

  3. #3
    Registered User SPiRiToFCaT's Avatar
    Join Date
    May 2002
    Posts
    35
    Oh okay.
    I thought it might be something like that. Just hoping there might have a been a simple way.
    i guess I'll go refresh my knowledge of python and use that instead.
    The win32 API? I don't see what http has to do with windows, I mean, you can use the internet from *nix and macs too...
    - Well that's my 4c Australian.

  4. #4
    Registered User SPiRiToFCaT's Avatar
    Join Date
    May 2002
    Posts
    35
    Thanks salem.
    I've just been off learning WinSock stuff, and now I know how to send a recieve data through a socket, all I need to know now is how to structure a GET request and send it through a socket.
    But I'll have a look at that link now too...
    - Well that's my 4c Australian.

  5. #5
    Registered User SPiRiToFCaT's Avatar
    Join Date
    May 2002
    Posts
    35
    I want to do it from within my own program, so the command line cURL isn't any use.
    And the libcurl doens't seem to have much documentation, see http://curl.haxx.se/libcurl/cplusplus/
    I think I'll stick with the WinSock and HTTP GET request, I've almost got that worked out now...
    - Well that's my 4c Australian.

  6. #6
    Rebooted Inquirer's Avatar
    Join Date
    Apr 2002
    Posts
    281

    HTTP get requests

    I need help with this too. I an currently trying to get DEV back up and working, so i haven't finished.

    Here is what i have gleaned from the RFCs:
    GET REQUESTS:
    GET *path* HTTP/*version*
    ex.:
    GET / HTTP/1.0

    But, i also believe that you have to include both \n and \r at the end of the line to make it work. This is after you have connected the socket to port 80 on the remote machine.

    Let me know if you find out how to do this!

    ~Inquirer
    Compilers:
    GCC on Red Hat 8.1 (Primary)
    GCC on Mac OS X 10.2.4 (Secondary)

    Others:
    MinGW on XP

  7. #7
    Registered User SPiRiToFCaT's Avatar
    Join Date
    May 2002
    Posts
    35
    Yeah, I'm up to about where you are Inq. Problem is now, at Uni we have a proxy or some such so I can't work on it becuase my socket cannot connect to anything. Gonna have to wait 'till I get home.
    Maybe I'll just go home now. There's only a formal logic tutorial and Lecture left today, 4 hours from now. Yeah, I think I will just go home now.
    - Well that's my 4c Australian.

  8. #8
    Registered User SPiRiToFCaT's Avatar
    Join Date
    May 2002
    Posts
    35
    I found a nice little program that does almost what I want, it has source code included and also when you run the program it shows you what it sends and recieves.
    Have a look here: http://www.programmersheaven.com/art...t/article.html
    - Well that's my 4c Australian.

  9. #9
    Refugee face_master's Avatar
    Join Date
    Aug 2001
    Posts
    2,052
    >> The win32 API? I don't see what http has to do with windows, I mean, you can use the internet from *nix and macs too... <<

    Ok, i'll change my answer:

    With in depth knowledge of C++, and a GUI api for whatever OS you're writing for and Wisock 1.1 / 2.0 APIs.
    Please direct all complaints regarding this post to the nearest brick wall Have a nice day.

  10. #10
    Rebooted Inquirer's Avatar
    Join Date
    Apr 2002
    Posts
    281
    face, there isn't winsock on mac. It is opentrasport.

    ~Inquirer
    Compilers:
    GCC on Red Hat 8.1 (Primary)
    GCC on Mac OS X 10.2.4 (Secondary)

    Others:
    MinGW on XP

  11. #11
    Rebooted Inquirer's Avatar
    Join Date
    Apr 2002
    Posts
    281
    spirit, that code tere is WAY out of my league... THanks for the help anyway! The program itself has helped.

    ~Inquirer
    Compilers:
    GCC on Red Hat 8.1 (Primary)
    GCC on Mac OS X 10.2.4 (Secondary)

    Others:
    MinGW on XP

  12. #12
    Registered User SPiRiToFCaT's Avatar
    Join Date
    May 2002
    Posts
    35
    Yeah, I looked at the source only after I'd posted the link.
    Only the program itself is of much use to me as well.
    I think I've got sending going, but I keep getting blank replies...
    - Well that's my 4c Australian.

  13. #13
    Rebooted Inquirer's Avatar
    Join Date
    Apr 2002
    Posts
    281
    ** opens eyes wide **

    Umm.. Me too. I think, however, that i have got it figured out. The thing i had to do is send the following:
    Code:
    GET / HTTP/1.0
    Accept: text/html, text/plain, */*\r\n
    Accept-Language: en-us\r\n
    Accept-Encoding: gzip, deflate\r\n
    Host: [SERVER_NAME_HERE]\r\n
    \r\n
    An example of the Host: line would be "Host: google.com\r\n"

    The "\r\n" apparently is what servers look for that i was missing. I also forgot to send the final "blank" line to terminate the request. I found this by having IE query one of my servers, and outputting all the text, and the nonprintables as their escaped strings. Hope this helps! (Hope this works... )

    My only problem, is that i kep getting the contents of other cariables, so i think i have an array problem somewhere.

    ~Inquirer
    Compilers:
    GCC on Red Hat 8.1 (Primary)
    GCC on Mac OS X 10.2.4 (Secondary)

    Others:
    MinGW on XP

  14. #14
    Rebooted Inquirer's Avatar
    Join Date
    Apr 2002
    Posts
    281
    My problem is for my outgoing socket, recv() always returns -1. It works fine for incoming sockets, bit not for outgoing. HEre is basically what my program is doing. Would someone tell me if i am missing something?

    1) Get the server from the user. Ex.: www.google.com
    2) Get the IP address of the server. Ex.: 216.239.53.101
    3) Get the port number from the user. Ex. 80
    4) Make the connection.
    5) Use recv() to try to get 1024 bytes of daya from the server at a time. // ERRORS HERE, so show the returned value and perror("recv")
    6) Output the read data
    7) Quit

    Help?

    ~Inquirer
    Compilers:
    GCC on Red Hat 8.1 (Primary)
    GCC on Mac OS X 10.2.4 (Secondary)

    Others:
    MinGW on XP

  15. #15
    End Of Line Hammer's Avatar
    Join Date
    Apr 2002
    Posts
    6,231
    What does perror("recv") tell you?

    Can you post your code, or at least some of it?
    When all else fails, read the instructions.
    If you're posting code, use code tags: [code] /* insert code here */ [/code]

Page 1 of 2 12 LastLast
Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Persistent HTTP Requests
    By reelbigtim in forum Networking/Device Communication
    Replies: 9
    Last Post: 10-08-2008, 12:06 AM
  2. Timing HTTP GET requests
    By traef06 in forum C Programming
    Replies: 9
    Last Post: 09-08-2008, 09:33 PM
  3. viewing HTTP requests
    By DavidP in forum A Brief History of Cprogramming.com
    Replies: 6
    Last Post: 04-09-2007, 06:44 PM
  4. Writing all HTTP requests from a website to a log file
    By goomyman in forum C# Programming
    Replies: 1
    Last Post: 07-29-2005, 09:18 AM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21