Submitting HTML form and parse respons

IdunnO · 02-09-2006, 06:54 AM

Hi everybody,

I'm currently working on a project which requires data which can be retreived by filling in a HTML on the Internet and submitting it. The server then sends back something like this:

Code:

#### Begin of results ####

108
107
101
107
102
106
90
100
106
94
101
93

#### End of results (lines produced: 14) ####

This is not in a HTML format, but raw code (a string is sent back with the data).

My problem is as follows: I have written a program in C which needs this data. Therefore I want the C program to retrieve that information automatically. I think I have to write a piece of code which opens a socket, sends the appropriate HTML headers to the server, waits for the response and parse it, am I correct?

Requesting the information is done by sending an URL such as http://server/retrieve.cgi?parameter1=xx&parameter2=xx to the server.

Does anyone know a tutorial about a similar problem or a piece of example code which I can alter?

by the way, I'm writing my programs on my Linux box (don't know if that's important information).

**Salem** · 02-09-2006, 06:59 AM

Or you could use the command line tools like wget and curl to send appropriate HTML requests and save the responses to a file.

Use ethereal to observe the message sequence between client and server, if you intend to write this yourself.

Also, moved to the networking forum, who's intro threads you should read.

octavian · 02-23-2006, 01:00 PM

sprintf(request, "GET %s HTTP/1.0\r\n"
"Host: %s\r\n"
"User-Agent: BlaBla/1.0\r\n\r\n",
"webpage.php",
"www.somehost.com");

s_socket.send((BYTE*)request, strlen(request)+1)
where the socket class is located at:
www.zalsoft.com getnet source code /baselib/ folder

IdunnO · 03-07-2006, 07:10 AM

Hi, thanks for the response!

I've created a string such as:

Code:

  data = "GET / HTTP/1.1\r\n"
  "Host: www.servername.nl\r\n"
  "Connection: close\r\n"
  "\r\n";

I'm getting the right response, so that's great!

In my case I have to have permission to access the data. This is done by a username and password. If I don't provide this information I get a

Code:

HTTP/1.1 401 Authorization Required

response.

Is there a way to provide the username and password in the http headers? My instinct says this isn't possible and I have to talk to the server several times (to provide the name and password) before I can retrieve the page. If this is the case, I have no idea how this is done through HTTP headers.

btw, I've tried ethereal (actually, tehtereal) but haven't made any good progress so far.

edit: I also see a

Code:

WWW-Authenticate: Basic realm="Monica"

in the server response....

**Salem** · 03-07-2006, 10:28 AM

Rather than just guessing, why not wander over to http://www.rfc-editor.org/
and grab the RFC's which define HTTP.

02-09-2006, 06:54 AM	#1
IdunnO Registered User Join Date: Feb 2006 Location: Leiden, NL Posts: 12	Submitting HTML form and parse respons Hi everybody, I'm currently working on a project which requires data which can be retreived by filling in a HTML on the Internet and submitting it. The server then sends back something like this: Code: #### Begin of results #### 108 107 101 107 102 106 90 100 106 94 101 93 #### End of results (lines produced: 14) #### This is not in a HTML format, but raw code (a string is sent back with the data). My problem is as follows: I have written a program in C which needs this data. Therefore I want the C program to retrieve that information automatically. I think I have to write a piece of code which opens a socket, sends the appropriate HTML headers to the server, waits for the response and parse it, am I correct? Requesting the information is done by sending an URL such as http://server/retrieve.cgi?parameter1=xx&parameter2=xx to the server. Does anyone know a tutorial about a similar problem or a piece of example code which I can alter? by the way, I'm writing my programs on my Linux box (don't know if that's important information).
IdunnO is offline

02-09-2006, 06:59 AM	#2
Salem and the hat of Jobseeking Join Date: Aug 2001 Location: The edge of the known universe Posts: 21,710	Or you could use the command line tools like wget and curl to send appropriate HTML requests and save the responses to a file. Use ethereal to observe the message sequence between client and server, if you intend to write this yourself. Also, moved to the networking forum, who's intro threads you should read.
Salem is offline

02-23-2006, 01:00 PM	#3
octavian Registered User Join Date: Feb 2006 Posts: 5	post message sprintf(request, "GET %s HTTP/1.0\r\n" "Host: %s\r\n" "User-Agent: BlaBla/1.0\r\n\r\n", "webpage.php", "www.somehost.com"); s_socket.send((BYTE*)request, strlen(request)+1) where the socket class is located at: www.zalsoft.com getnet source code /baselib/ folder
octavian is offline

03-07-2006, 07:10 AM	#4
IdunnO Registered User Join Date: Feb 2006 Location: Leiden, NL Posts: 12	getting there Hi, thanks for the response! I've created a string such as: Code: data = "GET / HTTP/1.1\r\n" "Host: www.servername.nl\r\n" "Connection: close\r\n" "\r\n"; I'm getting the right response, so that's great! In my case I have to have permission to access the data. This is done by a username and password. If I don't provide this information I get a Code: HTTP/1.1 401 Authorization Required response. Is there a way to provide the username and password in the http headers? My instinct says this isn't possible and I have to talk to the server several times (to provide the name and password) before I can retrieve the page. If this is the case, I have no idea how this is done through HTTP headers. btw, I've tried ethereal (actually, tehtereal) but haven't made any good progress so far. edit: I also see a Code: WWW-Authenticate: Basic realm="Monica" in the server response.... Last edited by IdunnO; 03-07-2006 at 07:17 AM.
IdunnO is offline

03-07-2006, 10:28 AM	#5
Salem and the hat of Jobseeking Join Date: Aug 2001 Location: The edge of the known universe Posts: 21,710	Rather than just guessing, why not wander over to http://www.rfc-editor.org/ and grab the RFC's which define HTTP.
Salem is offline

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode