Persistent HTTP Requests

I'm writing a program in C that connects through the S

ocket and establishes and TCP/IP connection.

The program connects to http://news.bbc.co.uk and relays the HTML code back for the front page. I have another task and that is to collect the HTML code for the BBC America's page.

I am able to do both separately, but struggle to do it persistently. It either returns a HTTP 400 or 500 error.

This is snippet of my code... Its still rough.

Code:

int main {
	int cs ;

	char req[] = "GET / HTTP/1.1\r\n\r\n" ;
	char req2[] = "GET /2/hi/americas/default.stm HTTP/1.1\r\n\r\n" ;
	FILE *fp;

	int rc, rc2 ;
	char buffer[BUFFER_SIZE];

	fp=fopen("Front1.txt", "w");	
	cs = Open ( "news.bbc.co.uk");


	send( cs , req , sizeof req , 0 ) ;
	send( cs, req2, sizeof req2, 0);
	printf( "Sent %d\n%s\n" , rc , req ) ;
	rc=recv( cs , &buffer , sizeof buffer , 0 ) ;
	fprintf(fp, "%s",buffer);
	printf( "Received: %d\n" , rc ) ;

    	/*---While there's data, read and print it---*/
    	do
    	{
        	bzero(buffer, sizeof(buffer));
        	rc = recv(cs, buffer, sizeof(buffer), 0);
        	if ( rc > 0 )
        		fprintf(fp, "%s",buffer);
    	}while ( rc > 0 );

I believe it my HTTP request. Does anyone know the actual HTTP request procedure for a persistent client? I've done all the reading I can do RFC, you name it.

Any help would be appreciated.

I presume you've tried staggering the send requests in a loop to recieve one, then send the other (I wouldn't think this would matter)? Or throwing a "sleep(1)" between sends?

I haven't done any net programming. Sorry. You might try the question about the headers at http://perlmonks.org, they do a lot of it.

Tried both those...

I always get nailed with a HTTP 400 error at the end... For the 2nd send.

Moved to Networking/Device Communication.

Is the GET request supposed to include a nul terminator?

I think you need to be looking at the KeepAlive settings. I only know Apache (not IIS, BBC is a M$ shop right?), and you can switch this off entirely at the server end.

I'm not not sure how you setup a client connection to enable it in C (assuming the server end is enabled), but I would look into the Perl source as you can set it there on a client connection. But it's an HTTP protocol setting, so will be a header you need to send.

Maybe you have to fake a normative "user-agent"

just checked the http 1.1 spec, and the client can request the connection state in the header:

Connection: Keep-Alive

Have a look at 10.9.1 in the http 1.1 spec.

I figured it out... Very strange solution; just a case of creating a function... Don't know why it works, but it works persistently.

Code:

#include<stdio.h>
#include<sys/types.h>
#include<sys/socket.h>
#include<netinet/in.h>
#include<arpa/inet.h>
#include<netdb.h>
#include<string.h>

#define PORT 80
#define BUFFER_SIZE 500

int Open( char * ) ;	// Its code is missing
void Read (int );	// Read Buffer Function
void getHTML ( int , char *);	//getHTML command



main()
{
	int cs ;
	char frontpage[] = "GET / HTTP/1.1\r\nHost: news.bbc.co.uk\r\n\r\n";
	char america[] = "GET /2/hi/americas/default.stm HTTP/1.1\r\nHost: news.bbc.co.uk\r\n\r\n";

	cs = Open ("news.bbc.co.uk");

	getHTML(cs,frontpage);
	getHTML(cs,america);

	close(cs);
}

void getHTML ( int cs, char *request)
{
	int rc ;
	char mess[BUFFER_SIZE];

	rc = send( cs, request , strlen(request) , 0 ) ;
	printf( "Sent %d\n%s\n" , rc , request ) ;
	rc = recv( cs , &mess , sizeof mess , 0 ) ;
	printf( "Received: %d\n" , rc ) ;
	mess[rc] = '\0' ;	// printf %s needs it
	printf( "%s\n" , mess ) ;
	Read(cs);
} 

int Open( char *url )
{
	int rc ;
	int cs ;
	struct sockaddr_in csSA, localAd;
	struct hostent *h;

	h = gethostbyname(url);

	csSA.sin_family = h->h_addrtype;

	memcpy((char *) &csSA.sin_addr.s_addr, h->h_addr_list[0], h->h_length);

	csSA.sin_port = htons(PORT);//(LOCAL_SERVER_PORT);

//
// csSA is filled right here. The code was deleted for your
// inconvenience; it contained references to htons() and gethostbyname().
// 
	cs = socket( PF_INET , SOCK_STREAM , 0 ) ; // 0 = TCP
	

	localAd.sin_family = AF_INET;

	localAd.sin_addr.s_addr = htonl(INADDR_ANY);

	localAd.sin_port = htons(0);

  

	rc = bind(cs, (struct sockaddr *) &localAd, sizeof(localAd));

	if( cs == -1 ) {
		printf( "Socket failed\n" ) ;
		exit( -1 ) ;
	}
	if( connect( cs , (void *) &csSA , sizeof csSA ) == -1 ) {
		printf( "Connect failed c\n" ) ;
		exit( -1 ) ;
	}
	return cs ;
}

My read function was mess, mainly because I was required to parse....

Thanks for your help!

Code:

void getHTML ( int cs, char *request)
{
    int rc ;
    char mess[BUFFER_SIZE];

    rc = send( cs, request , strlen(request) , 0 ) ;
    printf( "Sent %d\n%s\n" , rc , request ) ;
    rc = recv( cs , &mess , sizeof mess , 0 ) ;
    printf( "Received: %d\n" , rc ) ;
    mess[rc] = '\0' ;    // printf %s needs it but it could be a buffer overflow.
    printf( "%s\n" , mess ) ;
    Read(cs);
}