Persistent HTTP Requests

I'm writing a program in C that connects through the S

ocket and establishes and TCP/IP connection.

The program connects to http://news.bbc.co.uk and relays the HTML code back for the front page. I have another task and that is to collect the HTML code for the BBC America's page.

I am able to do both separately, but struggle to do it persistently. It either returns a HTTP 400 or 500 error.

This is snippet of my code... Its still rough.

Code:

---

int main {
        int cs ;

        char req[] = "GET / HTTP/1.1\r\n\r\n" ;
        char req2[] = "GET /2/hi/americas/default.stm HTTP/1.1\r\n\r\n" ;
        FILE *fp;

        int rc, rc2 ;
        char buffer[BUFFER_SIZE];

        fp=fopen("Front1.txt", "w");       
        cs = Open ( "news.bbc.co.uk");


        send( cs , req , sizeof req , 0 ) ;
        send( cs, req2, sizeof req2, 0);
        printf( "Sent %d\n%s\n" , rc , req ) ;
        rc=recv( cs , &buffer , sizeof buffer , 0 ) ;
        fprintf(fp, "%s",buffer);
        printf( "Received: %d\n" , rc ) ;

            /*---While there's data, read and print it---*/
            do
            {
                bzero(buffer, sizeof(buffer));
                rc = recv(cs, buffer, sizeof(buffer), 0);
                if ( rc > 0 )
                        fprintf(fp, "%s",buffer);
            }while ( rc > 0 );

---

I believe it my HTTP request. Does anyone know the actual HTTP request procedure for a persistent client? I've done all the reading I can do RFC, you name it.

Any help would be appreciated.

I presume you've tried staggering the send requests in a loop to recieve one, then send the other (I wouldn't think this would matter)? Or throwing a "sleep(1)" between sends?

I haven't done any net programming. Sorry. You might try the question about the headers at http://perlmonks.org, they do a lot of it.

Tried both those...

I always get nailed with a HTTP 400 error at the end... For the 2nd send.

Moved to Networking/Device Communication.

Is the GET request supposed to include a nul terminator?

I think you need to be looking at the KeepAlive settings. I only know Apache (not IIS, BBC is a M$ shop right?), and you can switch this off entirely at the server end.

I'm not not sure how you setup a client connection to enable it in C (assuming the server end is enabled), but I would look into the Perl source as you can set it there on a client connection. But it's an HTTP protocol setting, so will be a header you need to send.

Maybe you have to fake a normative "user-agent"

just checked the http 1.1 spec, and the client can request the connection state in the header:

Connection: Keep-Alive

Have a look at 10.9.1 in the http 1.1 spec.

I figured it out... Very strange solution; just a case of creating a function... Don't know why it works, but it works persistently.

Code:

---

#include<stdio.h>
#include<sys/types.h>
#include<sys/socket.h>
#include<netinet/in.h>
#include<arpa/inet.h>
#include<netdb.h>
#include<string.h>

#define PORT 80
#define BUFFER_SIZE 500

int Open( char * ) ;        // Its code is missing
void Read (int );        // Read Buffer Function
void getHTML ( int , char *);        //getHTML command



main()
{
        int cs ;
        char frontpage[] = "GET / HTTP/1.1\r\nHost: news.bbc.co.uk\r\n\r\n";
        char america[] = "GET /2/hi/americas/default.stm HTTP/1.1\r\nHost: news.bbc.co.uk\r\n\r\n";

        cs = Open ("news.bbc.co.uk");

        getHTML(cs,frontpage);
        getHTML(cs,america);

        close(cs);
}

void getHTML ( int cs, char *request)
{
        int rc ;
        char mess[BUFFER_SIZE];

        rc = send( cs, request , strlen(request) , 0 ) ;
        printf( "Sent %d\n%s\n" , rc , request ) ;
        rc = recv( cs , &mess , sizeof mess , 0 ) ;
        printf( "Received: %d\n" , rc ) ;
        mess[rc] = '\0' ;        // printf %s needs it
        printf( "%s\n" , mess ) ;
        Read(cs);
}

int Open( char *url )
{
        int rc ;
        int cs ;
        struct sockaddr_in csSA, localAd;
        struct hostent *h;

        h = gethostbyname(url);

        csSA.sin_family = h->h_addrtype;

        memcpy((char *) &csSA.sin_addr.s_addr, h->h_addr_list[0], h->h_length);

        csSA.sin_port = htons(PORT);//(LOCAL_SERVER_PORT);

//
// csSA is filled right here. The code was deleted for your
// inconvenience; it contained references to htons() and gethostbyname().
//
        cs = socket( PF_INET , SOCK_STREAM , 0 ) ; // 0 = TCP
       

        localAd.sin_family = AF_INET;

        localAd.sin_addr.s_addr = htonl(INADDR_ANY);

        localAd.sin_port = htons(0);

 

        rc = bind(cs, (struct sockaddr *) &localAd, sizeof(localAd));

        if( cs == -1 ) {
                printf( "Socket failed\n" ) ;
                exit( -1 ) ;
        }
        if( connect( cs , (void *) &csSA , sizeof csSA ) == -1 ) {
                printf( "Connect failed c\n" ) ;
                exit( -1 ) ;
        }
        return cs ;
}

---

My read function was mess, mainly because I was required to parse....

Thanks for your help!

Code:

---

void getHTML ( int cs, char *request)
{
    int rc ;
    char mess[BUFFER_SIZE];

    rc = send( cs, request , strlen(request) , 0 ) ;
    printf( "Sent %d\n%s\n" , rc , request ) ;
    rc = recv( cs , &mess , sizeof mess , 0 ) ;
    printf( "Received: %d\n" , rc ) ;
    mess[rc] = '\0' ;    // printf %s needs it but it could be a buffer overflow.
    printf( "%s\n" , mess ) ;
    Read(cs);
}

---