Persistent HTTP Requests

This is a discussion on Persistent HTTP Requests within the Networking/Device Communication forums, part of the General Programming Boards category; I'm writing a program in C that connects through the S ocket and establishes and TCP/IP connection. The program connects ...

  1. #1
    Registered User
    Join Date
    Oct 2007
    Posts
    8

    Persistent HTTP Requests

    I'm writing a program in C that connects through the S

    ocket and establishes and TCP/IP connection.

    The program connects to http://news.bbc.co.uk and relays the HTML code back for the front page. I have another task and that is to collect the HTML code for the BBC America's page.

    I am able to do both separately, but struggle to do it persistently. It either returns a HTTP 400 or 500 error.

    This is snippet of my code... Its still rough.

    Code:
    int main {
    	int cs ;
    
    	char req[] = "GET / HTTP/1.1\r\n\r\n" ;
    	char req2[] = "GET /2/hi/americas/default.stm HTTP/1.1\r\n\r\n" ;
    	FILE *fp;
    
    	int rc, rc2 ;
    	char buffer[BUFFER_SIZE];
    
    	fp=fopen("Front1.txt", "w");	
    	cs = Open ( "news.bbc.co.uk");
    
    
    	send( cs , req , sizeof req , 0 ) ;
    	send( cs, req2, sizeof req2, 0);
    	printf( "Sent %d\n%s\n" , rc , req ) ;
    	rc=recv( cs , &buffer , sizeof buffer , 0 ) ;
    	fprintf(fp, "%s",buffer);
    	printf( "Received: %d\n" , rc ) ;
    
        	/*---While there's data, read and print it---*/
        	do
        	{
            	bzero(buffer, sizeof(buffer));
            	rc = recv(cs, buffer, sizeof(buffer), 0);
            	if ( rc > 0 )
            		fprintf(fp, "%s",buffer);
        	}while ( rc > 0 );
    I believe it my HTTP request. Does anyone know the actual HTTP request procedure for a persistent client? I've done all the reading I can do RFC, you name it.

    Any help would be appreciated.

  2. #2
    spurious conceit MK27's Avatar
    Join Date
    Jul 2008
    Location
    segmentation fault
    Posts
    8,300
    I presume you've tried staggering the send requests in a loop to recieve one, then send the other (I wouldn't think this would matter)? Or throwing a "sleep(1)" between sends?

    I haven't done any net programming. Sorry. You might try the question about the headers at http://perlmonks.org, they do a lot of it.
    C programming resources:
    GNU C Function and Macro Index -- glibc reference manual
    The C Book -- nice online learner guide
    Current ISO draft standard
    CCAN -- new CPAN like open source library repository
    3 (different) GNU debugger tutorials: #1 -- #2 -- #3
    cpwiki -- our wiki on sourceforge

  3. #3
    Registered User
    Join Date
    Oct 2007
    Posts
    8
    Tried both those...

    I always get nailed with a HTTP 400 error at the end... For the 2nd send.

  4. #4
    C++ Witch laserlight's Avatar
    Join Date
    Oct 2003
    Location
    Singapore
    Posts
    22,303
    Moved to Networking/Device Communication.
    C + C++ Compiler: MinGW port of GCC
    Version Control System: Bazaar

    Look up a C++ Reference and learn How To Ask Questions The Smart Way

  5. #5
    Registered User
    Join Date
    Oct 2001
    Posts
    2,129
    Is the GET request supposed to include a nul terminator?

  6. #6
    Registered User
    Join Date
    Mar 2008
    Posts
    11
    I think you need to be looking at the KeepAlive settings. I only know Apache (not IIS, BBC is a M$ shop right?), and you can switch this off entirely at the server end.

    I'm not not sure how you setup a client connection to enable it in C (assuming the server end is enabled), but I would look into the Perl source as you can set it there on a client connection. But it's an HTTP protocol setting, so will be a header you need to send.

  7. #7
    spurious conceit MK27's Avatar
    Join Date
    Jul 2008
    Location
    segmentation fault
    Posts
    8,300
    Maybe you have to fake a normative "user-agent"
    C programming resources:
    GNU C Function and Macro Index -- glibc reference manual
    The C Book -- nice online learner guide
    Current ISO draft standard
    CCAN -- new CPAN like open source library repository
    3 (different) GNU debugger tutorials: #1 -- #2 -- #3
    cpwiki -- our wiki on sourceforge

  8. #8
    Registered User
    Join Date
    Mar 2008
    Posts
    11
    just checked the http 1.1 spec, and the client can request the connection state in the header:

    Connection: Keep-Alive

    Have a look at 10.9.1 in the http 1.1 spec.

  9. #9
    Registered User
    Join Date
    Oct 2007
    Posts
    8
    I figured it out... Very strange solution; just a case of creating a function... Don't know why it works, but it works persistently.

    Code:
    #include<stdio.h>
    #include<sys/types.h>
    #include<sys/socket.h>
    #include<netinet/in.h>
    #include<arpa/inet.h>
    #include<netdb.h>
    #include<string.h>
    
    #define PORT 80
    #define BUFFER_SIZE 500
    
    int Open( char * ) ;	// Its code is missing
    void Read (int );	// Read Buffer Function
    void getHTML ( int , char *);	//getHTML command
    
    
    
    main()
    {
    	int cs ;
    	char frontpage[] = "GET / HTTP/1.1\r\nHost: news.bbc.co.uk\r\n\r\n";
    	char america[] = "GET /2/hi/americas/default.stm HTTP/1.1\r\nHost: news.bbc.co.uk\r\n\r\n";
    
    	cs = Open ("news.bbc.co.uk");
    
    	getHTML(cs,frontpage);
    	getHTML(cs,america);
    
    	close(cs);
    }
    
    void getHTML ( int cs, char *request)
    {
    	int rc ;
    	char mess[BUFFER_SIZE];
    
    	rc = send( cs, request , strlen(request) , 0 ) ;
    	printf( "Sent %d\n%s\n" , rc , request ) ;
    	rc = recv( cs , &mess , sizeof mess , 0 ) ;
    	printf( "Received: %d\n" , rc ) ;
    	mess[rc] = '\0' ;	// printf %s needs it
    	printf( "%s\n" , mess ) ;
    	Read(cs);
    } 
    
    int Open( char *url )
    {
    	int rc ;
    	int cs ;
    	struct sockaddr_in csSA, localAd;
    	struct hostent *h;
    
    	h = gethostbyname(url);
    
    	csSA.sin_family = h->h_addrtype;
    
    	memcpy((char *) &csSA.sin_addr.s_addr, h->h_addr_list[0], h->h_length);
    
    	csSA.sin_port = htons(PORT);//(LOCAL_SERVER_PORT);
    
    //
    // csSA is filled right here. The code was deleted for your
    // inconvenience; it contained references to htons() and gethostbyname().
    // 
    	cs = socket( PF_INET , SOCK_STREAM , 0 ) ; // 0 = TCP
    	
    
    	localAd.sin_family = AF_INET;
    
    	localAd.sin_addr.s_addr = htonl(INADDR_ANY);
    
    	localAd.sin_port = htons(0);
    
      
    
    	rc = bind(cs, (struct sockaddr *) &localAd, sizeof(localAd));
    
    	if( cs == -1 ) {
    		printf( "Socket failed\n" ) ;
    		exit( -1 ) ;
    	}
    	if( connect( cs , (void *) &csSA , sizeof csSA ) == -1 ) {
    		printf( "Connect failed c\n" ) ;
    		exit( -1 ) ;
    	}
    	return cs ;
    }
    My read function was mess, mainly because I was required to parse....

    Thanks for your help!

  10. #10
    Registered User
    Join Date
    Oct 2001
    Posts
    2,129
    Code:
    void getHTML ( int cs, char *request)
    {
        int rc ;
        char mess[BUFFER_SIZE];
    
        rc = send( cs, request , strlen(request) , 0 ) ;
        printf( "Sent &#37;d\n%s\n" , rc , request ) ;
        rc = recv( cs , &mess , sizeof mess , 0 ) ;
        printf( "Received: %d\n" , rc ) ;
        mess[rc] = '\0' ;    // printf %s needs it but it could be a buffer overflow.
        printf( "%s\n" , mess ) ;
        Read(cs);
    }

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Timing HTTP GET requests
    By traef06 in forum C Programming
    Replies: 9
    Last Post: 09-08-2008, 10:33 PM
  2. http requests
    By xixpsychoxix in forum Networking/Device Communication
    Replies: 3
    Last Post: 05-04-2008, 02:13 PM
  3. viewing HTTP requests
    By DavidP in forum A Brief History of Cprogramming.com
    Replies: 6
    Last Post: 04-09-2007, 07:44 PM
  4. Writing all HTTP requests from a website to a log file
    By goomyman in forum C# Programming
    Replies: 1
    Last Post: 07-29-2005, 10:18 AM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21