Persistent HTTP Requests

This is a discussion on Persistent HTTP Requests within the Networking/Device Communication forums, part of the General Programming Boards category; I'm writing a program in C that connects through the S ocket and establishes and TCP/IP connection. The program connects ...

  1. #1
    Registered User
    Join Date
    Oct 2007
    Posts
    8

    Persistent HTTP Requests

    I'm writing a program in C that connects through the S

    ocket and establishes and TCP/IP connection.

    The program connects to http://news.bbc.co.uk and relays the HTML code back for the front page. I have another task and that is to collect the HTML code for the BBC America's page.

    I am able to do both separately, but struggle to do it persistently. It either returns a HTTP 400 or 500 error.

    This is snippet of my code... Its still rough.

    Code:
    int main {
    	int cs ;
    
    	char req[] = "GET / HTTP/1.1\r\n\r\n" ;
    	char req2[] = "GET /2/hi/americas/default.stm HTTP/1.1\r\n\r\n" ;
    	FILE *fp;
    
    	int rc, rc2 ;
    	char buffer[BUFFER_SIZE];
    
    	fp=fopen("Front1.txt", "w");	
    	cs = Open ( "news.bbc.co.uk");
    
    
    	send( cs , req , sizeof req , 0 ) ;
    	send( cs, req2, sizeof req2, 0);
    	printf( "Sent %d\n%s\n" , rc , req ) ;
    	rc=recv( cs , &buffer , sizeof buffer , 0 ) ;
    	fprintf(fp, "%s",buffer);
    	printf( "Received: %d\n" , rc ) ;
    
        	/*---While there's data, read and print it---*/
        	do
        	{
            	bzero(buffer, sizeof(buffer));
            	rc = recv(cs, buffer, sizeof(buffer), 0);
            	if ( rc > 0 )
            		fprintf(fp, "%s",buffer);
        	}while ( rc > 0 );
    I believe it my HTTP request. Does anyone know the actual HTTP request procedure for a persistent client? I've done all the reading I can do RFC, you name it.

    Any help would be appreciated.

  2. #2
    spurious conceit MK27's Avatar
    Join Date
    Jul 2008
    Location
    segmentation fault
    Posts
    8,300
    I presume you've tried staggering the send requests in a loop to recieve one, then send the other (I wouldn't think this would matter)? Or throwing a "sleep(1)" between sends?

    I haven't done any net programming. Sorry. You might try the question about the headers at http://perlmonks.org, they do a lot of it.
    C programming resources:
    GNU C Function and Macro Index -- glibc reference manual
    The C Book -- nice online learner guide
    Current ISO draft standard
    CCAN -- new CPAN like open source library repository
    3 (different) GNU debugger tutorials: #1 -- #2 -- #3
    cpwiki -- our wiki on sourceforge

  3. #3
    Registered User
    Join Date
    Oct 2007
    Posts
    8
    Tried both those...

    I always get nailed with a HTTP 400 error at the end... For the 2nd send.

  4. #4
    C++ Witch laserlight's Avatar
    Join Date
    Oct 2003
    Location
    Singapore
    Posts
    23,629
    Moved to Networking/Device Communication.
    Quote Originally Posted by Bjarne Stroustrup (2000-10-14)
    I get maybe two dozen requests for help with some sort of programming or design problem every day. Most have more sense than to send me hundreds of lines of code. If they do, I ask them to find the smallest example that exhibits the problem and send me that. Mostly, they then find the error themselves. "Finding the smallest program that demonstrates the error" is a powerful debugging tool.
    Look up a C++ Reference and learn How To Ask Questions The Smart Way

  5. #5
    Registered User
    Join Date
    Oct 2001
    Posts
    2,129
    Is the GET request supposed to include a nul terminator?

  6. #6
    Registered User
    Join Date
    Mar 2008
    Posts
    11
    I think you need to be looking at the KeepAlive settings. I only know Apache (not IIS, BBC is a M$ shop right?), and you can switch this off entirely at the server end.

    I'm not not sure how you setup a client connection to enable it in C (assuming the server end is enabled), but I would look into the Perl source as you can set it there on a client connection. But it's an HTTP protocol setting, so will be a header you need to send.

  7. #7
    spurious conceit MK27's Avatar
    Join Date
    Jul 2008
    Location
    segmentation fault
    Posts
    8,300
    Maybe you have to fake a normative "user-agent"
    C programming resources:
    GNU C Function and Macro Index -- glibc reference manual
    The C Book -- nice online learner guide
    Current ISO draft standard
    CCAN -- new CPAN like open source library repository
    3 (different) GNU debugger tutorials: #1 -- #2 -- #3
    cpwiki -- our wiki on sourceforge

  8. #8
    Registered User
    Join Date
    Mar 2008
    Posts
    11
    just checked the http 1.1 spec, and the client can request the connection state in the header:

    Connection: Keep-Alive

    Have a look at 10.9.1 in the http 1.1 spec.

  9. #9
    Registered User
    Join Date
    Oct 2007
    Posts
    8
    I figured it out... Very strange solution; just a case of creating a function... Don't know why it works, but it works persistently.

    Code:
    #include<stdio.h>
    #include<sys/types.h>
    #include<sys/socket.h>
    #include<netinet/in.h>
    #include<arpa/inet.h>
    #include<netdb.h>
    #include<string.h>
    
    #define PORT 80
    #define BUFFER_SIZE 500
    
    int Open( char * ) ;	// Its code is missing
    void Read (int );	// Read Buffer Function
    void getHTML ( int , char *);	//getHTML command
    
    
    
    main()
    {
    	int cs ;
    	char frontpage[] = "GET / HTTP/1.1\r\nHost: news.bbc.co.uk\r\n\r\n";
    	char america[] = "GET /2/hi/americas/default.stm HTTP/1.1\r\nHost: news.bbc.co.uk\r\n\r\n";
    
    	cs = Open ("news.bbc.co.uk");
    
    	getHTML(cs,frontpage);
    	getHTML(cs,america);
    
    	close(cs);
    }
    
    void getHTML ( int cs, char *request)
    {
    	int rc ;
    	char mess[BUFFER_SIZE];
    
    	rc = send( cs, request , strlen(request) , 0 ) ;
    	printf( "Sent %d\n%s\n" , rc , request ) ;
    	rc = recv( cs , &mess , sizeof mess , 0 ) ;
    	printf( "Received: %d\n" , rc ) ;
    	mess[rc] = '\0' ;	// printf %s needs it
    	printf( "%s\n" , mess ) ;
    	Read(cs);
    } 
    
    int Open( char *url )
    {
    	int rc ;
    	int cs ;
    	struct sockaddr_in csSA, localAd;
    	struct hostent *h;
    
    	h = gethostbyname(url);
    
    	csSA.sin_family = h->h_addrtype;
    
    	memcpy((char *) &csSA.sin_addr.s_addr, h->h_addr_list[0], h->h_length);
    
    	csSA.sin_port = htons(PORT);//(LOCAL_SERVER_PORT);
    
    //
    // csSA is filled right here. The code was deleted for your
    // inconvenience; it contained references to htons() and gethostbyname().
    // 
    	cs = socket( PF_INET , SOCK_STREAM , 0 ) ; // 0 = TCP
    	
    
    	localAd.sin_family = AF_INET;
    
    	localAd.sin_addr.s_addr = htonl(INADDR_ANY);
    
    	localAd.sin_port = htons(0);
    
      
    
    	rc = bind(cs, (struct sockaddr *) &localAd, sizeof(localAd));
    
    	if( cs == -1 ) {
    		printf( "Socket failed\n" ) ;
    		exit( -1 ) ;
    	}
    	if( connect( cs , (void *) &csSA , sizeof csSA ) == -1 ) {
    		printf( "Connect failed c\n" ) ;
    		exit( -1 ) ;
    	}
    	return cs ;
    }
    My read function was mess, mainly because I was required to parse....

    Thanks for your help!

  10. #10
    Registered User
    Join Date
    Oct 2001
    Posts
    2,129
    Code:
    void getHTML ( int cs, char *request)
    {
        int rc ;
        char mess[BUFFER_SIZE];
    
        rc = send( cs, request , strlen(request) , 0 ) ;
        printf( "Sent &#37;d\n%s\n" , rc , request ) ;
        rc = recv( cs , &mess , sizeof mess , 0 ) ;
        printf( "Received: %d\n" , rc ) ;
        mess[rc] = '\0' ;    // printf %s needs it but it could be a buffer overflow.
        printf( "%s\n" , mess ) ;
        Read(cs);
    }

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Timing HTTP GET requests
    By traef06 in forum C Programming
    Replies: 9
    Last Post: 09-08-2008, 09:33 PM
  2. http requests
    By xixpsychoxix in forum Networking/Device Communication
    Replies: 3
    Last Post: 05-04-2008, 01:13 PM
  3. viewing HTTP requests
    By DavidP in forum A Brief History of Cprogramming.com
    Replies: 6
    Last Post: 04-09-2007, 06:44 PM
  4. Writing all HTTP requests from a website to a log file
    By goomyman in forum C# Programming
    Replies: 1
    Last Post: 07-29-2005, 09:18 AM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21