sending an http GET request in C

here is my code:

Code:

/* 
 * File:   webClient.c
 * Author: Adam
 *
 * Created on November 3, 2011, 1:26 AM
 */

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <netdb.h>

/*
 * 
 */
int main(int argc, char** argv) {
    
    char arg[500];
    char firstHalf[500];
    char secondHalf[500];
    char request[1000];
    struct hostent *server;
    struct sockaddr_in serveraddr;
    int port = 80;
    
    strcpy(arg, argv[1]);
    
    int i;
    for (i = 0; i < strlen(arg); i++)
    {
        if (arg[i] == '/')
        {
                strncpy(firstHalf, arg, i);
                firstHalf[i] = '\0';
                break;
        }     
    }
    
    for (i; i < strlen(arg); i++)
    {
        strcat(secondHalf, &arg[i]);
        break;
    }
    
    printf("\nFirst Half: %s", firstHalf);
    
    printf("\nSecond Half: %s", secondHalf);
    
    int tcpSocket = socket(AF_INET, SOCK_STREAM, 0);
    
    if (tcpSocket < 0)
        printf("\nError opening socket");
    else
        printf("\nSuccessfully opened socket");
    
    server = gethostbyname(firstHalf);
    
    if (server == NULL)
    {
        printf("gethostbyname() failed\n");
    }
    else
    {
        printf("\n%s = ", server->h_name);
        unsigned int j = 0;
        while (server -> h_addr_list[j] != NULL)
        {
            printf("%s", inet_ntoa(*(struct in_addr*)(server -> h_addr_list[j])));
            j++;
        }
    }
    
    printf("\n");

    bzero((char *) &serveraddr, sizeof(serveraddr));
    serveraddr.sin_family = AF_INET;

    bcopy((char *)server->h_addr, (char *)&serveraddr.sin_addr.s_addr, server->h_length);
    
    serveraddr.sin_port = htons(port);
    
    if (connect(tcpSocket, (struct sockaddr *) &serveraddr, sizeof(serveraddr)) < 0)
        printf("\nError Connecting");
    else
        printf("\nSuccessfully Connected");
  
    bzero(request, 1000);

    sprintf(request, "Get %s HTTP/1.1\r\n Host: %s\r\n \r\n \r\n", secondHalf, firstHalf);
    
    printf("\n%s", request);
    
    if (send(tcpSocket, request, strlen(request), 0) < 0)
        printf("Error with send()");
    else
        printf("Successfully sent html fetch request");
    
    bzero(request, 1000);
    
    recv(tcpSocket, request, 999, 0);
    printf("\n%s", request);
    printf("\nhello");
    
    close(tcpSocket);
    
    return (EXIT_SUCCESS);
}

i get hung up on the send while the recv code is included. but if i comment the recv line out then the send works successfully. what am i doing wrong?

Originally Posted by c++guy

here is my code:

Code:

    sprintf(request, "Get %s HTTP/1.1\r\n Host: %s\r\n \r\n \r\n", secondHalf, firstHalf);

Technically, that is a bad request, because you have a space before Host:, but per RFC2616, there should not be any whitespace in a header field name.

Worse: you do not want a space between your /r/n's. Of which two should separate the header from the body ("/r/n/r/n"), not three. That may not seem like such a big issue because there is no body, but you still need an "/r/n/r/n" to indicate the header is complete, and you do not have that. Hence, I think many or most web servers will not respond in kind, and will instead terminate the connection immediately because the request is malformed. That termination takes place via tcp/ip, not http, and if you are doing the wrong thing with the socket (eg, waiting on recv) at the time, it could be fatal to the process.*

i get hung up on the send while the recv code is included. but if i comment the recv line out then the send works successfully.

How are you are deducing this? Do you mean in the former case "Error with send()" appears? Or nothing? If it is nothing, I would bet that the send did succeed, but the stdout buffer is not flushed before your process recieves a SIGPIPE or SIGHUP while waiting on the recv. Try it with:

Code:

    if (send(tcpSocket, request, strlen(request), 0) < 0)
        fprintf(stderr, "Error with send()");
    else
        fprintf(stderr, "Successfully sent html fetch request");

This will avoid buffering issues with stdout if the process has actually died prematurely while waiting on the recv(). Ie, the the connection was terminated via tcp/ip by the connected server, without sending any http reponse. How servers deal with non-conforming clients is...undefined.

Of course, if you do not wait on the recv(), you will get "Successfully sent html fetch request"; you sent a bad request and so the server (on a tcp/ip level) killed the connection (just as before), but it does not matter because your program already ended (and flushed the stdout buffer on exit).

* you almost certainly want to use a signal handler for SIGPIPE because of that.

Originally Posted by MK27

Technically, that is a bad request, because you have a space before Host:, but per RFC2616, there should not be any whitespace in a header field name.

Worse: you do not want a space between your /r/n's. Of which two should separate the header from the body ("/r/n/r/n"), not three. That may not seem like such a big issue because there is no body, but you still need an "/r/n/r/n" to indicate the header is complete, and you do not have that. Hence, I think many or most web servers will not respond in kind, and will instead terminate the connection immediately because the request is malformed. That termination takes place via tcp/ip, not http, and if you are doing the wrong thing with the socket (eg, waiting on recv) at the time, it could be fatal to the process.*



How are you are deducing this? Do you mean in the former case "Error with send()" appears? Or nothing? If it is nothing, I would bet that the send did succeed, but the stdout buffer is not flushed before your process recieves a SIGPIPE or SIGHUP while waiting on the recv. Try it with:

Code:

    if (send(tcpSocket, request, strlen(request), 0) < 0)
        fprintf(stderr, "Error with send()");
    else
        fprintf(stderr, "Successfully sent html fetch request");

This will avoid buffering issues with stdout if the process has actually died prematurely while waiting on the recv(). Ie, the the connection was terminated via tcp/ip by the connected server, without sending any http reponse. How servers deal with non-conforming clients is...undefined.

Of course, if you do not wait on the recv(), you will get "Successfully sent html fetch request"; you sent a bad request and so the server (on a tcp/ip level) killed the connection (just as before), but it does not matter because your program already ended (and flushed the stdout buffer on exit).

* you almost certainly want to use a signal handler for SIGPIPE because of that.

you are awesome! it was because of the spaces and the fact that i had 3 \r\n's. my teacher had 3 in his provided code for some reason so i had no idea that had a chance at being incorrect. thanks a lot!

Two threads on this forum, and cross-posted as well.