Thread: Why is my HTTP server slow

I've made a threaded server:

Code:

#include<stdio.h>
#include<stdlib.h>
#include<sys/socket.h>
#include<netinet/in.h>
#include <netdb.h>
#include<string.h>
#include <arpa/inet.h>
#include <fcntl.h> // for open
#include <unistd.h> // for close
#include<pthread.h>


#define BUFSIZE 1024

char client_message[2000];
char buffer[1024];

pthread_mutex_t lock = PTHREAD_MUTEX_INITIALIZER;

/*
 * error - wrapper for perror used for bad syscalls
 */
void error(char *msg) {
    perror(msg);
    exit(1);
}
void handle_connection(int);
void * socketThread(void *arg) {
  int childfd = *((int *)arg);
  printf("Handling socket thread on connection %d\n", childfd);
  handle_connection(childfd);
  printf("thread done\n");
  pthread_exit(NULL);
}

int main(int argc, char **argv){
  if (argc != 2) {
    fprintf(stderr, "usage: %s <port>\n", argv[0]);
    exit(1);
  }
  int portno = atoi(argv[1]);
  int childfd;
  int parentfd;          /* parent socket */


  struct sockaddr_in serveraddr;
  struct sockaddr_in clientaddr; /* client addr */

  socklen_t addr_size;

  parentfd = socket(PF_INET, SOCK_STREAM, 0);
  if (parentfd < 0)
    error("ERROR opening socket");


  serveraddr.sin_family = AF_INET;
  serveraddr.sin_addr.s_addr = INADDR_ANY; //htonl(INADDR_ANY);
  serveraddr.sin_port = htons((unsigned short)portno);

  bind(parentfd, (struct sockaddr *) &serveraddr, sizeof(serveraddr));


  struct hostent *hostp; /* client host info */
  char *hostaddrp;       /* dotted decimal host addr string */


  int MAX = 100;
  if(listen(parentfd,MAX)==0)
    printf("Listening on %d\n", portno);
  else
    printf("Error\n");


  pthread_t tid[MAX + 10];
  int i = 0;
  addr_size = sizeof clientaddr;


  while(1) {
        //Accept call creates a new socket for the incoming connection
    printf("Waiting for accept...\n");
        childfd = accept(parentfd, (struct sockaddr *) &clientaddr, &addr_size);
     /* determine who sent the message */
        hostp = gethostbyaddr((const char *)&clientaddr.sin_addr.s_addr,
                              sizeof(clientaddr.sin_addr.s_addr), AF_INET);
    if (hostp == NULL)
            error("ERROR on gethostbyaddr");
    hostaddrp = inet_ntoa(clientaddr.sin_addr);
    printf("%d %s connected\n", i, hostaddrp);


        //for each client request creates a thread and assign the client request to it to process
        //so the main thread can entertain next request
        if (pthread_create(&tid[i++], NULL, socketThread, &childfd) != 0)
           printf("Failed to create thread\n");
        if( i >= MAX)
        {
          i = 0;
          while(i < MAX)
          {
            pthread_join(tid[i++],NULL);
          }
          i = 0;
        }
  }
  printf("Exiting");
  return 0;
}
void * thread_connection(int* childfd) {
}
/**
 * Sends some data back to the client and closes the connection.
 */
void handle_connection(int childfd) {
  FILE *stream;          /* stream version of childfd */
  char buf[BUFSIZE];     /* message buffer */
  char method[BUFSIZE];  /* request method */
  char uri[BUFSIZE];     /* request uri */
  char version[BUFSIZE]; /* request method */

  if ((stream = fdopen(childfd, "r+")) == NULL)
  error("ERROR on fdopen");


  char response[] = "Hello World";
  fprintf(stream, "HTTP/1.1 200 OK\n");
  fprintf(stream, "Server: Seven\n");
  fprintf(stream, "Content-Length: %d\n", (int)strlen(response));
  fprintf(stream, "\r\n");
  fprintf(stream, "%s", response);
  fflush(stream);

  fclose(stream);

  sleep(2);

  close(childfd);
}

However the maximum throughput it can handle is 20 connections/second (benched with ab). And it's regardless of threads, if i use 1 thread or 100 it doesn't matter.

Code:

Server Port:            8131


Document Path:          /test
Document Length:        0 bytes


Concurrency Level:      10
Time taken for tests:   5.627 seconds
Complete requests:      100
Failed requests:        2
   (Connect: 0, Receive: 2, Length: 0, Exceptions: 0)
Total transferred:      6200 bytes
HTML transferred:       0 bytes
Requests per second:    17.77 [#/sec] (mean)
Time per request:       562.667 [ms] (mean)
Time per request:       56.267 [ms] (mean, across all concurrent requests)
Transfer rate:          1.08 [Kbytes/sec] received


Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       22   30  19.0     27     212
Processing:    21   26   7.4     24      60
Waiting:       21   26   7.3     24      60
Total:         46   56  21.3     52     236

The speed is good, but the number of concurrent request isn't. I've put the sleep command there just to make sure that the threads are really working.

So on the same machine, I have the simplest node server and it's 100 requests per second. The same if it runs behind nginx proxy or nginx unit. I can't believe that I can't write the simplest C server that cannot produce the same result? I must be doing something wrong.

Also when doing testing, I would get Connection reset by peer (54) quite often, so there must be something wrong with my code?

Additionally, it always prints

Handling socket thread on connection 4
Handling socket thread on connection 5
Handling socket thread on connection 4
Handling socket thread on connection 5
Handling socket thread on connection 4
Handling socket thread on connection 5

And so on I don't get it how can it use the same FD for different connections huh?

Finally, when i quit the server and restart it, I can't connect to it no more, probably for about 10 minutes or maybe less. Hence I have to change ports all the time. Is it because I need to unbind the socket or clean up threads or something? Thanks!

And actually, one last thing. ab won't spawn new concurrent connection until the server sends back something. I'd of thought that the TCP ACK would be enough but it needs to see some bytes. Does it basically mean I need to open r+ stream, send some headers, then close it, then do the job (eg query DB) then open it again and send the body?

Code:

        childfd = accept(parentfd, (struct sockaddr *) &clientaddr, &addr_size);
...
        if (pthread_create(&tid[i++], NULL, socketThread, &childfd) != 0)

This is probably why you're getting lots of connection errors.

If the main thread does another accept, before the previous instance of the created thread gets to do
> int childfd = *((int *)arg);
then you lose.

Hacky way.

Code:

int childfd = (int)arg;
...
if (pthread_create(&tid[i++], NULL, socketThread, (void*)childfd) != 0)

Better way.

Code:

typedef struct {
  int childfd;
} thread_arg_t;

void * socketThread(void *arg) {
  thread_arg_t *ap = arg;
  int childfd = ap->childfd;
  free(ap);

...

thread_arg_t *ap = malloc(sizeof(*ap));
ap->childfd = childfd;
if (pthread_create(&tid[i++], NULL, socketThread, ap) != 0)

Your thread reaping is also clunky, in that you have to periodically stop what you're doing and wait for everything to finish.
If one thread refuses to die in a timely manner, you're stuck.

Perhaps pthread_tryjoin_np(3) - Linux manual page

> Finally, when i quit the server and restart it, I can't connect to it no more, probably for about 10 minutes or maybe less
Read this.
Safari | Linux Socket Programming by Example -> Setting the SO_REUSEADDR Option