innercoder.com

A blog for Linux programming enthusiasts

Implementing a Prethreaded Web Server

| Comments

In this post I will analyze an implementation of a prethreaded TCP server that leverages the HTTP protocol. This is a personal ongoing project that will continue its development and I found it to be at a good place to start sharing what I have done so far. The whole purpose of this is to later on built a tool to test its performance and how to break it with malformed packets. It is such an interesting project that everytime I sit down with it, I start thinking about different features to implement on it. Again all this is coded in C with the POSIX threads and sockets API.

The current implementation of this server hanldes HTTP GET request with a pool of threads. The amount of threads in the pool is given as an argument to the program. One interesting feature that I have in mind is for the server to deduce the size of the thread pool based on the number of cores and the current load as a default option. The previous version of this server was handling threads per request, which is not very efficient.

The analysis I will do is quick and to the point. These are two very interesting APIs (threads, & sockets) that are relatively easy to follow, once you have some experience.

The complete code is at its github repository.

utils.c

1
2
3
4
5
6
/* usage function */
void usage(char *argv)
{
  printf("Usage: %s <PORT> <DIRECTORY> <#THREADS>\n", argv);
  exit(EXIT_SUCCESS);
}

Usage function, currently all arguments are obligatory. Another future feature to add is for the server to be binded to a specifi interface and work as a daemon.

main.c

1
2
3
4
5
6
7
8
9
/* pthread functions */
void *thr_func(void *);

/* initializing mutex lock */
pthread_mutex_t mtx = PTHREAD_MUTEX_INITIALIZER;

/* static globals */
static int listenfd;
static char fname[FILEDAT]; /* holds webserver directory */

Global variables. Here we have the funtion that is called by pthread_create(), the mutex initializer, the globally shared listening socket file descriptor, and a buffer for the server file directory.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
int main(int argc, char **argv)
{
  /* server socket variables */
  int x;
  int port;
  int nthreads;
  int optval = 1;
  struct sockaddr_in svaddr;
  
  /* handling webserver arguments */
  if (argc != 4)
      usage(argv[0]);
  port = atoi(argv[1]);      /* !!No bounds checking */
  strcpy(fname, argv[2]);        /* !!No bounds checking */

  /* thread structure initialization */
  nthreads = atoi(argv[3]);
  struct thread *thr_ctl = malloc(sizeof(struct thread) * nthreads);
  if (!thr_ctl)
      error_msg("error allocating thr_ctl");

  /* initializing server address socket */
  svaddr.sin_family = AF_INET;
  svaddr.sin_addr.s_addr = htonl(INADDR_ANY);
  svaddr.sin_port = htons((unsigned short) port);
  
  /* defining listening socket descriptor */
  if ((listenfd = socket(svaddr.sin_family, SOCK_STREAM, 0)) < 0 )
      error_msg("error on socket()");
  if (setsockopt(listenfd, SOL_SOCKET, SO_REUSEADDR, &optval,
              sizeof(optval)) < 0)
      error_msg("error on setsockopt()");
  
  /* bind socket to IP address */
  if (bind(listenfd,(struct sockaddr *) &svaddr,
              sizeof(struct sockaddr_in)) < 0)
      error_msg("error on bind()");

  /* listen on socket */
  if ((listen(listenfd, LISTENQ)) < 0)
      error_msg("error on listen()");
  

  /* pre-threading procedure */
  for (x = 0; x < nthreads; x++) {
      if((pthread_create(&thr_ctl[x].tid, NULL, thr_func,(void *)&x))
          != 0)
          error_msg("Thread create error");
      if ((pthread_detach(thr_ctl[x].tid)) != 0)
          error_msg("Thread detach error");
  }

  for(;;)
      pause();

  return(EXIT_SUCCESS);
}

Tried commenting as much as possible. It sums to initializing important socket variables, handling arguments for port and thread number, setting up the server socket address structure and its setup for connections, and at the end, the procedure for prethreading.

For the prethreading procedure, all threads are keep in control with a structure . Currently this structure is not doing anything but to keep an index to them, but I will later on use it for displaying statistics of thread usage and logging. Also, there’s not main thread waiting for other’s to finish, as soon as they are created they are detached for now.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
void *thr_func(void *arg)
{
  /* variables to handle http process */
  char buff[DATLEN];
  struct st_trx *wb_trx;
  struct sockaddr_in claddr;
  socklen_t sin_size = sizeof(claddr);

  /* main procedure */
  for(;;) {
      wb_trx = malloc((sizeof(struct st_trx)));
      if (!wb_trx)
          error_msg("error allocating wb_trx"); 
      
      /* initialize web file struct */
      strcpy(wb_trx->file_name, fname);
      wb_trx->stat_ct = 0;
      wb_trx->dyn_ct = 0;
      
      pthread_mutex_lock(&mtx);    
      if ((wb_trx->trx_fd = accept(listenfd,
                      (struct sockaddr *) &claddr,
                      &sin_size)) < 0)
          error_msg("error on accept()");
      pthread_mutex_unlock(&mtx);  

      printf("server: got connection from %s port %d\n",
              inet_ntoa(claddr.sin_addr),
              ntohs(claddr.sin_port));
      
      /* Read request line and headers */
      recv_msg(wb_trx->trx_fd, buff, DATLEN);

      sscanf(buff, "%s %s %s", wb_trx->method, wb_trx->uri,
             wb_trx->ver);
      printf("method: %s\nuri: %s\nver: %s\n\n", wb_trx->method,
             wb_trx->uri, wb_trx->ver);
      
      serve_rq(wb_trx);
  }

  return(EXIT_SUCCESS);

}

With main() and serve_rq(), this is one of the most important functions so far. The struct st_trx *wb_trx pointer variable is allocated and contains all the information from a client (web browser) request. This structure is filled with the data read from the socket buffer and send to serve_rq() for processing. There are still some variables that are initialized and serve no purpose.

utils.c

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
/* serve static content */
void serve_rq(struct st_trx *wb_trx)
{
  char *addr;
  
  /* get file stats */
  get_file_stats(wb_trx);
      
  if ((addr = mmap(0, wb_trx->file_size, PROT_READ, MAP_PRIVATE,
           wb_trx->file_fd, 0)) == ((void *) -1))
      error_msg("error on mmap()");
  
  if (close(wb_trx->file_fd) < 0)
      error_msg("error closing filefd");
  
  call_http("200", wb_trx->trx_fd, wb_trx);
  send_msg(wb_trx->trx_fd, addr);

  if (munmap(addr, wb_trx->file_size) < 0)
      error_msg("munmap() error");
  /* freeing web transaction */
  free(wb_trx);
}

This is the function that currently gets .html stored file data from server and calls respective http functions. The plan is to have one function with centralized movement, sort of spoke and hub communication. The current HTTP api functions need to be made to return status.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
/* receive msg function */
size_t recv_msg(int fd, char *usr_buff, size_t nbytes)
{
  size_t nleft = nbytes;
  ssize_t nrecv;
  char *recv_buff = usr_buff;

  while (nleft > 0) {
      if ((nrecv = recv(fd, recv_buff, nleft, 0)) < 0) {
          if (errno == EINTR)
              nrecv = 0;
          else
              error_msg("recv() error");
      } else if ((nrecv == 0 ) || strstr(recv_buff, "\r\n"))
          break;
      nleft -= nrecv;
      recv_buff += nrecv;
  }

  return nbytes - nleft;
  
}

/* send msg function */
size_t send_msg(int fd, char *buff)
{
  size_t nleft = strlen(buff);
  ssize_t nsend = 0;

  /* send content */
  printf("Bytes to send: %d\n", (int) nleft);
  while (nleft > 0) {
      if ((nsend = send(fd, buff, nleft, 0)) <= 0 ) {
          if ( nsend < 0 && errno == EINTR)
              nsend = 0;
          else
              error_msg("error with send()");
      }
      nleft -= nsend;
      buff += nsend;
  }
  if (nleft == 0)
      printf("Send success!!\n");
  return nleft;
}

These function still need some work with the return values and error detection . It is another of the high priorities. They are wrappers to the send() and recv() socket function calls.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
[jaime@LenoLX xwbserver]$ ./xwbserver 5600 . 8
server: got connection from 127.0.0.1 port 50451
method: GET
uri: /home-dyn.html
ver: HTTP/1.1

Filename: ./home-dyn.html
Bytes to send: 83
Send success!!
Bytes to send: 407
Send success!!
server: got connection from 127.0.0.1 port 50452
method: GET
uri: /home-pic1.html
ver: HTTP/1.1

Filename: ./home-pic1.html
Bytes to send: 83
Send success!!
Bytes to send: 270
Send success!!
server: got connection from 127.0.0.1 port 50453
method: GET
uri: /smiley.gif
ver: HTTP/1.1

Filename: ./smiley.gif
File not found
Bytes to send: 142
Send success!!
server: got connection from 127.0.0.1 port 50454
method: GET
uri: /landscape.jpg
ver: HTTP/1.1

Filename: ./landscape.jpg
Bytes to send: 85
Send success!!
Bytes to send: 4
Send success!!

Usage example using a browser.

Comments