NGINX Tutorial: Developing Modules (with Video)

This NGINX tutorial and the accompanying video will be a look into developing modules for the NGINX web server. Typically we use web servers like NGINX and Apache as simple reverse proxies for our web based software, leaving a lot of functionality on the table. We will explore not only how to build an NGINX module, but also build something that could provide value to a web application by allowing NGINX to handle some of the early routing process.

NGINX modules can be written in a number of languages. The most common are C, Perl, and Lua. This tutorial will use C as the implementation language and Ruby as a supporting language. It offers the greatest amount of control and integration with NGINX and can make a big difference in the performance department. Writing NGINX modules in C also helps you internalize the NGINX life-cycle and how it operates on requests.

As we develop our module we will want to make sure that the results are as expected. Let’s setup a build and test harness to help keep us in line as we write our code.

NGINX modules consist of two parts; a config file and a module file. The config file tells NGINX how to compile the module and what it is called. The module file contains the actual code. The module file can of course be broken up into multiple files, but we will only have one for this exercise.

Let’s start with the config file:

config

ngx_addon_name=ngx_http_auth_token_module
HTTP_MODULES="$HTTP_MODULES ngx_http_auth_token_module"
NGX_ADDON_SRCS="$NGX_ADDON_SRCS $ngx_addon_dir/ngx_http_auth_token_module.c"

This file tells the NGINX compilation process what the module name is and where the source code can be found. We will come back to this file later to add some additional items but for now this is all we need.

Now that we have a working module it’s time to write the real part of our module. The first part of which is finding the value of a cookie. In particular, we want to find the value of the cookie that contains an authentication token. To do this we will leverage a built in NGINX function called ngx_http_parse_multi_header_lines. The code is as follows:

ngx_int_t location;
ngx_str_t cookie = (ngx_str_t)ngx_string("auth_token");
ngx_str_t cookie_value;
location = ngx_http_parse_multi_header_lines(&r->headers_in.cookies, &cookie, &cookie_value);

This function takes the cookies from the request, the name of the cookie to extract, and a variable to store the value of the cookie. Right now the name of the cookie is hard coded. We will fix that when we add configuration directives. Let’s update our handler to extract the value and redirect if the cookie is not found.

ngx_http_auth_token_module.c

static ngx_int_t
ngx_http_auth_token_handler(ngx_http_request_t *r)
{
  if (r->main->internal) {
    return NGX_DECLINED;
  }

  r->main->internal = 1;

  ngx_str_t cookie = (ngx_str_t)ngx_string("auth_token");
  ngx_int_t location;
  ngx_str_t cookie_value;
  location = ngx_http_parse_multi_header_lines(&r->headers_in.cookies, &cookie, &cookie_value);

  if (location == NGX_DECLINED) {
    ngx_table_elt_t *h;
    h = ngx_list_push(&r->headers_out.headers);
    h->hash = 1;
    ngx_str_set(&h->key, "Location");
    ngx_str_set(&h->value, "http://google.com");
    return NGX_HTTP_MOVED_TEMPORARILY;
  } else {
    ngx_table_elt_t *h;
    h = ngx_list_push(&r->headers_out.headers);
    h->hash = 1;
    ngx_str_set(&h->key, "X-Auth-Token");
    h->value = cookie_value;
  }

  return NGX_DECLINED;
}

After adding the cookie extraction code, we added an if statement to either display the header back to the user or redirect if the header isn’t present. We will continue by adding some lookup code to validate that the auth token is in fact legitimate. Let’s recompile and give it a try.

$ rake nginx:stop
$ rake nginx:compile
$ rake nginx:start
$ curl -I localhost:8888
HTTP/1.1 302 Moved Temporarily
Server: nginx/1.4.7
Date: Mon, 30 Jun 2014 20:12:53 GMT
Content-Type: text/html
Content-Length: 160
Connection: keep-alive
Location: http://google.com

$ curl -I -b "auth_token=test" localhost:8888
HTTP/1.1 200 OK
Server: nginx/1.4.7
Date: Mon, 30 Jun 2014 20:13:07 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Mon, 30 Jun 2014 19:38:26 GMT
Connection: keep-alive
X-Auth-Token: test
ETag: "53b1bcb2-264"
Accept-Ranges: bytes

The next step is to lookup the auth token provided and ensure that it is valid. For this exercise, valid simply means that the token can be found. We will use Redis as our token storage system. There are plenty of other options, and some that work better in this scenario. Redis has a clean and easy to use C library, and makes for a lot less code. It is perfectly acceptable to use Redis in this situation.

We are now able to retrieve the value of a cookie and look it up in Redis. The last thing our module needs is some logic. We want to redirect if the request doesn’t contain the auth token, or if the auth token isn’t found. If it is found, we want to allow the request to continue. In order to do this effectively, we are going to want to do a little refactoring. Let’s start by extracting the two paths of our handler into functions.

ngx_http_auth_token_module.c

static ngx_int_t
redirect(ngx_http_request_t *r, ngx_str_t *location)
{
  ngx_table_elt_t *h;
  h = ngx_list_push(&r->headers_out.headers);
  h->hash = 1;
  ngx_str_set(&h->key, "Location");
  h->value = *location;

  return NGX_HTTP_MOVED_TEMPORARILY;
}

First, we create a redirect function. We will be using it twice, so it’s nice to have it tucked away for reuse. This is just a function extraction, nothing changed from the original code. Next, let’s
extract the code that’s left:

ngx_http_auth_token_module.c

static void
append_user_id(ngx_http_request_t *r, ngx_str_t *user_id)
{
  ngx_table_elt_t *h;
  h = ngx_list_push(&r->headers_out.headers);
  h->hash = 1;
  ngx_str_set(&h->key, "X-User-Id");
  h->value = *user_id;
}

Here’s where things change a bit. This function takes the user id that we looked up from Redis and appends it as the value in our new header. Notice that the name of the header has changed from X-Auth-Token to X-User-Id. This is because the value we will eventually pass to our downstream application is the user id. Now we can rewrite our handler to use these new functions and put in place
the logic we want.

ngx_http_auth_token_module.c

static ngx_int_t
ngx_http_auth_token_handler(ngx_http_request_t *r)
{
  if (r->main->internal) {
    return NGX_DECLINED;
  }

  r->main->internal = 1;

  ngx_str_t location = (ngx_str_t)ngx_string("http://google.com");
  ngx_str_t cookie = (ngx_str_t)ngx_string("auth_token");
  ngx_int_t lookup;
  ngx_str_t auth_token;
  lookup = ngx_http_parse_multi_header_lines(&r->headers_in.cookies, &cookie, &auth_token);

  if (lookup == NGX_DECLINED) {
    return redirect(r, &location);
  } else {
    ngx_str_t user_id;
    ngx_int_t lookup_result = lookup_user(&auth_token, &user_id);

    if (lookup_result == NGX_DECLINED) {
      return redirect(r, &location);
    } else {
      append_user_id(r, &user_id);
      return NGX_DECLINED;
    }
  }
}

The consequence of the cookie lookup now just invokes our redirect function. The alternative attempts to lookup the auth token using the lookup_user function. If it fails the redirect function is invoked. If it passes we invoke our append_user_id function and exit our handler. Let’s recompile and test out our module.

$ rake nginx:stop
$ rake nginx:compile
$ rake nginx:start
$ curl -I -b "auth_token=test" localhost:8888
HTTP/1.1 302 Moved Temporarily
Server: nginx/1.4.7
Date: Mon, 30 Jun 2014 20:31:27 GMT
Content-Type: text/html
Content-Length: 160
Connection: keep-alive
Location: http://google.com

$ curl -I localhost:8888
HTTP/1.1 302 Moved Temporarily
Server: nginx/1.4.7
Date: Mon, 30 Jun 2014 20:32:50 GMT
Content-Type: text/html
Content-Length: 160
Connection: keep-alive
Location: http://google.com

We can see that we are being redirected in both situations where we expect. Let’s add the test key into Redis and see what happens.

$ redis-cli set test abedra
OK
$ curl -I -b "auth_token=test" localhost:8888
HTTP/1.1 200 OK
Server: nginx/1.4.7
Date: Mon, 30 Jun 2014 20:33:36 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Mon, 30 Jun 2014 19:38:26 GMT
Connection: keep-alive
X-User-Id: abedra
ETag: "53b1bcb2-264"
Accept-Ranges: bytes

This time the lookup was successful and the X-User-Id header was set to the value of the test key.

This is the last time we are going to see the new header reflected back to us. It isn’t useful or necessary to add it to the response. We need to make sure that the application that comes after our NGINX server sees this header. Let’s modify our append_user_id function to make this happen.

ngx_http_auth_token_module.c

static void
append_user_id(ngx_http_request_t *r, ngx_str_t *user_id)
{
  ngx_table_elt_t *h;
  h = ngx_list_push(&r->headers_in.headers);
  h->hash = 1;
  ngx_str_set(&h->key, "X-User-Id");
  h->value = *user_id;
}

If we recompile and test again, we will see that the header is no longer in the response. To test this now we will need to add a backing application. We won’t do that quite yet though, there’s still some finishing touches we need to put on our module.

Until now all of the values in our module were hardcoded. This doesn’t make a great module. There are a few things we would like to configure:

• The Redis host
• The Redis port
• The cookie name for the auth token
• The location of the redirect

To get the idea of how we want the end result to look, let’s update our nginx.conf file to reflect the desired state.

nginx.conf

events {
  worker_connections 1024;
}

http {
  auth_token_redis_host "localhost";
  auth_token_redis_port 6379;
  auth_token_cookie_name "auth_token";
  auth_token_redirect_location "http://google.com";

  server {
    listen 8888;
    location / {

    }
  }
}

This example shows what our configuration will look like when we are finished. In NGINX there are multiple places that you can have configuration variables. The section in the http block is referred to as the NGX_HTTP_MAIN_CONF, the server block NGX_HTTP_SRV_CONF, and the location block NGX_HTTP_LOC_CONF. There are additional options that won’t be discussed in this NGINX tutorial. For a full list you can visit
http://lxr.nginx.org/source/src/http/ngx_http_config.h. Directives are allowed to be declared in any combination of locations. To make things a little clearer we will only use NGX_HTTP_MAIN_CONF in this example. One might argue that it would make more sense to have them in other places. While that is correct this is an exercise left to the reader.

To get started we need to create a place to hold the configuration. We will declare a struct at the top of our module.

ngx_http_auth_token_module.c

typedef struct {
  ngx_str_t redis_host;
  ngx_int_t redis_port;
  ngx_str_t cookie_name;
  ngx_str_t redirect_location;
} auth_token_main_conf_t;

Next, we need to add the definition for our directives.

ngx_http_auth_token_module.c

static ngx_command_t ngx_http_auth_token_commands[] = {
  {
    ngx_string("auth_token_redis_host"),
    NGX_HTTP_MAIN_CONF|NGX_CONF_TAKE1,
    ngx_conf_set_str_slot,
    NGX_HTTP_MAIN_CONF_OFFSET,
    offsetof(auth_token_main_conf_t, redis_host),
    NULL
  },
  {
    ngx_string("auth_token_redis_port"),
    NGX_HTTP_MAIN_CONF|NGX_CONF_TAKE1,
    ngx_conf_set_num_slot,
    NGX_HTTP_MAIN_CONF_OFFSET,
    offsetof(auth_token_main_conf_t, redis_port),
    NULL
  },
  {
    ngx_string("auth_token_cookie_name"),
    NGX_HTTP_MAIN_CONF|NGX_CONF_TAKE1,
    ngx_conf_set_str_slot,
    NGX_HTTP_MAIN_CONF_OFFSET,
    offsetof(auth_token_main_conf_t, cookie_name),
    NULL
  },
  {
    ngx_string("auth_token_redirect_location"),
    NGX_HTTP_MAIN_CONF|NGX_CONF_TAKE1,
    ngx_conf_set_str_slot,
    NGX_HTTP_MAIN_CONF_OFFSET,
    offsetof(auth_token_main_conf_t, redirect_location),
    NULL
  },

  ngx_null_command
};

This section of code is pretty straight forward. For each directive you have a name, the configuration location, the number of arguments it accepts, the type of argument, the configuration offset, and the value in the configuration struct to populate. There are just two more things left to do and we are ready to test it out.

Next, we need to tell the NGINX how to find these commands and what to default them to. First lets wire them in:

ngx_http_auth_token_module.c

static ngx_http_module_t ngx_http_auth_token_module_ctx = {
  NULL,                                 /* preconfiguration */
  ngx_http_auth_token_init,             /* postconfiguration */
  ngx_http_auth_token_create_main_conf, /* create main configuration */
  NULL,                                 /* init main configuration */
  NULL,                                 /* create server configuration */
  NULL,                                 /* merge server configuration */
  NULL,                                 /* create location configuration */
  NULL                                  /* merge location configuration */
};

ngx_module_t ngx_http_auth_token_module = {
  NGX_MODULE_V1,
  &ngx_http_auth_token_module_ctx, /* module context */
  ngx_http_auth_token_commands,    /* module directives */
  NGX_HTTP_MODULE,                 /* module type */
  NULL,                            /* init master */
  NULL,                            /* init module */
  NULL,                            /* init process */
  NULL,                            /* init thread */
  NULL,                            /* exit thread */
  NULL,                            /* exit process */
  NULL,                            /* exit master */
  NGX_MODULE_V1_PADDING
};

Note that the module directives has changed from NULL to the command list we just created. Along with this we added a reference to a function named ngx_http_auth_token_create_main_conf. This function will allocate and set default values for our configuration struct. Let’s write that now.

ngx_http_auth_token_module.c

static void*
ngx_http_auth_token_create_main_conf(ngx_conf_t *cf)
{
  auth_token_main_conf_t *conf;

  conf = ngx_pcalloc(cf->pool, sizeof(auth_token_main_conf_t));
  if (conf == NULL) {
    return NULL;
  }

  conf->redis_port = NGX_CONF_UNSET_UINT;

  return conf;
}

This code allocates the auth_token_main_conf_t value to ensure that it is usable by the module. Now we need to update our code to use the configuration values. Let’s update our handler.

ngx_http_auth_token_module.c

static ngx_int_t
ngx_http_auth_token_handler(ngx_http_request_t *r)
{
  if (r->main->internal) {
    return NGX_DECLINED;
  }

  r->main->internal = 1;

  auth_token_main_conf_t *conf = ngx_http_get_module_main_conf(r, ngx_http_auth_token_module);

  ngx_int_t cookie_location;
  ngx_str_t auth_token;
  cookie_location = ngx_http_parse_multi_header_lines(&r->headers_in.cookies, &conf->cookie_name, &auth_token);

  if (cookie_location == NGX_DECLINED) {
    return redirect(r, &conf->redirect_location);
  } else {
    ngx_str_t user_id;
    ngx_int_t lookup_result = lookup_user(conf, &auth_token, &user_id);

    if (lookup_result == NGX_DECLINED) {
      return redirect(r, &conf->redirect_location);
    } else {
      append_user_id(r, &user_id);
      return NGX_DECLINED;
    }
  }
}

We have now removed the hardcoded values. In their place we created an instance of our auth_token_main_conf_t and populated it using the ngx_http_get_module_main_conf function. After this we are able to use the conf variable to get to our configuration values. There is one more change that we still have to make. We added an extra argument to the lookup_user function. This is the configuration information so that the function can connect to Redis using the configuration values rather than the hardcoded ones. Let’s update the function to reflect the new signature and remove the hardcoded data.

ngx_http_auth_token_module.c

static ngx_int_t
lookup_user(auth_token_main_conf_t *conf, ngx_str_t *auth_token, ngx_str_t *user_id)
{
  redisContext *context = redisConnect((const char*)conf->redis_host.data, conf->redis_port);
  redisReply *reply = redisCommand(context, "GET %s", auth_token->data);
  if (reply->type == REDIS_REPLY_NIL) {
    return NGX_DECLINED;
  } else {
    ngx_str_set(user_id, reply->str);
    return NGX_OK;
  }
}

These changes are pretty straight forward. We add conf to the function signature and use its values inside the function. Remember that the conf contains NGINX data structures, so to get string values you need to use the data member of the value. Now we’re ready to give it a try!

Instead of firing everything up and running a bunch of curl commands, let’s update our test suite to do this automatically.

spec/integration/spec_helper.rb

require 'spec_helper'

describe "Integration Specs" do

  before do
    @redis = Redis.new
    @redis.flushdb
  end

  after  { @redis.flushdb }

  describe "Use cases" do
    it "Redirects when no auth token is present" do
      http = Curl.get("http://127.0.0.1:8888") do |http|
        http.headers['Cookie'] = ""
      end

      expect(http.response_code).to eq(302)
    end

    it "Redirects when the auth token is invalid" do
      @redis.set("test", "sucess")

      http = Curl.get("http://127.0.0.1:8888") do |http|
        http.headers['Cookie'] = "auth_token=invalid"
      end

      expect(http.response_code).to eq(302)
    end

    it "Allows the request when the auth token is valid" do
      @redis.set("test", "sucess")

      http = Curl.get("http://127.0.0.1:8888") do |http|
        http.headers['Cookie'] = "auth_token=test"
      end

      expect(http.response_code).to eq(200)
    end
  end
end

Before each test we reset our Redis database. This will make sure we only have the values we want inside of each of the tests. We then create three tests that express the boundaries of our module. We check the HTTP status code of the response to ensure that the module is acting the way we intend it to. Let’s recompile and run our tests.

$ rake nginx:compile
...
$ rake
/home/abedra/.rvm/rubies/ruby-2.0.0-p451/bin/ruby -I/home/abedra/.rvm/gems/ruby-2.0.0-p451/gems/rspec-core-3.0.2/lib:/home/abedra/.rvm/gems/ruby-2.0.0-p451/gems/rspec-support-3.0.2/lib -S /home/abedra/.rvm/gems/ruby-2.0.0-p451/gems/rspec-core-3.0.2/exe/rspec spec/integration/integration_spec.rb

Integration Specs
  Use cases
    Redirects when no auth token is present
    Redirects when the auth token is invalid
    Allows the request when the auth token is valid

Finished in 0.00552 seconds (files took 0.18856 seconds to load)
3 examples, 0 failures

Now that we have a working test suite we have a nice base for future additions to our module.

In order to see the results of our module in action we need to have an application running behind our NGINX server. It doesn’t need to be fancy. In fact, it really doesn’t need to do much of anything except print the user id so that we can see everything working as expected. We already have an environment setup for Ruby, so let’s create a simple Ruby web application that does just this.

app.rb

require 'sinatra'

get '/' do
  "Hello #{request.env['HTTP_X_USER_ID']}"
end

This program will respond if the proper auth token is presented to NGINX. Before we can test everything we need to update our NGINX configuration to add our new program as the receiver of our requests.

nginx.conf

events {
  worker_connections 1024;
}

http {
  auth_token_redis_host "localhost";
  auth_token_redis_port 6379;
  auth_token_cookie_name "auth_token";
  auth_token_redirect_location "http://google.com";

  upstream app {
    server localhost:4567;
  }

  server {
    listen 8888;
    location / {
      proxy_pass http://app;
    }
  }
}

We added an upstream block that points to our local host at port 4567, which is where our program will run by default. Now we just need to start NGINX and our new program:

$ rake nginx:start
$ ruby app.rb
[2014-06-30 16:23:11] INFO  WEBrick 1.3.1
[2014-06-30 16:23:11] INFO  ruby 2.0.0 (2014-02-24) [x86_64-linux]
== Sinatra/1.4.5 has taken the stage on 4567 for development with backup from WEBrick
[2014-06-30 16:23:11] INFO  WEBrick::HTTPServer#start: pid=35606 port=4567
127.0.0.1 - - [30/Jun/2014 16:23:44] "GET / HTTP/1.1" 200 12 0.0069
localhost - - [30/Jun/2014:16:23:44 CDT] "GET / HTTP/1.0" 200 12

Let’s make sure we have a token in our Redis database then query the server.

$ redis-cli set test abedra
OK
$ curl -b "auth_token=test" localhost:8888
Hello abedra

We can see that our module is correctly passing the information back to the downstream application.

We now have a fully functioning NGINX module complete with test suite. We accomplished a lot and hopefully learned some new things. There are lots of places we can go with this module but there are a few things we should discuss before wrapping up.

• Should I use this module in production?
  Probably not. It is lacking some flexibility in configuration and general robustness and error handling. Along with that this module is still pretty plain. Additional features may be necessary to not take away from the intended security mechanisms that may already be in place for your application. That being said, this is a solid foundation and the rest of the work doesn’t take too much additional effort.
• Where do I go from here?
  There are quite a few resources available. The general go to place for
  information is the mailing lists. There is also the canonical guide to NGINX module writing from Evan Miller. It’s a great resource for general information about getting started writing modules.

Finally, you should experiment with this module and see what else you can add to it and share what you have created with the world!