Development guide
Introduction
Code layout
-
auto
- build scripts -
src
-
core
- basic types and functions - string, array, log, pool etc -
event
- event core-
modules
- event notification modules: epoll, kqueue, select etc
-
-
http
- core HTTP module and common code-
modules
- other HTTP modules -
v2
- HTTPv2
-
-
mail
- mail modules -
os
- platform-specific code-
unix
-
win32
-
-
stream
- stream modules
-
Include files
Each nginx file should start with including the following two files:
#include <ngx_config.h> #include <ngx_core.h>
In addition to that, HTTP code should include
#include <ngx_http.h>
Mail code should include
#include <ngx_mail.h>
Stream code should include
#include <ngx_stream.h>
Integers
For general purpose, nginx code uses the following two integer types
ngx_int_t
and ngx_uint_t
which are
typedefs for intptr_t
and uintptr_t
.
Common return codes
Most functions in nginx return the following codes:
-
NGX_OK
- operation succeeded -
NGX_ERROR
- operation failed -
NGX_AGAIN
- operation incomplete, function should be called again -
NGX_DECLINED
- operation rejected, for example, if disabled in configuration. This is never an error -
NGX_BUSY
- resource is not available -
NGX_DONE
- operation done or continued elsewhere. Also used as an alternative success code -
NGX_ABORT
- function was aborted. Also used as an alternative error code
Error handling
For getting the last system error code, the ngx_errno
macro
is available.
It's mapped to errno
on POSIX platforms and to
GetLastError()
call in Windows.
For getting the last socket error number, the
ngx_socket_errno
macro is available.
It's mapped to errno
on POSIX systems as well,
and to WSAGetLastError()
call on Windows.
For performance reasons the values of ngx_errno
or
ngx_socket_errno
should not be accessed more than
once in a row.
The error value should be stored in a local variable of type
ngx_err_t
for using multiple times, if required.
For setting errors, ngx_set_errno(errno)
and
ngx_set_socket_errno(errno)
macros are available.
The values of ngx_errno
or
ngx_socket_errno
can be passed to logging functions
ngx_log_error()
and ngx_log_debugX()
, in
which case system error text is added to the log message.
Example using ngx_errno
:
void ngx_my_kill(ngx_pid_t pid, ngx_log_t *log, int signo) { ngx_err_t err; if (kill(pid, signo) == -1) { err = ngx_errno; ngx_log_error(NGX_LOG_ALERT, log, err, "kill(%P, %d) failed", pid, signo); if (err == NGX_ESRCH) { return 2; } return 1; } return 0; }
Strings
Overview
For C strings, nginx code uses unsigned character type pointer
u_char *
.
The nginx string type ngx_str_t
is defined as follows:
typedef struct { size_t len; u_char *data; } ngx_str_t;
The len
field holds the string length,
data
holds the string data.
The string, held in ngx_str_t
, may or may not be
null-terminated after the len
bytes.
In most cases it’s not.
However, in certain parts of code (for example, when parsing configuration),
ngx_str_t
objects are known to be null-terminated, and that
knowledge is used to simplify string comparison and makes it easier to pass
those strings to syscalls.
A number of string operations are provided in nginx.
They are declared in src/core/ngx_string.h
.
Some of them are wrappers around standard C functions:
-
ngx_strcmp()
-
ngx_strncmp()
-
ngx_strstr()
-
ngx_strlen()
-
ngx_strchr()
-
ngx_memcmp()
-
ngx_memset()
-
ngx_memcpy()
-
ngx_memmove()
Some nginx-specific string functions:
-
ngx_memzero()
fills memory with zeroes -
ngx_cpymem()
does the same asngx_memcpy()
, but returns the final destination address This one is handy for appending multiple strings in a row -
ngx_movemem()
does the same asngx_memmove()
, but returns the final destination address. -
ngx_strlchr()
searches for a character in a string, delimited by two pointers
Some case conversion and comparison functions:
-
ngx_tolower()
-
ngx_toupper()
-
ngx_strlow()
-
ngx_strcasecmp()
-
ngx_strncasecmp()
Formatting
A number of formatting functions are provided by nginx. These functions support nginx-specific types:
-
ngx_sprintf(buf, fmt, ...)
-
ngx_snprintf(buf, max, fmt, ...)
-
ngx_slrintf(buf, last, fmt, ...)
-
ngx_vslprint(buf, last, fmt, args)
-
ngx_vsnprint(buf, max, fmt, args)
The full list of formatting options, supported by these functions, can be found
in src/core/ngx_string.c
. Some of them are:
%O - off_t %T - time_t %z - size_t %i - ngx_int_t %p - void * %V - ngx_str_t * %s - u_char * (null-terminated) %*s - size_t + u_char *
The ‘u’ modifier makes most types unsigned, ‘X’/‘x’ convert output to hex.
Example:
u_char buf[NGX_INT_T_LEN]; size_t len; ngx_int_t n; /* set n here */ len = ngx_sprintf(buf, "%ui", n) - buf;
Numeric conversion
Several functions for numeric conversion are implemented in nginx:
-
ngx_atoi(line, n)
- converts a string of given length to a positive integer of typengx_int_t
. ReturnsNGX_ERROR
on error -
ngx_atosz(line, n)
- same forssize_t
type -
ngx_atoof(line, n)
- same foroff_t
type -
ngx_atotm(line, n)
- same fortime_t
type -
ngx_atofp(line, n, point)
- converts a fixed point floating number of given length to a positive integer of typengx_int_t
. The result is shifted left bypoints
decimal positions. The string representation of the number is expected to have no more thanpoints
fractional digits. ReturnsNGX_ERROR
on error. For example,ngx_atofp("10.5", 4, 2)
returns1050
-
ngx_hextoi(line, n)
- converts hexadecimal representation of a positive integer tongx_int_t
. ReturnsNGX_ERROR
on error
Containers
Array
The nginx array type ngx_array_t
is defined as follows
typedef struct { void *elts; ngx_uint_t nelts; size_t size; ngx_uint_t nalloc; ngx_pool_t *pool; } ngx_array_t;
The elements of array are available through the elts
field.
The number of elements is held in the nelts
field.
The size
field holds the size of a single element and is set
when initializing the array.
An array can be created in a pool with the
ngx_array_create(pool, n, size)
call.
An already allocated array object can be initialized with the
ngx_array_init(array, pool, n, size)
call.
ngx_array_t *a, b; /* create an array of strings with preallocated memory for 10 elements */ a = ngx_array_create(pool, 10, sizeof(ngx_str_t)); /* initialize string array for 10 elements */ ngx_array_init(&b, pool, 10, sizeof(ngx_str_t));
Adding elements to array are done with the following functions:
-
ngx_array_push(a)
adds one tail element and returns pointer to it -
ngx_array_push_n(a, n)
addsn
tail elements and returns pointer to the first one
If currently allocated memory is not enough for new elements, a new memory for elements is allocated and existing elements are copied to that memory. The new memory block is normally twice as large, as the existing one.
s = ngx_array_push(a); ss = ngx_array_push_n(&b, 3);
List
List in nginx is a sequence of arrays, optimized for inserting a potentially large number of items. The list type is defined as follows:
typedef struct { ngx_list_part_t *last; ngx_list_part_t part; size_t size; ngx_uint_t nalloc; ngx_pool_t *pool; } ngx_list_t;
The actual items are store in list parts, defined as follows:
typedef struct ngx_list_part_s ngx_list_part_t; struct ngx_list_part_s { void *elts; ngx_uint_t nelts; ngx_list_part_t *next; };
Initially, a list must be initialized by calling
ngx_list_init(list, pool, n, size)
or created by calling
ngx_list_create(pool, n, size)
.
Both functions receive the size of a single item and a number of items per list
part.
The ngx_list_push(list)
function is used to add an item to the
list. Iterating over the items is done by direct accessing the list fields, as
seen in the example:
ngx_str_t *v; ngx_uint_t i; ngx_list_t *list; ngx_list_part_t *part; list = ngx_list_create(pool, 100, sizeof(ngx_str_t)); if (list == NULL) { /* error */ } /* add items to the list */ v = ngx_list_push(list); if (v == NULL) { /* error */ } ngx_str_set(v, “foo”); v = ngx_list_push(list); if (v == NULL) { /* error */ } ngx_str_set(v, “bar”); /* iterate over the list */ part = &list->part; v = part->elts; for (i = 0; /* void */; i++) { if (i >= part->nelts) { if (part->next == NULL) { break; } part = part->next; v = part->elts; i = 0; } ngx_do_smth(&v[i]); }
The primary use for the list in nginx is HTTP input and output headers.
The list does not support item removal.
However, when needed, items can internally be marked as missing without actual
removing from the list.
For example, HTTP output headers which are stored as
ngx_table_elt_t
objects, are marked as missing by setting
the hash
field of ngx_table_elt_t
to
zero. Such items are explicitly skipped, when iterating over the headers.
Queue
Queue in nginx is an intrusive doubly linked list, with each node defined as follows:
typedef struct ngx_queue_s ngx_queue_t; struct ngx_queue_s { ngx_queue_t *prev; ngx_queue_t *next; };
The head queue node is not linked with any data. Before using, the list head
should be initialized with ngx_queue_init(q)
call.
Queues support the following operations:
-
ngx_queue_insert_head(h, x)
,ngx_queue_insert_tail(h, x)
- insert a new node -
ngx_queue_remove(x)
- remove a queue node -
ngx_queue_split(h, q, n)
- split a queue at a node, queue tail is returned in a separate queue -
ngx_queue_add(h, n)
- add second queue to the first queue -
ngx_queue_head(h)
,ngx_queue_last(h)
- get first or last queue node -
ngx_queue_sentinel(h)
- get a queue sentinel object to end iteration at -
ngx_queue_data(q, type, link)
- get reference to the beginning of a queue node data structure, considering the queue field offset in it
Example:
typedef struct { ngx_str_t value; ngx_queue_t queue; } ngx_foo_t; ngx_foo_t *f; ngx_queue_t values; ngx_queue_init(&values); f = ngx_palloc(pool, sizeof(ngx_foo_t)); if (f == NULL) { /* error */ } ngx_str_set(&f->value, “foo”); ngx_queue_insert_tail(&values, f); /* insert more nodes here */ for (q = ngx_queue_head(&values); q != ngx_queue_sentinel(&values); q = ngx_queue_next(q)) { f = ngx_queue_data(q, ngx_foo_t, queue); ngx_do_smth(&f->value); }
Red-Black tree
The src/core/ngx_rbtree.h
header file provides access to the
effective implementation of red-black trees.
typedef struct { ngx_rbtree_t rbtree; ngx_rbtree_node_t sentinel; /* custom per-tree data here */ } my_tree_t; typedef struct { ngx_rbtree_node_t rbnode; /* custom per-node data */ foo_t val; } my_node_t;
To deal with a tree as a whole, you need two nodes: root and sentinel. Typically, they are added to some custom structure, thus allowing to organize your data into a tree which leaves contain a link to or embed your data.
To initialize a tree:
my_tree_t root; ngx_rbtree_init(&root.rbtree, &root.sentinel, insert_value_function);
The insert_value_function
is a function that is
responsible for traversing the tree and inserting new values into correct
place.
For example, the ngx_str_rbtree_insert_value
functions is
designed to deal with ngx_str_t
type.
void ngx_str_rbtree_insert_value(ngx_rbtree_node_t *temp, ngx_rbtree_node_t *node, ngx_rbtree_node_t *sentinel)
Its arguments are pointers to a root node of an insertion, newly created node to be added, and a tree sentinel.
The traversal is pretty straightforward and can be demonstrated with the following lookup function pattern:
my_node_t * my_rbtree_lookup(ngx_rbtree_t *rbtree, foo_t *val, uint32_t hash) { ngx_int_t rc; my_node_t *n; ngx_rbtree_node_t *node, *sentinel; node = rbtree->root; sentinel = rbtree->sentinel; while (node != sentinel) { n = (my_node_t *) node; if (hash != node->key) { node = (hash < node->key) ? node->left : node->right; continue; } rc = compare(val, node->val); if (rc < 0) { node = node->left; continue; } if (rc > 0) { node = node->right; continue; } return n; } return NULL; }
The compare()
is a classic comparator function returning
value less, equal or greater than zero. To speed up lookups and avoid comparing
user objects that can be big, integer hash field is used.
To add a node to a tree, allocate a new node, initialize it and call
ngx_rbtree_insert()
:
my_node_t *my_node; ngx_rbtree_node_t *node; my_node = ngx_palloc(...); init_custom_data(&my_node->val); node = &my_node->rbnode; node->key = create_key(my_node->val); ngx_rbtree_insert(&root->rbtree, node);
to remove a node:
ngx_rbtree_delete(&root->rbtree, node);
Memory management
Heap
To allocate memory from system heap, the following functions are provided by nginx:
-
ngx_alloc(size, log)
- allocate memory from system heap. This is a wrapper aroundmalloc()
with logging support. Allocation error and debugging information is logged tolog
-
ngx_calloc(size, log)
- same asngx_alloc()
, but memory is filled with zeroes after allocation -
ngx_memalign(alignment, size, log)
- allocate aligned memory from system heap. This is a wrapper aroundposix_memalign()
on those platforms which provide it. Otherwise implementation falls back tongx_alloc()
which provides maximum alignment -
ngx_free(p)
- free allocated memory. This is a wrapper aroundfree()
Pool
Most nginx allocations are done in pools. Memory allocated in an nginx pool is freed automatically when the pool in destroyed. This provides good allocation performance and makes memory control easy.
A pool internally allocates objects in continuous blocks of memory. Once a block is full, a new one is allocated and added to the pool memory block list. When a large allocation is requested which does not fit into a block, such allocation is forwarded to the system allocator and the returned pointer is stored in the pool for further deallocation.
Nginx pool has the type ngx_pool_t
.
The following operations are supported:
-
ngx_create_pool(size, log)
- create a pool with given block size. The pool object returned is allocated in the pool as well. -
ngx_destroy_pool(pool)
- free all pool memory, including the pool object itself. -
ngx_palloc(pool, size)
- allocate aligned memory from pool -
ngx_pcalloc(pool, size)
- allocated aligned memory from pool and fill it with zeroes -
ngx_pnalloc(pool, size)
- allocate unaligned memory from pool. Mostly used for allocating strings -
ngx_pfree(pool, p)
- free memory, previously allocated in the pool. Only allocations, forwarded to the system allocator, can be freed.
u_char *p; ngx_str_t *s; ngx_pool_t *pool; pool = ngx_create_pool(1024, log); if (pool == NULL) { /* error */ } s = ngx_palloc(pool, sizeof(ngx_str_t)); if (s == NULL) { /* error */ } ngx_str_set(s, “foo”); p = ngx_pnalloc(pool, 3); if (p == NULL) { /* error */ } ngx_memcpy(p, “foo”, 3);
Since chain links ngx_chain_t
are actively used in nginx,
nginx pool provides a way to reuse them.
The chain
field of ngx_pool_t
keeps a
list of previously allocated links ready for reuse. For efficient allocation of
a chain link in a pool, the function
ngx_alloc_chain_link(pool)
should be used.
This function looks up a free chain link in the pool list and only if it's
empty allocates a new one. To free a link ngx_free_chain(pool, cl)
should be called.
Cleanup handlers can be registered in a pool. Cleanup handler is a callback with an argument which is called when pool is destroyed. Pool is usually tied with a specific nginx object (like HTTP request) and destroyed in the end of that object’s lifetime, releasing the object itself. Registering a pool cleanup is a convenient way to release resources, close file descriptors or make final adjustments to shared data, associated with the main object.
A pool cleanup is registered by calling ngx_pool_cleanup_add(pool,
size)
which returns ngx_pool_cleanup_t
pointer to
be filled by the caller. The size
argument allows allocating
context for the cleanup handler.
ngx_pool_cleanup_t *cln; cln = ngx_pool_cleanup_add(pool, 0); if (cln == NULL) { /* error */ } cln->handler = ngx_my_cleanup; cln->data = “foo”; ... static void ngx_my_cleanup(void *data) { u_char *msg = data; ngx_do_smth(msg); }
Shared memory
Shared memory is used by nginx to share common data between processes.
Function ngx_shared_memory_add(cf, name, size, tag)
adds a
new shared memory entry ngx_shm_zone_t
to the cycle. The
function receives name
and size
of the
zone.
Each shared zone must have a unique name.
If a shared zone entry with the provided name exists, the old zone entry is
reused, if its tag value matches too.
Mismatched tag is considered an error.
Usually, the address of the module structure is passed as tag, making it
possible to reuse shared zones by name within one nginx module.
The shared memory entry structure ngx_shm_zone_t
has the
following fields:
-
init
- initialization callback, called after shared zone is mapped to actual memory -
data
- data context, used to pass arbitrary data to theinit
callback -
noreuse
- flag, disabling shared zone reuse from the old cycle -
tag
- shared zone tag -
shm
- platform-specific object of typengx_shm_t
, having at least the following fields:-
addr
- mapped shared memory address, initially NULL -
size
- shared memory size -
name
- shared memory name -
log
- shared memory log -
exists
- flag, showing that shared memory was inherited from the master process (Windows-specific)
-
Shared zone entries are mapped to actual memory in
ngx_init_cycle()
after configuration is parsed.
On POSIX systems, mmap()
syscall is used to create shared
anonymous mapping.
On Windows, CreateFileMapping()/MapViewOfFileEx()
pair is
used.
For allocating in shared memory, nginx provides slab pool
ngx_slab_pool_t
.
In each nginx shared zone, a slab pool is automatically created for allocating
memory in that zone.
The pool is located in the beginning of the shared zone and can be accessed by
the expression (ngx_slab_pool_t *) shm_zone->shm.addr
.
Allocation in shared zone is done by calling one of the functions
ngx_slab_alloc(pool, size)/ngx_slab_calloc(pool, size)
.
Memory is freed by calling ngx_slab_free(pool, p)
.
Slab pool divides all shared zone into pages. Each page is used for allocating objects of the same size. Only the sizes which are powers of 2, and not less than 8, are considered. Other sizes are rounded up to one of these values. For each page, a bitmask is kept, showing which blocks within that page are in use and which are free for allocation. For sizes greater than half-page (usually, 2048 bytes), allocation is done by entire pages.
To protect data in shared memory from concurrent access, mutex is available in
the mutex
field of ngx_slab_pool_t
.
The mutex is used by the slab pool while allocating and freeing memory.
However, it can be used to protect any other user data structures,
allocated in the shared zone.
Locking is done by calling
ngx_shmtx_lock(&shpool->mutex)
, unlocking is done by
calling ngx_shmtx_unlock(&shpool->mutex)
.
ngx_str_t name; ngx_foo_ctx_t *ctx; ngx_shm_zone_t *shm_zone; ngx_str_set(&name, "foo"); /* allocate shared zone context */ ctx = ngx_pcalloc(cf->pool, sizeof(ngx_foo_ctx_t)); if (ctx == NULL) { /* error */ } /* add an entry for 65k shared zone */ shm_zone = ngx_shared_memory_add(cf, &name, 65536, &ngx_foo_module); if (shm_zone == NULL) { /* error */ } /* register init callback and context */ shm_zone->init = ngx_foo_init_zone; shm_zone->data = ctx; ... static ngx_int_t ngx_foo_init_zone(ngx_shm_zone_t *shm_zone, void *data) { ngx_foo_ctx_t *octx = data; size_t len; ngx_foo_ctx_t *ctx; ngx_slab_pool_t *shpool; value = shm_zone->data; if (octx) { /* reusing a shared zone from old cycle */ ctx->value = octx->value; return NGX_OK; } shpool = (ngx_slab_pool_t *) shm_zone->shm.addr; if (shm_zone->shm.exists) { /* initialize shared zone context in Windows nginx worker */ ctx->value = shpool->data; return NGX_OK; } /* initialize shared zone */ ctx->value = ngx_slab_alloc(shpool, sizeof(ngx_uint_t)); if (ctx->value == NULL) { return NGX_ERROR; } shpool->data = ctx->value; return NGX_OK; }
Logging
For logging nginx code uses ngx_log_t
objects.
Nginx logger provides support for several types of output:
- stderr - logging to standard error output
- file - logging to file
- syslog - logging to syslog
- memory - logging to internal memory storage for development purposes. The memory could be accessed later with debugger
A logger instance may actually be a chain of loggers, linked to each other with
the next
field.
Each message is written to all loggers in chain.
Each logger has an error level which limits the messages written to that log. The following error levels are supported by nginx:
-
NGX_LOG_EMERG
-
NGX_LOG_ALERT
-
NGX_LOG_CRIT
-
NGX_LOG_ERR
-
NGX_LOG_WARN
-
NGX_LOG_NOTICE
-
NGX_LOG_INFO
-
NGX_LOG_DEBUG
For debug logging, debug mask is checked as well. The following debug masks exist:
-
NGX_LOG_DEBUG_CORE
-
NGX_LOG_DEBUG_ALLOC
-
NGX_LOG_DEBUG_MUTEX
-
NGX_LOG_DEBUG_EVENT
-
NGX_LOG_DEBUG_HTTP
-
NGX_LOG_DEBUG_MAIL
-
NGX_LOG_DEBUG_STREAM
Normally, loggers are created by existing nginx code from
error_log
directives and are available at nearly every stage
of processing in cycle, configuration, client connection and other objects.
Nginx provides the following logging macros:
-
ngx_log_error(level, log, err, fmt, ...)
- error logging -
ngx_log_debug0(level, log, err, fmt)
,ngx_log_debug1(level, log, err, fmt, arg1)
etc - debug logging, up to 8 formatting arguments are supported
A log message is formatted in a buffer of size
NGX_MAX_ERROR_STR
(currently, 2048 bytes) on stack.
The message is prepended with error level, process PID, connection id (stored
in log->connection
) and system error text.
For non-debug messages log->handler
is called as well to
prepend more specific information to the log message.
HTTP module sets ngx_http_log_error()
function as log
handler to log client and server addresses, current action (stored in
log->action
), client request line, server name etc.
Example:
/* specify what is currently done */ log->action = "sending mp4 to client”; /* error and debug log */ ngx_log_error(NGX_LOG_INFO, c->log, 0, "client prematurely closed connection”); ngx_log_debug2(NGX_LOG_DEBUG_HTTP, mp4->file.log, 0, "mp4 start:%ui, length:%ui”, mp4->start, mp4->length);
Logging result:
2016/09/16 22:08:52 [info] 17445#0: *1 client prematurely closed connection while sending mp4 to client, client: 127.0.0.1, server: , request: "GET /file.mp4 HTTP/1.1” 2016/09/16 23:28:33 [debug] 22140#0: *1 mp4 start:0, length:10000
Cycle
Cycle object keeps nginx runtime context, created from a specific
configuration.
The type of the cycle is ngx_cycle_t
.
Upon configuration reload a new cycle is created from the new version of nginx
configuration.
The old cycle is usually deleted after a new one is successfully created.
Currently active cycle is held in the ngx_cycle
global
variable and is inherited by newly started nginx workers.
A cycle is created by the function ngx_init_cycle()
.
The function receives the old cycle as the argument.
It's used to locate the configuration file and inherit as much resources as
possible from the old cycle to keep nginx running smoothly.
When nginx starts, a fake cycle called "init cycle" is created and is then
replaced by a normal cycle, built from configuration.
Some members of the cycle:
-
pool
- cycle pool. Created for each new cycle -
log
- cycle log. Initially, this log is inherited from the old cycle. After reading configuration, this member is set to point tonew_log
-
new_log
- cycle log, created by the configuration. It's affected by the root scopeerror_log
directive -
connections
,connections_n
- per-worker array of connections of typengx_connection_t
, created by the event module while initializing each nginx worker. The number of connections is set by theworker_connections
directive -
free_connections
,free_connections_n
- the and number of currently available connections. If no connections are available, nginx worker refuses to accept new clients -
files
,files_n
- array for mapping file descriptors to nginx connections. This mapping is used by the event modules, having theNGX_USE_FD_EVENT
flag (currently, it's poll and devpoll) -
conf_ctx
- array of core module configurations. The configurations are created and filled while reading nginx configuration files -
modules
,modules_n
- array of modulesngx_module_t
, both static and dynamic, loaded by current configuration -
listening
- array of listening objectsngx_listening_t
. Listening objects are normally added by the thelisten
directive of different modules which call thengx_create_listening()
function. Based on listening objects, listen sockets are created by nginx -
paths
- array of pathsngx_path_t
. Paths are added by calling the functionngx_add_path()
from modules which are going to operate on certain directories. These directories are created by nginx after reading configuration, if missing. Moreover, two handlers can be added for each path:- path loader - executed only once in 60 seconds after starting or reloading nginx. Normally, reads the directory and stores data in nginx shared memory. The handler is called from a dedicated nginx process "nginx cache loader"
- path manager - executed periodically. Normally, removes old files from the directory and reflects these changes in nginx memory. The handler is called from a dedicated nginx process "nginx cache manager"
-
open_files
- list ofngx_open_file_t
objects. An open file object is created by calling the functionngx_conf_open_file()
. After reading configuration nginx opens all files from theopen_files
list and stores file descriptors in thefd
field of each open file object. The files are opened in append mode and created if missing. The files from this list are reopened by nginx workers upon receiving the reopen signal (usually it'sUSR1
). In this case thefd
fields are changed to new descriptors. The open files are currently used for logging -
shared_memory
- list of shared memory zones, each added by calling thengx_shared_memory_add()
function. Shared zones are mapped to the same address range in all nginx processes and are used to share common data, for example HTTP cache in-memory tree
Buffer
For input/output operations, nginx provides the buffer type
ngx_buf_t
.
Normally, it's used to hold data to be written to a destination or read from a
source.
Buffer can reference data in memory and in file.
Technically it's possible that a buffer references both at the same time.
Memory for the buffer is allocated separately and is not related to the buffer
structure ngx_buf_t
.
The structure ngx_buf_t
has the following fields:
-
start
,end
- the boundaries of memory block, allocated for the buffer -
pos
,last
- memory buffer boundaries, normally a subrange ofstart
..end
-
file_pos
,file_last
- file buffer boundaries, these are offsets from the beginning of the file -
tag
- unique value, used to distinguish buffers, created by different nginx module, usually, for the purpose of buffer reuse -
file
- file object -
temporary
- flag, meaning that the buffer references writable memory -
memory
- flag, meaning that the buffer references read-only memory -
in_file
- flag, meaning that current buffer references data in a file -
flush
- flag, meaning that all data prior to this buffer should be flushed -
recycled
- flag, meaning that the buffer can be reused and should be consumed as soon as possible -
sync
- flag, meaning that the buffer carries no data or special signal likeflush
orlast_buf
. Normally, such buffers are considered an error by nginx. This flags allows skipping the error checks -
last_buf
- flag, meaning that current buffer is the last in output -
last_in_chain
- flag, meaning that there's no more data buffers in a (sub)request -
shadow
- reference to another buffer, related to the current buffer. Usually current buffer uses data from the shadow buffer. Once current buffer is consumed, the shadow buffer should normally also be marked as consumed -
last_shadow
- flag, meaning that current buffer is the last buffer, referencing a particular shadow buffer -
temp_file
- flag, meaning that the buffer is in a temporary file
For input and output buffers are linked in chains.
Chain is a sequence of chain links ngx_chain_t
, defined as
follows:
typedef struct ngx_chain_s ngx_chain_t; struct ngx_chain_s { ngx_buf_t *buf; ngx_chain_t *next; };
Each chain link keeps a reference to its buffer and a reference to the next chain link.
Example of using buffers and chains:
ngx_chain_t * ngx_get_my_chain(ngx_pool_t *pool) { ngx_buf_t *b; ngx_chain_t *out, *cl, **ll; /* first buf */ cl = ngx_alloc_chain_link(pool); if (cl == NULL) { /* error */ } b = ngx_calloc_buf(pool); if (b == NULL) { /* error */ } b->start = (u_char *) "foo"; b->pos = b->start; b->end = b->start + 3; b->last = b->end; b->memory = 1; /* read-only memory */ cl->buf = b; out = cl; ll = &cl->next; /* second buf */ cl = ngx_alloc_chain_link(pool); if (cl == NULL) { /* error */ } b = ngx_create_temp_buf(pool, 3); if (b == NULL) { /* error */ } b->last = ngx_cpymem(b->last, "foo", 3); cl->buf = b; cl->next = NULL; *ll = cl; return out; }
Networking
Connection
Connection type ngx_connection_t
is a wrapper around a
socket descriptor. Some of the structure fields are:
-
fd
- socket descriptor -
data
- arbitrary connection context. Normally, a pointer to a higher level object, built on top of the connection, like HTTP request or Stream session -
read
,write
- read and write events for the connection -
recv
,send
,recv_chain
,send_chain
- I/O operations for the connection -
pool
- connection pool -
log
- connection log -
sockaddr
,socklen
,addr_text
- remote socket address in binary and text forms -
local_sockaddr
,local_socklen
- local socket address in binary form. Initially, these fields are empty. Functionngx_connection_local_sockaddr()
should be used to get socket local address -
proxy_protocol_addr
,proxy_protocol_port
- PROXY protocol client address and port, if PROXY protocol is enabled for the connection -
ssl
- nginx connection SSL context -
reusable
- flag, meaning, that the connection is at the state, when it can be reused -
close
- flag, meaning, that the connection is being reused and should be closed
An nginx connection can transparently encapsulate SSL layer.
In this case the connection ssl
field holds a pointer to an
ngx_ssl_connection_t
structure, keeping all SSL-related data
for the connection, including SSL_CTX
and
SSL
.
The handlers recv
, send
,
recv_chain
, send_chain
are set as well to
SSL functions.
The number of connections per nginx worker is limited by the
worker_connections
value.
All connection structures are pre-created when a worker starts and stored in
the connections
field of the cycle object.
To reach out for a connection structure, ngx_get_connection(s,
log)
function is used.
The function receives a socket descriptor s
which needs to
be wrapped in a connection structure.
Since the number of connections per worker is limited, nginx provides a
way to grab connections which are currently in use.
To enable or disable reuse of a connection, function
ngx_reusable_connection(c, reusable)
is called.
Calling ngx_reusable_connection(c, 1)
sets the
reuse
flag of the connection structure and inserts the
connection in the reusable_connections_queue
of the cycle.
Whenever ngx_get_connection()
finds out there are no
available connections in the free_connections
list of the
cycle, it calls ngx_drain_connections()
to release a
specific number of reusable connections.
For each such connection, the close
flag is set and its read
handler is called which is supposed to free the connection by calling
ngx_close_connection(c)
and make it available for reuse.
To exit the state when a connection can be reused
ngx_reusable_connection(c, 0)
is called.
An example of reusable connections in nginx is HTTP client connections which
are marked as reusable until some data is received from the client.
Events
Event
Event object ngx_event_t
in nginx provides a way to be
notified of a specific event happening.
Some of the fields of the ngx_event_t
are:
-
data
- arbitrary event context, used in event handler, usually, a pointer to a connection, tied with the event -
handler
- callback function to be invoked when the event happens -
write
- flag, meaning that this is the write event. Used to distinguish between read and write events -
active
- flag, meaning that the event is registered for receiving I/O notifications, normally from notification mechanisms like epoll, kqueue, poll -
ready
- flag, meaning that the event has received an I/O notification -
delayed
- flag, meaning that I/O is delayed due to rate limiting -
timer
- Red-Black tree node for inserting the event into the timer tree -
timer_set
- flag, meaning that the event timer is set, but not yet expired -
timedout
- flag, meaning that the event timer has expired -
eof
- read event flag, meaning that the eof has happened while reading data -
pending_eof
- flag, meaning that the eof is pending on the socket, even though there may be some data available before it. The flag is delivered viaEPOLLRDHUP
epoll event orEV_EOF
kqueue flag -
error
- flag, meaning that an error has happened while reading (for read event) or writing (for write event) -
cancelable
- timer event flag, meaning that the event handler should be called while performing nginx worker graceful shutdown, event though event timeout has not yet expired. The flag provides a way to finalize certain activities, for example, flush log files -
posted
- flag, meaning that the event is posted to queue -
queue
- queue node for posting the event to a queue
I/O events
Each connection, received with the
ngx_get_connection()
call, has two events attached to it:
c->read
and c->write
.
These events are used to receive notifications about the socket being ready for
reading or writing.
All such events operate in Edge-Triggered mode, meaning that they only trigger
notifications when the state of the socket changes.
For example, doing a partial read on a socket will not make nginx deliver a
repeated read notification until more data arrive in the socket.
Even when the underlying I/O notification mechanism is essentially
Level-Triggered (poll, select etc), nginx will turn the notifications into
Edge-Triggered.
To make nginx event notifications consistent across all notifications systems
on different platforms, it's required, that the functions
ngx_handle_read_event(rev, flags)
and
ngx_handle_read_event(wev,flags)
are called after handling
an I/O socket notification or calling any I/O functions on that socket.
Normally, these functions are called once in the end of each read or write
event handler.
Timer events
An event can be set to notify a timeout expiration.
The function ngx_add_timer(ev, timer)
sets a timeout for an
event, ngx_del_timer(ev)
deletes a previously set timeout.
Timeouts currently set for all existing events, are kept in a global timeout
Red-Black tree ngx_event_timer_rbtree
.
The key in that tree has the type ngx_msec_t
and is the time
in milliseconds since the beginning of January 1, 1970 (modulus
ngx_msec_t
max value) at which the event should expire.
The tree structure provides fast inserting and deleting operations, as well as
accessing the nearest timeouts.
The latter is used by nginx to find out for how long to wait for I/O events
and for expiring timeout events afterwards.
Posted events
An event can be posted which means that its handler will be called at some
point later within the current event loop iteration.
Posting events is a good practice for simplifying code and escaping stack
overflows.
Posted events are held in a post queue.
The macro ngx_post_event(ev, q)
posts the event
ev
to the post queue q
.
Macro ngx_delete_posted_event(ev)
deletes the event
ev
from whatever queue it's currently posted.
Normally, events are posted to the ngx_posted_events
queue.
This queue is processed late in the event loop - after all I/O and timer
events are already handled.
The function ngx_event_process_posted()
is called to process
an event queue.
This function calls event handlers until the queue is not empty. This means
that a posted event handler can post more events to be processed within the
current event loop iteration.
Example:
void ngx_my_connection_read(ngx_connection_t *c) { ngx_event_t *rev; rev = c->read; ngx_add_timer(rev, 1000); rev->handler = ngx_my_read_handler; ngx_my_read(rev); } void ngx_my_read_handler(ngx_event_t *rev) { ssize_t n; ngx_connection_t *c; u_char buf[256]; if (rev->timedout) { /* timeout expired */ } c = rev->data; while (rev->ready) { n = c->recv(c, buf, sizeof(buf)); if (n == NGX_AGAIN) { break; } if (n == NGX_ERROR) { /* error */ } /* process buf */ } if (ngx_handle_read_event(rev, 0) != NGX_OK) { /* error */ } }
Event loop
All nginx processes which do I/O, have an event loop.
The only type of process which does not have I/O, is nginx master process which
spends most of its time in sigsuspend()
call waiting for
signals to arrive.
Event loop is implemented in ngx_process_events_and_timers()
function.
This function is called repeatedly until the process exits.
It has the following stages:
-
find nearest timeout by calling
ngx_event_find_timer()
. This function finds the leftmost timer tree node and returns the number of milliseconds until that node expires -
process I/O events by calling a handler, specific to event notification
mechanism, chosen by nginx configuration.
This handler waits for at least one I/O event to happen, but no longer, than
the nearest timeout.
For each read or write event which has happened, the
ready
flag is set and its handler is called. For Linux, normally, thengx_epoll_process_events()
handler is used which callsepoll_wait()
to wait for I/O events -
expire timers by calling
ngx_event_expire_timers()
. The timer tree is iterated from the leftmost element to the right until a not yet expired timeout is found. For each expired node thetimedout
event flag is set,timer_set
flag is reset, and the event handler is called -
process posted events by calling
ngx_event_process_posted()
. The function repeatedly removes the first element from the posted events queue and calls its handler until the queue gets empty
All nginx processes handle signals as well.
Signal handlers only set global variables which are checked after the
ngx_process_events_and_timers()
call.
Processes
There are several types of processes in nginx.
The type of current process is kept in the ngx_process
global variable:
-
NGX_PROCESS_MASTER
- the master process runs thengx_master_process_cycle()
function. Master process does not have any I/O and responds only to signals. It reads configuration, creates cycles, starts and controls child processes -
NGX_PROCESS_WORKER
- the worker process runs thengx_worker_process_cycle()
function. Worker processes are started by master and handle client connections. They also respond to signals and channel commands, sent from master -
NGX_PROCESS_SINGLE
- single process is the only type of processes which exist in themaster_process off
mode. The cycle function for this process isngx_single_process_cycle()
. This process creates cycles and handles client connections -
NGX_PROCESS_HELPER
- currently, there are two types of helper processes: cache manager and cache loader. Both of them share the same cycle functionngx_cache_manager_process_cycle()
.
All nginx processes handle the following signals:
-
NGX_SHUTDOWN_SIGNAL
(SIGQUIT
) - graceful shutdown. Upon receiving this signal master process sends shutdown signal to all child processes. When no child processes are left, master destroys cycle pool and exits. A worker process which received this signal, closes all listening sockets and waits until timeout tree becomes empty, then destroys cycle pool and exits. A cache manager process exits right after receiving this signal. The variablengx_quit
is set to one after receiving this signal and immediately reset after being processed. The variablengx_exiting
is set to one when worker process is in shutdown state -
NGX_TERMINATE_SIGNAL
(SIGTERM
) - terminate. Upon receiving this signal master process sends terminate signal to all child processes. If child processes do not exit in 1 second, they are killed with theSIGKILL
signal. When no child processes are left, master process destroys cycle pool and exits. A worker or cache manager process which received this signal destroys cycle pool and exits. The variablengx_terminate
is set to one after receiving this signal -
NGX_NOACCEPT_SIGNAL
(SIGWINCH
) - gracefully shut down worker processes -
NGX_RECONFIGURE_SIGNAL
(SIGHUP
) - reconfigure. Upon receiving this signal master process creates a new cycle from configuration file. If the new cycle was created successfully, the old cycle is deleted and new child processes are started. Meanwhile, the old processes receive the shutdown signal. In single-process mode, nginx creates a new cycle as well, but keeps the old one until all clients, tied to the old cycle, are gone. Worker and helper processes ignore this signal -
NGX_REOPEN_SIGNAL
(SIGUSR1
) - reopen files. Master process passes this signal to workers. Worker processes reopen allopen_files
from the cycle -
NGX_CHANGEBIN_SIGNAL
(SIGUSR2
) - change nginx binary. Master process starts a new nginx binary and passes there a list of all listen sockets. The list is passed in the environment variable"NGINX"
in text format, where descriptor numbers separated with semicolons. A new nginx instance reads that variable and adds the sockets to its init cycle. Other processes ignore this signal
While all nginx worker processes are able to receive and properly handle POSIX
signals, master process normally does not pass any signals to workers and
helpers with the standard kill()
syscall.
Instead, nginx uses inter-process channels which allow sending messages between
all nginx processes.
Currently, however, messages are only sent from master to its children.
Those messages carry the same signals.
The channels are socketpairs with their ends in different processes.
When running nginx binary, several values can be specified next to
-s
parameter.
Those values are stop
, quit
,
reopen
, reload
.
They are converted to signals NGX_TERMINATE_SIGNAL
,
NGX_SHUTDOWN_SIGNAL
, NGX_REOPEN_SIGNAL
and NGX_RECONFIGURE_SIGNAL
and sent to the nginx master
process, whose pid is read from nginx pid file.