Docker has been around for roughly a year,  it was open-sourced by the guys at dotCloud ( the PAAS provider)  and has since caused a revolution in the devops community. It has the backing of industry heavy weights such as Google, Rackspace and Redhat . In this blog post, I will attempt to give an initial look at monitoring docker, with a specific focus on system health. The technology stack under Docker is a large topic so I will not be going into much detail on setting up a docker environment. Before we dig right in let’s find out a bit about docker.

What is Docker

Docker is a technology that leverages LXC ( Linux containers). LXC uses cgroups and kernels namespace to isolate system resources. See the Wikipedia article on cgroups for a bit of background. Docker is easy to use and has a clean command-line interface bringing light weight virtualisation without the cost of a virtual machine.

Our set up

We’ll take a look at monitoring a mongodb cluster running in 3 docker containers. A 4th docker container is started and all it will do is run a few mongostat scripts for collecting mongodb metrics.

 

Each container is assigned a container id. We’ll need to know what this is to locate resource allocation metrics. To see the running containers type ‘sudo docker ps’

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 10d285d0c5c0 logscape/mongodb-cluster:latest bash 6 hours ago Up 6 hours 27017/tcp romantic_pare 93633c57b193 logscape/mongodb-cluster:latest mongod -f /data/mong 6 hours ago Up 6 hours 27017/tcp mm_cluster 343d798a2430 logscape/mongodb-cluster:latest mongod -f /data/mong 6 hours ago Up 6 hours 27017/tcp mm_cluster efb27dd1443a logscape/mongodb-cluster:latest mongod -f /data/mong 27 hours ago Up 6 hours 27017/tcp mm_cluster

When the containers run they immediately start a mongo instance within the cluster. The mongodb cluster consists of one primary node and two secondary nodes.

How to monitor container resource usage

Remember when I mentioned something about cgroups at the beginning? Well you can find all device metrics in the sys filesystem under the cgroup/docker folder.  Here’s an example

/sys/fs/cgroup/cpu/docker/*

Each docker container will have its cpu,device and memory statistics stored under a docker folder using the container’s id.  We’ll use a similar workflow to what we used in the article Unix Command Visualization and the  2 minute Analytics Workflow to import and visualise our data.

Getting the data into Logscape

Let’s work through one example. I’ll choose the cpu metrics since they are quite straight forward.

/sys/fs/cgroup/<FOLDER>/docker/<CONTAINER>/cpuacct.stat

The cpuacct.stat file contains two lines, the user and system cpu utilization. The metric used is jiffies which used to map to kernel scheduler tick. On modern motherboards this is no longer a direct mapping so it is only useful for showing relative usage on a system. The contents of the cpuacct.stat file for the container 10d285d0c5c0

user 320 system 254

To get the metrics in Logscape we need to create a command that will do the following 1.) Cat the contents of the cpuacct.stat     

cat /sys/fs/cgroup/*/docker/$containerID*/cpuacct.stat

2.) Label the metric 

 awk ‘{print “cmd:docker-stats metric:cpu containerID:’$containerID’ ” $0} ‘

3.) Append a timestamp: Adding a timestamp isn’t strictly necessary since Logscape will attach its own on receiving the data. To append a timestamp use the ts command  from moreutils or write a quick python or perl script to do this. I’, using the date-appender – script.

 ./date-appender.py -

4.) Combine with ‘|’ pipe symbol  Combine all these commands using the pipe symbol and pipe it all of to Logscape using netcat  <HOST> <PORT>

cat /sys/fs/cgroup/*/docker/$containerID*/cpuacct.stat |     awk ‘{print “cmd:docker-stats metric:cpu containerID:’$containerID’ ” $0} ‘|  ./date-appender.py –  | netcat 10.28.1.160 10000

The output from this command will look like this:

2014-06-25 15:15:19.266467 deathstar cmd:docker-stats metric:cpu containerID:10d285d0c5c0 user 320 2014-06-25 15:15:19.266536 deathstar cmd:docker-stats metric:cpu containerID:10d285d0c5c0 system 254 2014-06-25 15:15:19.284133 deathstar cmd:docker-stats metric:cpu containerID:93633c57b193 user 299034 2014-06-25 15:15:19.284199 deathstar cmd:docker-stats metric:cpu containerID:93633c57b193 system 6780

Construct a search string that will parse these lines  in Logscape use something like this.

split(\s+) | _tag.equals(socket-server)      include(total_rss,) 8.max(containerID,rss)    chart(line-zero)

The split command will tokenize the line on the spaces and each field will be labelled with a number. counting from the left the system metric value is in the 8th column. You will get a chart like this : You can then explore the rest of the sys fs and use a combination of shell scripts to collect block device and memory stats and ship those into Logscape Raw Socket Server.

Getting Mongostats into logscape

Once we have that in place we may want to see some mongo stats. That can be done quickly with this command.

mongostat -h <HOST> <sleep in seconds>  | awk ‘{print “cmd:mongostat $host”}’ |  ts  | netcat 10.28.1.160 10000 &

The awk command is important since it helps to distinguish the mongostats from other data being collected by the Raw Socket Server. Once we have some data we can chart the operations on our mongodb cluster. Using this search

split(\s+)  | _tag.equals(socket-server)  _filename.equals(unknown-14Jun25.log) cmd.contains(mongostats) 6.max(host,inserts) 7.max(host,queries) 8.max(host,deletes) bucketWidth(1m)  chart(pie) buckets(1)

we get a pie chart that looks like this

This was a quick rapid prototyping exercise into monitoring Docker. Docker container monitoring is large topic. In part II we’ll be looking at Docker container logs and how to effectively monitor these. Don’t forget to follow us on twitter for tips, videos and tutorials! Follow @logscape