Machine learning and containers

Machine Learning (ML) and Artificial Intelligence (AI) are really hot topics now in the IT industry. As are containers. In this blog I try to draw both into the same picture and see if there is any synergy.

Goal

Although pure experimentation is always fun, it’s maybe a bit more focused when there’s at least some type of goal set. For my experiments I set the following goals:

  • Learn what ML is and TensorFlow in general
  • See if there is any synergy between ML and containers
  • Deploy a running ML solution on Kontena

During the journey I added one extra “stretch” goal, to learn a bit of Go. More on that later. :)
What I envisioned as the end goal is something like this:
The big idea is threefold:

  1. Have a simple to use API where users could send a jpg picture for classification
  2. Run the ML model on multiple instances so that we can scale up the processing if needed
  3. Follow microservice patterns

So the journey begins…

All the code is available at https://github.com/jnummelin/tensorflow-inception-example.


The Tools

TensorFlow

TensorFlow is an open source software library for numerical computation using data flow graphs. The graph nodes represent mathematical operations, while the graph edges represent the multidimensional data arrays (tensors) that flow between them. This flexible architecture lets you deploy computation to one or more CPUs or GPUs on a desktop, server, or mobile device without rewriting code.

Very much simplified, you use TensorFlow to train a computer model with a set of training data. Once the model is trained it can be used to analyse as yet unknown data. The analysis could be for example image classification, as in my adventure here. In general the model can predict how well the input data matches some “known” patterns in the trained models.

In this blog, I won’t go deep into how to train the models since that requires deeper understanding of the concepts of machine learning in general and also deep knowledge of TensorFlow. The TensorFlow folks have prepared a good tutorial on model training, you should definitely check that out. I also recommend checking a longer introduction about how HBO’s Silicon Valley built their famous hotdog or not-hotdog mobile app. It’s both hilarious and highly educational. :)

One of the cool things about TensorFlow models is that once the model is built, it can be used pretty easily without any heavy backend servers needed. As they’ve shown with the hotdog or not app. In that case the model itself “runs” on mobile devices.

TensorFlow models and containers

One of the goals for the experiments was to find out if there are any synergies between machine learning and containers. Turns out there actually is, at least from my perspective.

TensorFlow allows one to export a pre-trained model to be used later on elsewhere. This allows one to even use the ML model on a mobile device to see if a picture contains a hotdog or not. :) This also makes containers truly a great vessel to ship and run your machine learning models.

One seemingly good way to utilise containers is with Docker’s new multi-stage builds.

FROM bitnami/tensorflow-inception:latest as model-builder  
RUN mkdir -p /model-data/ && \  
    curl -o '/model-data/inception-v3-2016-03-01.tar.gz' 'http://download.tensorflow.org/models/image/imagenet/inception-v3-2016-03-01.tar.gz' && \
    cd /model-data && tar zxf inception-v3-2016-03-01.tar.gz
RUN inception_saved_model --checkpoint_dir=/model-data/inception-v3 --output_dir=/model-data/inception-export

FROM bitnami/tensorflow-serving:latest  
COPY --from=model-builder /model-data/inception-export/* /bitnami/model-data/1  

The first step, named model-builder downloads a pre-trained model checkpoint. It then goes and exports the model to be usable by the TensorFlow Serving system.

The second step copies the prepared model data from step 1 to the image to be served by TensorFlow Serving. So the final output is a Docker image which contains everything pre-packaged so we can serve our machine learning model with a single docker run ... command. If that’s not a good synergy, then nothing is. From a machine learning newbie point of view it sounds pretty awesome to be able to run machine learning with a single command.

I’m using ready-made base images as the starting point just to save some effort in installing the TensorFlow packages. The sources for those is available at https://github.com/bitnami/bitnami-docker-tensorflow-serving and https://github.com/bitnami/bitnami-docker-tensorflow-inception.

The API

TensorFlow Serving serves the model using grpc API. Due to the complex nature of machine learning in general, do’h :), the API is somewhat complex as well. At least it’s not really suitable for any random client program to easily send a jpg image for classification. Using the grpc API would mean compiling protobuf IDL’s and making complex requests. So I thought the solution really needed a more suitable API where one could just POST an image, maybe through a web page, and get the classification results.

As mentioned, during my journey I ended up adding a new goal: to learn a bit of Go. Go came into the list of goals with the API as it seemed fairly straight-forward to write an API that receives a jpg image and calls the TensorFlow Serving grpc API with it to classify it. Well, as always, theory and practice are two different things. The API itself is actually really simple to get up-and-running. The difficulties came only with the generated code for grpc protocol buffers. It seems there’s some issues with the protocol -> Go transformation handling for multiple packages. As I’m really a newbie on all things Go I ended up “fixing” some of the package imports in the generated code with a quick search-and-replace. One shouldn’t really ever modify generated code, but I just didn’t want to get stuck on this.

So basically the API just takes in a jpg file, transforms that to a grpc request for TensorFlow Serving and replies back with the given classification results, in json of course.

Running the model and API

Once everything is in container images, it’s of course pretty trivial to deploy it all on any container orchestration system. Surprise surprise, I’m gonna use Kontena as the deployment target.

The most complex part of the solution is the machine learning model, but now when even that is running as a self-contained container, things become really simple:

stack: jnummelin/tensorflow  
version: 1.3.0  
description: Tensorflow inception service  
services:  
  api:
    image: jnummelin/tensor-inception:latest
    environment:
      TF_ADDRESS: "serving:9000"
    ports:
      - "8080:8080"
  serving:
    # Model pre-imported into the image
    image: jnummelin/tensorflow-serving-inception:latest

I’ve omitted the loadbalancer config in this example. See the GitHub repository for more detailed deployment.

Testing

Now with the simplified API in front of the TensorFlow model, it’s easy to test out image classification using plain curl:

$ curl -s -XPOST -F "file=@/Users/jussi/Downloads/cropped_panda.jpg" image-classifier.kontena.works/classify | jq .
[
  {
    "Class": "giant panda, panda, panda bear, coon bear, Ailuropoda melanoleuca",
    "Score": 9.546637
  },
  {
    "Class": "indri, indris, Indri indri, Indri brevicaudatus",
    "Score": 6.6261067
  },
  {
    "Class": "gibbon, Hylobates lar",
    "Score": 4.3301826
  },
  {
    "Class": "lesser panda, red panda, panda, bear cat, cat bear, Ailurus fulgens",
    "Score": 4.0944114
  },
  {
    "Class": "titi, titi monkey",
    "Score": 2.8160584
  }
]

The higher the score is, the better. So yes, our ML model was able to figure out that the pic was about a panda. For reference, this is the image posted:

Wonder how well this setup copes with hotdog images? :D

$ curl -s -XPOST -F "file=@/Users/jussi/Downloads/hot-dog-with-mustard.jpg" image-classifier.kontena.works/classify | jq --tab .
[
{
  "Class": "hotdog, hot dog, red hot",
  "Score": 11.738452
},
{
  "Class": "bell pepper",
  "Score": 3.9924777
},
{
  "Class": "great grey owl, great gray owl, Strix nebulosa",
  "Score": 3.7373521
},
{
  "Class": "chiton, coat-of-mail shell, sea cradle, polyplacophore",
  "Score": 2.9231932
},
{
  "Class": "balance beam, beam",
  "Score": 2.4462078
}
]

Seems pretty good, the hotdog class received a good score when compared to any other category.


Summary

Using TensorFlow models with containers does provide a very nice way to deploy them. By using the architectural pattern shown in the examples, it’s pretty easy to set up a scalable solution to serve basically any TensorFlow model. But using the models with any client software clearly needs some kind of API wrapper, making each client deal with the TensorFlow gRPC complexity is something that I would not want to do at least. 

What next?

Using pre-created models is of course not the reality in many cases. As with any learning, it’s a process that needs feedback which amplifies the learning and produces more and more accurate results. I’m thinking to expand my approach in the future by setting up a constant model trainer where you could push back the results. Users could select in some web UI for example which class was the correct one, or even post new classes. This would feed the information to something that builds the model constantly. That something could also export the model periodically and thus trigger a new build for the model container. It would be fairly straightforward, the infamous last words before trouble, to build full scale automation that puts new ML models into use as they learn more and more. Let me know what you think!


Image credits: Abysmal / Void by Ars Electronica

About Kontena

Kontena provides the most easy-to-use, fully integrated solution for DevOps and software development teams to deploy, run, monitor and operate containers on the cloud. The underlying Kontena Platform technology is open source and available under Apache 2.0 license. It is used by hundreds of startups and software development teams working for some of the biggest enterprises in the world. www.kontena.io