Go to the profile of Sergio Sola
Sergio Sola
Senior Software Engineer at HelloFresh.com

Building Microservices with Python , Part I

Currently I am working in my current job as a Software Engineer at HelloFresh on the DataWarehouse Team. I am working on data pipelines and building tools around our infrastructure, like Documentr.

But at the end, I am a Software Engineer interested in many different areas. This post it is going to be focus on Backend Development and how I am building microservices on a personal project I am working.

You can find all the code in this repo: https://github.com/ssola/python-flask-microservice

Purpose

Nowadays it is a common practice to work in smaller applications, sharing the responsibility among many different services. I believe it is critical to have some standard tools between the teams working solving those problems.

In my opinion a Microservices should have some basic features:

  • Easy to start coding your logic, stop worrying about tools/patterns.
  • Documentation, an essential feature to share how your service it is going to work. In this case, Swagger works pretty well.
  • Serializing your input/output in a way shared among all applications. You need to chose a technology like Avro/Protobuf. This is mandatory to be sure all services are sharing the same entities.
  • Events, probably your application it is going to generate events that can be consumed by others. For instance, in my project, every time a new room is inserted into the system, that piece of information is serialized with avro and spread among all the other services.

Stack and Patterns

In this basic setup we are going to include those packages:

  • Flask (as a Framework)
  • connexion (helpful tool to generate routes and Swagger docs)
  • Flask-Injector (Dependency Injection package)
  • Avro (or any data serialization package)

In this case, I chose Flask because I find it super useful to build small services without all the learning curve of Django. I just need something to help to do the routing and I will include whatever I need on it.

Connexion is a project from Zalando that adds a layer on top of Flask to help you building your RESTFul API in a simpler manner, with the great benefit to have at the end Swagger docs. Connexion gives you as well an elegant solution to protect your service behind oAuth2 and a way to versioning your API.

Dependency Injection is a nice way to inject the dependencies your need in your methods/classes. For this purpose, I chose Flask-Injector, as you can see by the name it’s completely integrated on Flask. With this tool using the decorator @inject I can have the service I need, for instance, ElasticSearch or SQLAlchemy.

Did you have problems with services sending corrupted data or some payload with the wrong schema? Well, that can be solved using some Serialization tool like Avro or Protobuf. These tools help you to ensure whatever you are receiving or sending has the proper schema.

I am not covering all the topics related to Microservices, but this is a good starting point, and you are covering some of the important pieces of the puzzle.

Building your more than a Hello World service

Starting your environment

I am going to specify a step-by-step guide with al the steps you need to do to have a simple service up-and-running on your machine.

First of all, create a new virtual environment with Python 3.6.0. If you are not used to working with virtual environments in Python, stop reading and jump to this other article.

Then create a new project on your favorite IDE, I like to work with PyCharm. In the root of your project create a new requirements.txt file with these dependencies:

Flask
connexion
Flask-Injector
fastavro

Starting with Connexion

After installing the dependencies, we can start coding our app.py script on the same path we placed the requirements file.

As you can see, we are already using the package connexion. It is pretty simple what it is doing there.

  1. Creates a new application defining to which port it will start (later on we can change which server we want to use to execute the service)
  2. Defines to which directory your swagger configurations are placed.
  3. You add a new api (you can have more than one in the same app), mainly the yaml file where you are going to write your routes and definitions.

At this point, we need to read a little bit how the connexion yaml file works.

Let’s create our first endpoint on the swagger/my_super_app.yaml file:

In this example, you can start understanding some features of connexion.

  1. We have a get endpoint on the /v1.0/items/ path.
  2. When this endpoit it’s called it will execute the method search on the module api.items
  3. This module is exposed on the __init__.py file in the api module.
  4. This endpoint will return an array of objects of type Item , we defined this item as an object with an id and a name

Here you can see the code on the api.items module:

Executing this endpoint, it will return a json like this:

{
    "0": {
        "name": "First item"
    }
}

And going to your http://localhost:9090/ui you will find the Swagger docs.

Swagger docs

Without that much effort, we have an up-and-running simple API returning items. Obviously, this is a silly example. So the next step is to make use of injection to inject an ItemProvider in our search method.

Working with Flask Injector

Dependency Injection it is a great pattern to help us to organize our dependencies among objects.

In software engineering, dependency injection is a technique whereby one object supplies the dependencies of another object. A dependency is an object that can be used (a service). An injection is the passing of a dependency to a dependent object (a client) that would use it. The service is made part of the client’s state.[1] Passing the service to the client, rather than allowing a client to build or find the service, is the fundamental requirement of the pattern.

In our previous example, we have in the very same file a list of Items , as you can imagine that is not the way we should retrieve data. We can connect to databases to get this information. To achieve that, we can use the Repository Pattern or create a Data Service to make this example simple enough; we are going to create a Data Service where we can retrieve some data.

This simple example can be easily extended to get the data from any database or third party API.

Now, let’s bind this service and inject it in our search method. First of all, we need to do some changes on our app.py , we need to add the bindings on the Flask Injector.

Here, as you can see, we have a new method called configure , this will handle all the bindings to create the container. In this case, we are creating a new ItemsProvider with some default items.

Now, in our api.items we can inject this ItemsProvider.

This is great as well for testing. When we need to test our search method we can create new bindings to a mocked ItemsProvider. We are simplifying our dependencies and improving our code for testing purposes.

As I said, this is not a complicated example. But probably you can see now how you can inject SQLAlchemy into your ItemsProvider to get dat from a database.

Serializing your payloads

A common pitfall building services are sharing wrong payloads. For instance, you expect an Item object with a name and an id. But working with plain JSON is not possible to enforce it.

We can do it right now with your connexion yaml, it will throw exceptions if something is wrong. But in this case, we need to ensure the data between different services using for instance RabbitMQ.

We can use JSON as I said, but then for each application, you will need to have all the required validations. If your platform works with several programming languages that can be a hassle.

With some tools like Avro or Protobuf, we can have a common serialization objects among different programming languages. A good practice would be to have a repo where you have all your Avro/Protobuf serializers. Those can be used almost for any programming language.

In this case, Avro gives us:

  • Rich data structures.
  • A compact, fast, binary data format.
  • A container file, to store persistent data.
  • Remote procedure call (RPC).
  • Simple integration with dynamic languages. Code generation is not required to read or write data files nor to use or implement RPC protocols. Code generation as an optional optimization, only worth implementing for statically typed languages.

A simple way to work with Avro and Python is using this library called fastavro.

You can find many examples in the GitHub repo.

Conclusion

With these small steps we can have a microservice up-and-running pretty quickly. With some nice features as:

  • Documentation
  • Dependency Injection
  • Global serializer

As you can imagine, this is not everything we need to do to have a production ready microservice. Here we have the steps to start building a solid skeleton.

In the next post I am going to write about other important features on microservices, like:

  • Logging & Metrics
  • CI/CD
  • Service Discovery
  • Dev environment with Docker

Please, let me know with your comments if you find any flaw or you have better ideas about how to improve those ideas.

You can continue reading part 2 here.