Docker Patterns – The Volatile Configuration Pattern

It was about 25 years ago when the Gang of Four initially got together. From that confluence of vision came the notion of Object Oriented Design Techniques. If DevOps is about adopting the very best practices of developers and applying them to operations, then the time has come to apply the power of Design Patterns to containerized service development.

By taking a complicated idea and giving it a proper name, operations professionals are able to quickly communicate the ideas and implementations, “I have a volatile configuration problem, so I am going to use the StackEngine implementation.”

Compare that statement to the description and definition below, and realize that the words “Volatile Configuration Pattern” now become a proxy for the remainder of this the blog post.

Definition

The Volatile Configuration Pattern can be recognized by the need to rewrite configuration files for services that consume multiple backends. Its use is prescribed when such backends are often frequently started and stopped with different IP addresses or DNS/service names.

E.G. A load balancer with three web server backends must scale out to five backends during peak load times, and scale back to three when load is reduced.

E.G. A monitoring service must collect container and host information from multiple hosts that join and leave the resource pool as dictated by customer activity.

A Volatile Configuration Scenario

Docker zealots often cite scaling up and scaling down with ease as a great use case for adoption. While it is absolutely true that Docker makes this sort of scaling much easier, it is neither easy, nor obvious, how to capture magical scaling fairies and put them to work for you.

Consider a load balancer. Now consider a situation where back end web servers are coming and going frequently due to auto-scaling events. Both the load balancer and the web servers are Docker containers.

The load balancer keeps track of all its back ends in its configuration file. This file must change for every scaling event. Thus, the configuration file experiences a volatility generally not thought about in pre-container days.

What Tooling is Needed to Address this Issue

There are a number of things the load balancer service must know and be able to do. Below are the basic components and some example tooling. If your favorite tool is missing, don’t hesitate to add a comment!

Service Registry

A service registry (sometimes known as Service Discovery) stores a service name and the location of services within an application topology. For example a service name of nginx-80 might have five unique values that are IP address and ports. These are the locations of that service.

• consul – Consul is the most mature of the service registry solutions today. It comes with a DNS interface, service health checks and a key-value store.
• etcd – etcd is most often associated with CoreOS, but don’t be fooled. It is a standalone service as well. etcd expects to run on each host in a resource pool and this makes connecting to it from a container extremely easy. etcd also has a well designed CLI which means scripting against it is a snap.
• stackengine – StackEngine is a commercial orchestration product with service registry functionality built-in. This gives it the advantages of etcd in making it easy for containers to find. Further it means the tool doing container scheduling is also tracking available services.

Automatic Service Registration

When talking about volatile services, the idea that they are registered and de-registered manually is out of the question. This does not scale, and it is error prone. On the other hand, building registration functionality into a service is often not possible, or is simply too complex.

• registrator – Registrator is a container image by GliderLabs that watches the Docker socket for container start events. When registrator sees a published port, it makes an entry in the service registry layer (e.g. consul, etcd, zookeeper, et al). Because it listens to a socket it must be deployed on every host.
• stackengine – As part of its service registry, StackEngine offers automatic service registration when a port is explicitly published.
• docker-gen – Docker-gen is a tool that reads container metadata and writes configuration files. Unlike registrator and StackEngine, it is meant to provide more granular control for building services. Like Registrator it must run on each host.

Automatic Configuration File Rewriting

This process generally runs along side the backend process. This could be done as a side-kick container, however this seems more complex and more prone to error. Instead this process with run next to the backend process in the same container (e.g. nginx and confd running in the same container).

The automatic configuration rewriting process watches the service registry for changes to a specified key. When such a change occurs, it rewrites the configuration file and reloads the process.

• consul-template – Works only with Consul, but provides a simple and seamless way to rewrite a configuration template
• confd – Works with most backends. A maturing project that works well. Could use a bit more verbosity in debugging situations.

Container Init System

By now it should be obvious that two processes will need to run. The actual desired service (load balancer) and the config rewriting service. Docker does not support such a model directly, so we will need to put an init system in the container to manage both processes.

• supervisord – Supervisord is very popular because of its ease of use and simple DSL. Its downside is specific to container size. You will need 100mb of python and dependencies. This makes the container very large. When developing containers based on a supervisord base, this also means the need to work directly on a file, thus breaking the Docker inheritance model.
• runit – runit is very small (1mb) and very fast. It works by dropping simple bash scripts into a service directory. This means that when extending a container a new file need only land in a new place. This works will with Docker’s inheritance model. Runit is a very mature project, but the documentation is still pretty sparse.

Current Implementations of the Volatile Configuration Pattern

There are a number of examples of this pattern in the wild. A couple that served as inspirations for this article and many of the useful services at Stackhub. Paul Czarkowski’s work on Faktorish inspired the idea of a base image to build your own services from this pattern.

Jason Wilder – jwilder/nginx-proxy

Jason’s containers are wildly popular (pun intended), but jwilder/nginx-proxy is the first magic container I encountered when starting my own Docker journey. It is ideal for using nginx to serve static content while reverse-proxying to a backend service like php-fpm or gunicorn.

It works by using Docker-gen (so must be on the same host as the backing services) to watch the Docker socket file for containers starting with the environment variable VIRTUAL_HOST. Upon seeing such a start, the nginx-proxy container will rewrite the nginx config on the fly and reload the nginx service.

It’s worth pointing out here that the environment variables are serving as the service registry layer and auto registration. While this is not ideal, it is sometime just elegantly simple and worth understanding.

Paul Czarkowski – paulczar/percona-galera

As a MySQL DBA it was natural for me to go looking for a Docker image that solved the replication problem in an elegant way. The paulczar/percona-galera project is a great example of dealing with volatile configuration files. The image uses confd, etcd and Percona MySQL to set up a database replication topology.

Paul’s container runs on CoreOS so that it can make the assumption that etcd is available locally, or that you must pass it an etcd host environment variable.

StackEngine – stackhub/base-confd

Thinking hard about Paul and Jason’s examples led me to realize there is a general solution for rewriting a configuration file given backend events. Again Czarkowski put ideas in my head during a talk at the Docker Austin Meet Up with his work on Faktorish which is a realistic take on the purity of Heroku’s 12 Factor App.

In Faktorish, Paul argues that many existing applications will run effectively in a Docker container without being rewritten into a set of micro-services. If you are trying to build a container around a legacy application, it is worth the read. At the Continuous Delivery Summit in Austin this year, Jez Humble mentioned the Strangler Pattern. Coupling Faktorish with Strangler answers the question of, “How do I Sanely Adopt Docker?”

Ultimately, Faktor III is where the thinking about backing services becomes clear. What stackhub/base-confd does is make it simple for anyone to use The Volatile Configuration pattern.

Based on gliderlabs/alpine, the image provides a way to build very small (30mb) images by simply:

• Assuming the StackEngine ecosystem for Service Registry and automated service registration
• apk add the desired service (e.g. haproxy) or curl | bash (yeah, we know)
• creating a couple of configuration file templates

Really, that’s it. Check out the README for details on how to build your own. The examples below are live.

stackhub/haproxy

The haproxy service will recognize new backends and scale out accordingly. Conversely it will adjust when backends are killed during scale back events.

stackhub/nginx-lb

The ‘nginx-lb’ service works the same way as haproxy, but it uses nginx as the load balancer.

stackhub/prometheus

The prometheus service demonstrates how easy it is to generalize this pattern. The google/cadvisor container is used to collect container and host metrics on each node. Prometheus reads these metrics and aggregates them so the can be visualized by tools such at Prometheus Dashboard, Grafana and Graphite.

Conclusion

When working with common services that need to be aware of ever changing backend service availability, consider applying The Volatile Configuration Pattern. stackhub/base-confd provides a simple, small and easily extensible way for you to implement this pattern for your own situations.

Use it, fork it, contribute bugs, comments and code. Go build something spiffy.