The Double-Edged Sword of Kubernetes

Developing and operating distributed architectures is complex and tedious. Over the past decade, many orchstration technologies helped bring the development environment closer to the actual deployment system to simplify development.

As you’re probably aware, one of the later arrivals, Kubernetes (or K8s), emerged as the de facto standard, with most orchestration technology either created with Kubernetes at its core or using a K8s distribution (OpenShift, for example). Now, nearly every cloud provider has a fully managed Kubernetes cluster to offer. There are many reasons that Kubernetes won the orchestrator game; it’s a valuable tool for Dev, Ops, SREs and more for provisioning and managing their containerized infrastructure (and applications).

But be careful! While Kubernetes empowers Dev and Ops teams to deploy highly functional distributed applications, it’s not a panacea; and there are plenty of pitfalls.

What is Kubernetes?

So what exactly is Kubernetes? Kubernetes is an orchestration platform and an abstraction layer for containerized applications and services. As such, K8s manages and limits container-available resources on the physical machine, as well as managing deployment, (re)start, stop and scaling of service instances.

In addition to resource limitations, Kubernetes enables many services, such as a software-defined network (SDN), to remove the visibility of independent cluster nodes from the user and services. That said, to applications deployed into Kubernetes, the network seems like an actual physical network environment without NAT or firewalls – like a VPN tunnel between all cluster nodes.

Finally, Kubernetes provides a plug-in infrastructure to offer additional services like external load balancers, service discovery, certificate management and many more. Because the ecosystem offers almost any feature you can think of, K8s provides a holistic solution to operating your microservices-based architecture.

Sharpen your Sword

Kubernetes is a pretty extensive toolbox to ease many aspects of development, testing and operations. Given the ecosystem surrounding Kubernetes, most (services) wishes are only one or two commands away. This is possible due to tools such as Helm or via the many available Kubernetes operators, which are special applications to manage the installation, life cycle and operation of the service being deployed. For example, Instana can be deployed using a Kubernetes operator implementation. In that case, the single command pictured here is enough to get you started straight away.

Single command to install Instana using the operator

Furthermore, Kubernetes provides applications with multiple options for simple and sophisticated service discovery; no manual configuration of IP addresses here. Service discovery ranges from simple DNS entries with multiple assigned internal IPs, to directly accessing the K8s API to retrieve additional information (such as open ports) to custom discovery implementations offered as their own Kubernetes service.

Kubernetes provides simple load balancing and failover directly built on the service discovery functionality. Since the service instances are registered and unregistered automatically, K8s always has an up-to-date view of the deployment landscape and routes requests according to the current setup. This functionality provides autoscaling, too, by just registering new instances of the service into the service discovery capability. Similarly, health checks can automatically mark service instances as broken or failing, have them removed from distribution, automatically restart them and rejoin.

A Kubernetes deployment descriptor defining a container with readiness and health probes to automatically manage the container’s life cycle

But the greatest feature by far, in my opinion, is the abstraction of the actual underlying cloud provider, operating system, or on-premises environment. As a developer, I work with the same basic conditions as I would in production, just by using a development Kubernetes system, such as minikube, on my local machine. While in the past, when systems were very different, “Works on my machine!” was an infamous running gag, this actually is true with Kubernetes; it removes quite a few obstacles along the way.

The other massive benefit of the abstraction away from the fundamental runtime platform is how it simplifies multi-cloud environments. Even a hybrid setup of cloud and on-premises operations is simple. Except for some specific elements, like external load balancers providing externally reachable IP addresses for your ingress services, or disk space often available in multi-region, multi-speed, multi-type (SDD, HDD, NVME) setups specific to your provider, all resources are defined by Kubernetes itself. For the cloud-specific configurations, there’s often very simple changes in the deployment descriptors based on additional configuration.

The Other Side of the Blade

But where there is good, there is evil. Kubernetes is not immune – quite the opposite, actually. Kubernetes, due to all the possibilities and benefits, is pretty hard to grasp at first glance. For practically every feature, such as ingress, API gateway or deployment options, there is more than one tool. You have to decide which one to use – not an easy choice, especially when you’re just starting out with K8s.

This problem is amplified by the fact that Kubernetes, from the application’s view, is a black box. As long as stuff “Just Works,” everything’s great, but debugging issues can be a challenge. You’ll want a great observability tool in place to help out.

This brings us to the next problem, which, to be fair, is not specific to Kubernetes; distributed applications and architectures are complex. Debugging is difficult; through multiple services, instances, maybe even over cluster borders. There is also a lot more networking going on in between services that can fail, no matter if HTTP, gRPC or any other similar technology is used for interservice communication. All those elements increase the complexity, but the black box design of Kubernetes makes it exponentially worse.

And while, from the outside, Kubernetes looks like a simple service, it is not even close to being as simple as it looks. Any Kubernetes setup consists of at least seven or more services that need to play nicely together. When they stop working, good luck finding the issue.

The falsely simplified view of Kubernetes is amplified by how simple it is to set up a new hosted Kubernetes on cloud providers, such as Google, AWS or Azure.

The Kubernetes Architecture is not simple (Kubernetes Components, CC-BY 4.0)

Last but not least is the learning curve. The complexity of Kubernetes is a hard beast to tame at any time, but it’s especially difficult for someone new to the game.

Kubernetes’ learning curve is extremely steep, no two ways about it. There are a lot of new concepts and terms to learn. Often, terms are explained using other terms that also are unknown to newbies. It can feel a bit like reading a book about medicine if you’re not a physician. However, the documentation has made great strides over the last two years, specifically in the beginner’s section. There are also a bunch of really good tech talks, blog posts and examples out there. Don’t get discouraged; learning about K8s will help you build better software and can help you land a coveted position in engineering and DevOps, if you’re looking to advance your career.

The Double-Edged Sword

Whether you’re using Kubernetes or not, it is a trade-off to consider. The learning curve is steep, which means there are a lot of terms, architectural patterns, and entirely new concepts to learn before you know what to do. This process can be cumbersome, and it can even be tough to know where to start learning. If you stick with it and make your way through to a basic understanding of Kubernetes, additional concepts become easier to grasp.

From my perspective, learning Kubernetes also teaches a mandatory understanding of distributed architectures, as well as service health, scalability and fault tolerance, especially with regard to how those elements depend on each other – all of which are important skills to have, going forward.

So is Kubernetes the best thing since sliced bread? Well, yes and no. It is a great help, simplifies architectural decisions by providing existing patterns and eases development with its abstraction of the underlying infrastructure. But, that abstraction itself is like all abstractions – it comes at its own cost. Abstractions are normally black boxes, and can create blind spots while in development or in operation. Additional care needs to be taken when using a tool like Kubernetes.

For me, at least, Kubernetes was worth the effort it took to learn the technology and put it to use. In my experience, the advantages, such as easy multi-cloud deployments (or hybrid deployments with cloud and on-premises) outweigh the initial learning difficulties by an order of magnitude.

Chris Engelbert

Chris Engelbert a senior developer advocate at Instana, an IBM Company. Chris is an open source enthusiast and always interested in “Java Next”. In his spare time, he struggles with performance optimizations, the JVM internals or the garbage collector. He also firmly believes in Polyglot and is familiar with Java, Kotlin, Go, TypeScript and other languages.

Chris Engelbert has 1 posts and counting. See all posts by Chris Engelbert