Kubernetes has solidified itself as a top container orchestrator tool. And use only continues to expand—a 2020 CNCF survey found that 92% of cloud-native developers have used Kubernetes in production. That’s up 300% from 2016. The use of Kubernetes as a platform of choice for container orchestration is no longer a question. It has brought significant resource optimization and deployment benefits to most enterprise workloads.
Yet, this ubiquity is bringing some unintended consequences. The cloud-native market has become permeated with Kubernetes tooling options, varying Kubernetes control layers, new gateways and different ways to host and manage Kubernetes in production. All these options are culminating in a Kubernetes sprawl, says Reza Shafii, vice president of products, Kong Inc.
I recently met with Shafii to discuss the impending Kubernetes sprawl. According to Shafii, when any technology becomes abundant, it’s common for the excitement around it to lead to similar sprawl issues. We’ve witnessed similar ‘sprawls’ around cloud, VMs, containers and APIs. Below, we’ll consider what has given rise to Kubernetes sprawl and see what cloud-native operators should keep in mind as they navigate this space.
What Is Sprawl?
First off, what exactly do we mean by ‘sprawl’? Techopedia defines application sprawl as “the growth of an IT system to include more applications, and to use more resources overall.” Unbridled tech sprawl can lead to complex, unmaintainable software. Tooling complexity can consume more resources and lead to differing approaches, causing confusion when making architectural decisions. Sprawl could also cause technical debt and decreased efficiency.
What Is Kubernetes Sprawl?
If we look at the Kubernetes space, similar currents are occurring, resulting in sprawl. “The variety of Kubernetes offerings available (across cloud providers and private or so-called ‘hybrid’ offerings) has only increased,” explains Shafii. Kubernetes environments are inherently complex. As a result, 78% of developers say Kubernetes complexity is a source of pain in their daily work, a recent D2iQ study finds.
Reasons Behind Kubernetes Sprawl
This pain could be a result of numerous factors. One is new multi-cluster management concerns. A single Kubernetes cluster currently supports an impressive 5000 nodes. As if one cluster with thousands of nodes wasn’t complex enough to maintain, companies are now beginning to support multiple clusters. Creating multiple clusters may be necessary to run K8s in different data centers, separate them by country restrictions or to meet privacy limitations. As the number of clusters grows, so does complexity, and along with it, various approaches and tooling arise to respond to new maintenance hurdles.
According to Shafii, the multi-cluster management concerns are a clear indicator there is a K8s sprawl. “With the ease of deployment of Kubernetes clusters also decreasing, the stage is well set for ‘sprawl’ problems.” Maintaining clusters across teams and divisions can also complicate the sprawl.
“Agility is the reason behind the sprawl,” says Shafii. And to deliver that agility, engineers often introduce tools and libraries in the moment. This tension between standardization and “Let’s get sh** done” could bring about differing approaches, explains Shafii.
We’ve also seen an explosion of different ways to run Kubernetes. Of course, teams might opt for a bare-bones approach using only the open-source software itself. But more often than not, they use managed Kubernetes services. Nearly 90% of Kubernetes users leverage cloud-managed services such as Amazon Elastic Kubernetes Service (EKS), Google Kubernetes Engine (GKE) and Azure Kubernetes Service (AKS), Datadog reports. Though Kubernetes can help enable the hybrid multi-cloud revolution, managing it across various clouds could be a maintenance challenge.
K8s is taking over—and still has plenty of room to grow. It has a thriving community—over 4200 companies have contributed to building Kubernetes to date. It’s the ideal scheduler to perfect the Tetris game of container orchestration, says Shafii. It’s still doing that, but the “conversation has moved toward framing it as a platform for standardizing deployment,” he explains.
While there are certainly Ops efficiency benefits to standardization, unnecessary tech adoption can lead to sprawl. People are assuming they must use K8s at all times, but not all cases warrant its use. For example, launching satellites to space with two onboard processors probably doesn’t require a local Kubernetes installation, says Shafii.
Drawbacks of a Kubernetes Sprawl
If there is a Kubernetes sprawl, teams will likely suffer the same issues that occur with other technology sprawls. For example, inefficiencies at scale could make it so K8s can’t be operated consistently across the board. Other potential outcomes include:
- As complexity increases, visibility shrinks. A high number of clusters can be challenging to observe while ensuring security. It can be difficult to track efficiency issues and resource optimizations, which could harm performance.
- Higher latencies: Many providers are offering inter-Kubernetes gateways and service mesh options. Cloud-native designs may often end up throwing one gateway in front of another previously built gateway, explains Shafii. This could add additional latencies, especially when legacy gateways are used.
- Poor developer experience. Kubernetes sprawl could easily result in poor developer experience. “Developers often end up getting dragged into this,” Shafii notes. For smaller teams with no DevOps division, K8s sprawl could take time away from application development.
- Persistence and networking issues. Most applications must access a persistent storage database. Yet, Kubernetes isn’t designed for stateful conditions, requiring some workarounds. “As 95% of K8s problems are networking problems,” says Shafii, “Persistence and networking are going to be the first problems to solve.”
- Lack of alignment. An organization with a large footprint could suffer from inconsistent K8s implementations. Teams may have coded a preferred K8s approach into their platform, which could inhibit alignment across an organization.
History Repeats Itself
An article about technology sprawl is by no means unique. Other analysts have pointed to similar sprawl challenges with previous software trends, like virtual machines (VMs). “Container sprawl is the new VM sprawl,” wrote Tobi Knaup in 2020. And while we’re arguably still in that phase, today’s cloud-native sprawl exists at the container orchestration layer, as well.
“History is repeating itself in so many ways,” said Shafii. Sprawl issues of the same variety have arisen in the past, around multiple cloud accounts and, before that, VMs in the virtualized infrastructure world. He points to OS sprawl, API sprawl and application server sprawl as other previous (and ongoing) trends. Therefore, we can look to these trends to consider the repercussions of sprawl in the era of Kubernetes.
“This will make the ability to manage the life cycle and applications on these differing Kubernetes providers in a consistent manner, as well as the ability to consolidate them easily, more and more critical,” says Shafii.