Continuous Evolution: The CI/CD Story

Often, developers’ tools are as important as the software they’re building. Billions of dollars, millions of lines of code and thousands of engineers are involved in the development of the toolchain that enables developers to turn their ideas into programs that, in turn, will help them elevate their craft.

The present generation of container-based deployments and the future of Kubernetes-everywhere has transformed all parts of this developer toolchain. Software engineering teams have had to rethink each of these tools deployed as containers and orchestrated by Kubernetes. The driving forces are the ability to deploy these tools uniformly along with the core application and extract predictable (and high) reliability and performance.

DevOps Connect:DevSecOps @ RSAC 2022

Of the many tools that developer teams make use of, CI/CD tools are at the fulcrum of dev-and-deploy operations. The spectrum of these tools can be classified broadly into rudimentary ones (basic), purpose-built (specialized) and finally, modern CI/CD tools (cloud-native). Let’s take a look at a brief taxonomy of these tools to understand their evolution and present state.

Basic Tools

The inception of CI/CD was in cron jobs that were custom built by engineers who would copy binaries over to remote servers. The next step in this evolution, along with improvements in version control, was ‘build systems’ (read: makefiles). These systems were employed to build executable binaries on demand from centralized code repositories.

Today, even rudimentary CI/CD has come a long way from its start as simple shell scripts. Several workflow automation tools exist that help connect build and deploy processes. For engineers, the simplest to use are those which come integrated with source code management systems.

The most notable of these are GitHub Actions and GitLab Runner. Both are deeply integrated with version control systems and are very simple for those just getting started. Plus, there are no installation, setup, licensing or other non-technical barriers to entry for engineers.

Specialized Tools

Currently, engineering teams are spoiled for choice when it comes to the sheer number of CI/CD tools available. There is no excuse not to use one! It has become a central pillar of DevOps best practices and is very critical for achieving the velocity that engineering teams desire.

The past decade has seen a tremendous rise in the increase of CI/CD tools. Several niche features like SaaS/on-premises installation, reporting, analytics and integrations that enabled GitOps, ChatOps and the like, along with many customization options, help differentiate each offering.

No list can do justice to the sheer number of tools, but noteworthy examples are Jenkins, Travis and CircleCI. While Travis and CircleCI are SaaS frontrunners, Jenkins offers enterprise teams the ability to install a functional CI/CD server on-prem. Niche analytics and extensive reporting features differentiate each product. Any technical or usability features one tool lacks can easily be found in the larger community and integrated with other complementary tools.

The mainstream adoption of containers happened in parallel with the evolution of CI/CD tools. This concurrent evolution resulted in several new patterns by which to achieve CI/CD. Support for Docker became a distinguishing feature for many of these tools. Another interesting development was the native integration of deployment automation by all major infrastructure providers. This helped offer a simplified way to apply infrastructure-as-code practices to dev tools.

Cloud-Native Tools

Regardless of who operates the CI/CD tools, they all experience issues with regard to reliability, especially if they ran externally. To make CI/CD software run as reliably and consistently as business applications, engineering teams began to look for ways to containerize CI/CD tools and deploy them to Kubernetes infrastructure. Also, the pressure for package registries and container registries increased, and the ability for CI/CD tools to pull from common repositories became crucial in realizing cloud-native continuous delivery.

There are fewer cloud-native CI/CD tools, but Tekton is the frontrunner in this paradigm. ArgoCD and JenkinsX are two other tools that engineers have seen success with. Cloud-native CI/CD is most suitable for teams that are in pursuit of homogeneity and want the reliability of Kubernetes for their developer tooling, as well.

Suitability For Use With Kubernetes

For many engineering teams today, the ultimate goal is to enable CI/CD with remote endpoints as Kubernetes infrastructure.

With basic CI/CD tools, it can take some shoe-horning to get deployments working on Kubernetes. Engineering teams have to bear the full cognitive load of the complexity that comes with Kubernetes. This can be a barrier to entry for small projects and it’s also a process that does not scale well.

The more specialized tools come with containerization baked into their approach. This allows engineers to (re)use existing workflows and point the deployments to container runtimes within Kubernetes infrastructure.

Simplification by PaaS

Platform-as-a-Service (PaaS) tools, typically meant to simplify developer workflows, also end up simplifying other automation use cases. In particular, configuring CI/CD workflows becomes so much simpler using specific platforms, since build and deploy activities are converged into a single trigger. Several PaaS tools exist that simplify the CI/CD experience, including Heroku, OpenShift, and Cloud Foundry. Of these, OpenShift and Cloud Foundry are more flexible for engineering teams because they are largely extensible and have the ability to deploy to Kubernetes infrastructure.

The singular focus of these tech products/projects has been to reduce cycle time. They are meant to make it seamless for the engineer to iterate, get feedback fast and keep iterating – getting from source code to production – as quickly as possible. Continuous integration and continuous deployment software is a critical part of this experience. The velocity with which engineering teams can deliver software to market matters quite a lot. Each evolution of these developer tools helps to accelerate that even more.