Optimizing Developer Impact on K8s Security

Kubernetes just passed its eighth birthday (hope you didn’t forget to send a card). In that time, the DevSecOps movement has decisively shifted security left. The evolution continues to add work to a software engineer’s plate—a plate that is already heaped with Kubernetes complexity. While the now-de facto choice for container orchestration has matured into an enterprise-grade solution, Kubernetes security controls still often boil down to preventing developers from direct access to production clusters or namespaces. In the early days of Kubernetes, development teams commonly operated Kubernetes clusters on their own. This practice was replaced as Kubernetes workloads became more critical; developers now generally own workload security while platform engineers take responsibility for securing the Kubernetes platform. 

However, the shift left movement has created a new attack surface and security category that lies between Kubernetes workloads and the cluster. Continuous delivery pipelines and GitOps models that deploy workloads to clusters can serve up these intermediate targets for attackers. You needn’t look further than the SolarWinds attack as Exhibit A.

That attack spurred wider adoption of chain-of-custody practices, including container signing and projects such as Cosign. A similar attack surface falling between developer and platform engineer responsibilities also emerges when running a service mesh, which continues to increase in popularity. In scenarios like that, the question of who owns network security can be answered by giving developers responsibility for configurations by writing custom resource definitions (CRDs) to configure service meshes, while platform engineers ensure the stability and security of the service mesh itself. 

Developers, Know Your Dependencies

Modern applications rely on a multitude of third-party open source dependencies to deliver their functionality. While that’s great for developing modern applications, it also complicates workload security. Organizations often don’t have deep in-house expertise in those open source technologies. Transitive dependencies then add to this complexity, since major open source projects may themselves be built upon tens or hundreds of open source dependencies.

To address this issue, developers are taking a cue from the manufacturing industry and embracing the software bill of materials (SBOM) movement, cataloging dependencies and using tooling that links dependencies to known vulnerabilities. This strategy offers advantages by enabling developers to recognize, address and prioritize issues—especially when vulnerabilities exist in deployed production applications that call for urgent remediation.

Balance Access Controls With Developers’ Visibility Into K8s Services

Parity between environments is an important feature of Kubernetes, enabling a Kubernetes manifest to easily create and command resources on any cluster. From a security perspective, though, this ease with which developers could exercise their power to change Kubernetes environments has led to greater access limitations. For example, organizations often deny developers access to kubectl for production deployments and disallow a workload’s author from being its deployer. 

At the same time, Kubernetes and microservices make it so that an application’s functionality is created out of many granular deployments and services, all in communication with each other. Nefarious attack activities can hide among those communications, requiring a high number of networking policies with rules that safeguard container security. With security shifting left, these policies are increasingly implemented and managed in Kubernetes manifests rather than via networking or infrastructure engineering. Empowering developers with access to data and metrics that enable them to understand service deployments and communications is critical to successful application security. Organizations should implement tooling that protects developer behavior with role-based access controls while allowing access to production application logs with no code changes or redeployments necessary.

Harden Service Communications (Even Outgoing Traffic)

Given the risk of attacks arriving via ingress or egress communications and escalating within service-to-service traffic, Kubernetes application security requires hardening all inputs and outputs as appropriate. It’s important to introduce secure ingress and egress policies and enforce hygienic practices, with attention to providing an excellent DevSecOps experience for developers.

Responses to and from external application users touch multiple services, requiring thoughtfully tailored security. Naturally, a service that processes public traffic should accept all traffic. However, a service that communicates with a login or authentication service should only communicate directly with that service, and not via an external connection. 

Use DevSecOps to Enhance the Developer Experience

Developer culture and the quality of an organization’s developer experience impact everything from the outcomes of digital transformation initiatives to recruiting and retaining talent to long-term competitiveness. Digital transformation is never a singular finite project but is instead a process of continuous improvements. The continuity of that process depends on retaining knowledgeable veteran developers and earning their steadfast commitment to the organization’s capabilities and future. Introducing modernized capabilities will similarly often require adding fresh talent to the development team. Cultivating a positive, efficient and enjoyable developer experience is foundational to an organization’s ability to stay current and competitive. 

The stresses of Kubernetes complexity, and the drag on productivity that are inevitable with inelegant approaches to Kubernetes security, absolutely can and will hamper developers’ enthusiasm for their work. Developers are human beings able to handle only so much cognitive load; the less unnecessary load, the happier they’ll be. Constant context switching, whether between development and Kubernetes security or other areas that rightfully deserve their own focus, will wear away at developers’ goodwill. 

Building an inviting developer experience boils down to increasing the simplicity with which developers can apply their expertise and navigate the processes of advancing their applications into production. DevSecOps offers to provide valuable advantages toward achieving that goal. Simplifying security while empowering developers with data and protecting them with the right policies can pave a glide path, and help define an experience that makes developers feel supported and fulfilled.

While developers play a key role in securing Kubernetes, cumbersome policies and tasks that are tedious or outside a developer’s comfort zone will only lead to dissatisfaction and poorer security. Instead, making an elegant developer experience part of DevSecOps goals pays dividends across the board, enabling developers to do what they do best—whether contributing to successful security or application features.

Vivek Pandey

Vivek Pandey is the Vice President of Engineering at Shipa.io, a cloud-native application-as-code platform. Prior to Shipa, Vivek was a co-founder at CloudBees and has held engineering roles at Sun Microsystems and VMware.

Vivek Pandey has 1 posts and counting. See all posts by Vivek Pandey