NSA Security Best Practices for Kubernetes
In this series, I’ve outlined why every organization should care about the NSA’s Kubernetes Hardening Guidelines and examined different areas of the guidelines. What you may suspect is that most of the guidelines outlined by the NSA are simply Kubernetes security best practices.
Kubernetes abstracts away just enough of the infrastructure layer so that developers can freely deploy while ops teams retain access to important governance and risk controls. The challenge is that development teams new to Kubernetes may neglect some critical security features. Often the easiest way to get something working is to soften its security.
A focus area in the NSA Kubernetes Hardening Guidelines is upgrading and application security practices. And it’s just plain good advice.
Promptly Apply Security Patches and Updates
We often see a pattern in which organizations neglect to fix security issues in their dependencies: A vulnerability is identified, a patch created but then not deployed. Often this is because a person or team is afraid of what might break if an upgrade occurs.
The NSA guide says:
“Security is an ongoing process, and it is vital to keep up with patches, updates and upgrades. The specific software components vary depending on the individual configuration, but each piece of the overall system must be kept as secure as possible. This includes updating Kubernetes, hypervisors, virtualization software, plugins, operating systems on which the environment is running, applications running on the servers, all elements of the organization’s continuous integration/continuous delivery (CI/CD) pipeline and any other software hosted in the environment.”
Keeping Kubernetes and all the underlying components and container images up-to-date can be a full-time job (or even multiple full-time jobs). Having the right tooling in place can help reduce the toil and engineering effort involved. Fortunately, there are good open source tools that can help teams identify what needs to be updated.
Nova, for example, is an open source tool that scans an environment for any Helm releases which are out-of-date or deprecated. Pluto, another open source tool, helps users find deprecated Kubernetes API versions in their code repositories and their Helm releases.
Teams need to run regular vulnerability scans of the container environment, either in the operating system or in the software installed on top, and patches. It’s just as important to ensure Kubernetes itself stays up to date as each version is only kept in support for about one year.
Perform Periodic Vulnerability Scans and Penetration Tests
The NSA recommends “periodic vulnerability scans and penetration tests on the various system components to proactively look for insecure configurations and zero-day vulnerabilities.”
A Kubernetes cluster is not a static piece of infrastructure. It is a constantly evolving ecosystem with new applications and updates being applied daily, often automatically. It’s imperative that organizations routinely scan their Kubernetes environment for vulnerabilities using a combination of automation and manual penetration testing.
Uninstall and Delete Unused Components
Another security best practice recommended by the NSA includes “uninstalling any old, unused components from the environment and deployment pipeline.” The goal is to reduce the attack surface and avoid having outdated software putting the environment at risk.
Making sure that stale and unused deployments don’t linger inside your Kubernetes environment is important for both security and general hygiene. Unfortunately, Kubernetes doesn’t have a generic way to check the staleness of resources in the cluster. There is an open issue, though some things can be checked on a case-by-case basis.
Kubernetes governance tools that allow for teams to customize policies can detect Helm charts and deployments that have not been updated for a certain number of days, which greatly helps reduce the toil and manual effort involved in detecting stale resources.
NSA Guidelines are Simply Kubernetes Security Best Practices
The entire NSA Kubernetes Hardening Guide reminds us that these are just good solid security best practices. We should all ensure that whatever system is in place it is regularly scanned for vulnerabilities, patches are implemented and unused components are deleted. Following these guidelines, whatever the industry, will help any Kuberentes environment protect against risky security activity.