The NSA’s Kubernetes Hardening Guide offers a lot of advice on implementing the best defense-in-depth approach to your workloads. Part of that guide outlines how Kubernetes users can secure and harden the network.
Network is King
Kubernetes resources—pods, containers and external services—need to be able to communicate. By default, these resources are not isolated and therefore lateral movement or escalation is possible if breached. The NSA recommends using network policies and firewalls to separate and isolate resources, secure control planes and encrypt traffic and sensitive data at rest.
Secure the Control Plane
The NSA guidelines offer a number of suggestions for securing the control plane from the network. It first requires users to “lock down access to the control plane using a firewall … [because] the control plane should be highly protected.” It also states that “The Kubernetes API server should not be exposed to the internet or an untrusted network.”
Securing the control plane can be easily done using a managed Kubernetes service like GKE or EKS. VPCs and network access control lists (NACL) can be used to control access to control plane nodes if you are managing it internally. We recommend gating access to the control plane through an SSH bastion in the same VPC as your control plane nodes.
The NSA also suggests using role-based access control (RBAC) to lock down access to the control plane. Enabled by default, RBAC is one method used to control access to a cluster resource based on the roles of individuals within an organization. RBAC can be used to restrict access to user and service accounts but provisioning and assigning roles can be tricky. Two open source projects that can help include:
- RBAC Manager provides a simpler interface for provisioning permissions.
- RBAC Lookup allows users to determine precisely who has access to what.
While every organization will have different needs and make different design decisions about how to provision RBAC, it’s imperative to make use of Roles and ClusterRoles to provision different levels of access for different personas. This helps with adherence to the principle of least privilege, with the added benefit of preventing honest mistakes.
Another guideline includes separating networks for the control plane components and nodes including user-facing applications. Doing so limits the attack surface as worker nodes and other network segments do not need to communicate with the worker nodes or Kubernetes services. In the event of an attack, this network segmentation activity will help to limit the blast radius.
Open source tooling like kube-hunter can help you better understand which nodes have control plane access. To get a sense of what access those nodes have to the control plane, we recommend configuring kube-hunter to run on application nodes.
etcd Network Security
Another step to securing the control plane is to configure it to use authenticated, encrypted communications using TLS. The etcd backend database stores state information and cluster secrets. Without configuring it properly, a malicious actor could gain write access to etcd, as well as root access to the entire cluster.
The NSA also recommends encrypting etcd at rest and to use a separate TLS certificate. The etcd backend database is a critical control plane component and the most important piece to secure within the control plane.
Again, by using a managed Kubernetes service like GKE or EKS, you can avoid having to worry about your control plane configuration. But if you do need to manage your control plane nodes, be sure to encrypt the underlying volumes, as well as set up a unique TLS certificate for communication with etcd.
Finally, the NSA suggests setting up network policies to isolate resources. Network policies control traffic flow between pods, namespaces and external IP addresses. By default, no network policies are applied to pods or namespaces, resulting in unrestricted ingress and egress traffic within the pod network.
While Kubernetes comes with a built-in NetworkPolicy resource, it is fairly limited in scope. We recommend using an open source service mesh like Linkerd. This service can capture Layer 7 semantics, allowing for more readable, maintainable and fine-grained network policies. Linkerd also enforces its policies at the pod level rather than the CNI level, which gets closer to the zero-trust ideal.
An Extensive Network Security List
Kubernetes networks need special attention. The NSA guidelines are lengthy but are not prescriptive on what to do. Get more help on steps to meeting NSA Kubernetes Hardening Guidelines and check out parts one and two of this series.