Building Secure Production-Ready Kubernetes Clusters and Containers, Part 2
The adoption of Kubernetes is growing rapidly as organizations reap the benefits of containers to construct advanced and scalable systems and applications. Building containerized, cloud-native infrastructure is a complex process, but it can pay off in the form of IT cost reduction, advanced application delivery and operational transformation. Using Kubernetes as a container orchestrator enables users to enhance the operational potential of containers by providing a simplified solution for container workflows and automating networking, storage, logs, alerting and other functions for all your containers. While Kubernetes has native controls for security as well, they aren’t typically enabled by default.
In Part 1 of this series, we highlighted how to construct and organize your Kubernetes stack. In this follow-on discussion, we’ll highlight several advanced steps that can make your environment more secure and stable before it is activated and implemented.
Secure
Implementing the native security features of Kubernetes is critical. While there is no way to discover ALL the vulnerabilities within a container environment without dedicated Kubernetes security tools, you can apply several best practices to start using Kubernetes’ native security features.
Role-Based Access Control
Role-based access control (RBAC) allows you to control who can view or modify different aspects of your cluster. To follow the widely recommended principle of least privilege, you will need to have RBAC implemented to limit what your cluster users and deployments can do.
Once RBAC is turned on, you will need to adapt the permissions to suit your needs. The Kubernetes project site provides a walk-through on setting up Roles and RoleBindings. Managed Kubernetes services require custom steps for enabling RBAC, such as Google’s guide for GKE or Microsoft’s instructions for AKS.
Pod Security Policies
Pod Security Policies are a resource, like a deployment, and can be created and updated through kubectl in the same way. Each holds a collection of flags you can use to prevent specific unsafe behaviors in your cluster.
Getting them working can be difficult—the best method is to get RBAC up and running, and then check out the guide from the Kubernetes project.
Network Policies
Network policies are objects that allow you to state explicitly which traffic is permitted; Kubernetes will block all other non-conforming traffic. Limiting network traffic in your cluster is a basic but vital security measure. Kubernetes by default enables open communication between all services, meaning an internet-connected service could be connected to a database storing sensitive information, leaving a vulnerable path.
You can find best practices on creating network policies here.
Secrets
Services may need to authenticate one another, other third-party services or users. Secrets can accomplish this task and store sensitive data such as passwords securely in Kubernetes.
One strategy is to put secrets into read-only volumes in your container. You can find an example in this Using Secrets write up.
Scanners
Scanners are useful for finding out what vulnerabilities exist in the versions of software your image contains by inspecting the components installed in your images, everything from the OS to your application stack. Most cloud providers offer an option, but you can host it yourself with the open source Clair project.
Stabilize
Kubernetes clusters can become complicated environments to maintain as you add components—a lot can go wrong. It’s important to reduce risks that can emerge in the development cycle and part of that process is ensuring the stability of your clusters. You need to service clusters consistently and actively to prevent problems that will likely arise. Use the following guidelines to help create a stable foundation for your containerized workflows.
Continuous Integration/Continuous Delivery
Continuous integration/continuous delivery (CI/CD) is a concept that operates with the belief that every modification committed to your codebase should add incremental value and be production-ready. If something in your codebase changes, you probably want to launch a new version of your service.
This mindset helps your engineering team keep quality assurance a focal point. Fixes become an immediate priority for the whole team, because every change thereafter that relies on the broken commit would also be broken.
Use CI/CD systems to help manage this process. Several options, from managed to self-hosted, are available.
Canary
Canary is a way to bring service changes from a commit in your codebase to your users. Bring up a new instance running your latest version and migrate users to the new instance slowly, gaining confidence in your updates over time, as opposed to swapping over all at once.
Unit and integration tests can’t completely simulate running in production—there’s always the chance something will not function as intended. Using canary limits your users’ exposure to these issues.
Kubernetes provides many routes to incrementally update deployments. The most straightforward approach is to create a separate deployment that shares a load balancer with currently running instances; the idea is you scale up the new deployment while scaling down the old until all running instances are on the new version.
Monitoring
To do load balancing, you’ll need to track and record what your services are doing. Unfortunately, things do go wrong, and when they do, you have to know what happened.
Two steps are crucial to successfully monitoring a service: The code needs to be instrumented, and the output of that instrumentation needs to be fed somewhere for storage, retrieval and analysis.
How you perform instrumentation is largely dependent on your toolchain. For storing the output, using a managed SIEM is recommended unless you have specialized knowledge or need. DIY always requires way more time and effort than you expect when it comes to anything storage-related.
Next Steps
Container systems aren’t perfect, so you will need to continuously adapt your environment as it grows in response to your changing stack. Unfortunately, these changes make it difficult to enforce best practices as Kubernetes deployments increase in size and complexity. Managing growth manually becomes impossible, resulting in challenges to the safety and stability of your systems. You can address these challenges by adopting some of these advanced tools for better cluster management and improved activity monitoring.
Service Meshes
To effectively create a virtual network that can be used when implementing services, you can use services meshes to manage interservice communications. They can alleviate some of the more tedious aspects of cluster management.
Service meshes vary greatly in complexity and setup time, so it’s better to set it up earlier rather than later.
Read this beginner’s guide to Istio service mesh to learn more.
Admission Controllers
Admission controllers provide a great catchall tool for managing what’s going into your cluster. They allow you to set up webhooks that Kubernetes will consult during bring up. They come in two flavors: mutating and validating. Mutating admission controllers alter the configuration of the deployment before it is launched. Validating admission controllers confer with your webhooks that a given deployment is allowed to be launched.
Admission controller use cases are broad and numerous, providing a great way to iteratively improve your cluster’s stability with homegrown logic and restrictions, as seen here. You can also follow these 11 tips to help you operationalize admission controllers.
Setting up and organizing containers and Kubernetes clusters is tedious. Stabilizing and securing them can be even more complex. However, you’ll gain immense operational benefits from adopting containerized, Kubernetes-based cloud-native infrastructure. Containers can empower organizations to scale while promoting organizational security. Following this two-part guide will give you a quick and easy framework to start to explore both the foundational and advanced capabilities of Kubernetes that will help lead your organization’s migration to containers.