Container Security: From Development to Runtime

Containers have become a critical part of cloud environments because they are inherently equipped to run arbitrary code with minimal overhead. This makes them a valuable asset to organizations trying to deliver code at speed and scale rapidly. But the speed they facilitate is predicated upon an agile structure that allows for widespread access and resource efficiency.

Container security tends to be discussed in terms of DevOps efforts, especially as it pertains to what developers bring to containers and the administration of these containers (recent research shows that lack of security around container orchestration tools is a challenge). What’s needed, however, is a more systematic approach to container security that includes the full life cycle from development to runtime as well as container repository management.

Among the biggest threats to security are container images. These point-in-time instances often contain credentials and other privileged information. Access to images can give a hacker broad range throughout the container and into other layers of a cloud production environment.

Security efforts focused on container images are critical. Two things to consider:

  • Vulnerable Software: A container image contains everything but the Linux Kernel laid out in layers—base, runtime and application. Creating an image has a shared responsibility between dev and ops. Dev folks publish the list of application dependencies which is reviewed for known vulnerabilities and advice on security errata. This responsibility is critical to success bridging the gap between application dependencies and security vulnerabilities.
  • Image authenticity: Container images are designed to be unchangeable. The best practice for updating an image is to create a new one with the desired changes (and remove the old one) and submit it to the container repository. At the heart of it is a unique digest, usually a sha256 sum of the container image stamped at image creation time as seen here:

While the container tools can’t prevent the image from being mutated (with malicious intent), it is still fairly easy to detect a change and hence be prepared with a remediation strategy.

Having secured the container images we can focus our efforts on ensuring issues related to container host misconfiguration and container startup capabilities. A lot of effort has gone into this aspect of security both from the standardization and tooling perspectives. Governance organizations such as NIST (SCAP specification) and CIS (Docker Benchmark) have published standards defining guidelines around proper container security. These standards focus on patch installations, system security configuration and system compromises, and they help identify and mitigate known security vulnerabilities establishing a secure configuration posture. Community efforts have delivered tools and methods for scanning host systems, container images and container runtime, making sure no malicious or vulnerable code gets distributed as part of a container image and/or host system compromise, causing application failure.

Runtime measures, on the other hand, are post-development initiatives, with a deeper reach into production deployments. These issues range across the following aspects:

  • Kernel security: Despite the maturity of the Linux kernel and its components, new issues will continue to surface that allow bad actors to use kernel-user space interaction to launch exploits. The responsibility to manage these threats rests on users to catalogue the application dependencies using continuous and automated tools and be engaged with the community monitoring Common Vulnerability Databases (CVDs). Kernel-based vulnerabilities typically are difficult to deal with because of the inherent dependencies on vendor update release cycle and the associated application downtime.
  • Container runtime compromise: Container runtime is yet another application sanctioned to run in an environment and is subject to similar types of threats and vulnerabilities. However, the use of resource isolation artifacts by container runtime elevates the risk level.
  • Container escape: Part of the value proposition in running containers is there’s still a single kernel shared among all the isolated processes, which is the gatekeeper for all kernel-level system calls. While this keeps the containers lightweight, it also can be the source of common issues. To illustrate this further, let’s take an example of user privilege handling in the context of a file write operation. The  kernel validates privileges of the UID/GID associated with the process that’s initiating the write file system request before dispatching it to the underlying file system driver. Since you should not have different users correspond to the same UID/GID for the processes running in a container and the host system, this delicate dance of usernames, group names, user IDs (UID) and group IDs (GID) inside and outside poses a severe challenge in building a secure system at scale.
  • Data exfiltration: An essential function of a security team is to ensure security of company secrets by limiting accessibility of it to unauthorized users. In traditional data centers, the security posture focuses on the physical perimeter of the network. In the cloud, users are sharing the network fabric  with other tenants. This control is further weakened with the adoption of microservices and the dynamicity using container orchestration blurring the lines between computation nodes and data access. A new architecture must be implemented that prevents data exfiltration through continuous and rapid discovery of risky activity, as well as a response infrastructure that can support remediation.
  • Resource abuse: Container runtimes have built-in safeguards for the required resources such as CPU, memory and block IO. While deploying containers, one must consider using these guardrails in all instances. Putting your application through an x-ray scanner and continuously monitoring user and event behavior, recording every change (application, user, process, network, machine) is critical, while recognizing that not every change is anomalous. This offers a sliver of hope that there’s some order to taming the container beast.
  • Administration of shared secrets: Admission control, access control and permissions all require a secret one that auditors hyperfocus on. In the elastic world of cloud and containers, the challenge is not just limited to securely distributing the secrets but also to ensure the secrecy of secrets itself. The perils of baking the secrets inside the container image range from pilferage to secrets lifecycle management. Secrets management in an elastic environment is undergoing a correction to a longstanding structural problem. The new requirements evolving from this change is to consolidate secret operations—creation, reading, revocation, rotation and auditing under a common head as part of the continuous security model.

Containers provide an application distribution mechanism, and it’s also an ecosystem of existing images that makes it beneficial to develop a better understanding of the lower-level technologies and deliver on the scale and speed promise of this fascinating technology. In the true spirit of serving the larger cause of securing applications, machines are employed to theorize, test and confirm the impact of any security event. The alert models should be trained to only notify the user if it detects any suspicious activity, attack or breach, based on continuous monitoring of every activity of every cloud component at every layer all in near real time.

Dan Hubbard

Dan Hubbard is Chief Product Officer at Lacework responsible for driving the company’s product and security strategy for public and private clouds and security research. A pioneering force in Internet security, Dan’s expertise spans from reputation and advanced classification systems to large-scale security data mining, and cloud security. Prior to Lacework, Dan was CTO at OpenDNS, helped deliver the world’s largest cloud security network that led to the $600M acquisition by Cisco. Prior to OpenDNS, Dan was CTO at Websense, led R&D, launched the Websense Security Labs, and was instrumental in the company’s success from early days through successful IPO. Dan owns several patents in the areas of data classification and cloud security and is a frequent speaker at security conferences globally.

Dan Hubbard has 2 posts and counting. See all posts by Dan Hubbard