Understanding Container Security – Part One: Security Gaps

In our recent interview with Aqua Security’s Rory McCune about CRI-O vulnerability CVE-2022-0811, he mentioned that he’d seen a sharp rise in container cybersecurity threats this year. His experience isn’t unusual; last June, a Red Hat survey found that 94% of respondents had suffered a Kubernetes security incident in the last year.

Is this a surprise? Not really. Scalable and efficient, containers are increasingly popular. A Cloud Native Computing Foundation survey found container use ballooned 300% between 2016 and 2020. Organizations using more than 5,000 containers rose 109%, while those using more than 250 containers increased by 61%.

This kind of rapid ascent makes containers a naturally attractive target for threat actors. But the rise in container popularity hasn’t been matched by a rise in security expertise for containerized applications and data.

“With any technology adoption, there’s a window when the technology is gaining critical mass, but security controls and practices haven’t adjusted yet,” said Mark Nunnikhoven, distinguished cloud strategist at Lacework. “Containers are at that point in their evolution now.”

So, with threats on the rise, how many teams are adequately prepared with a strong container security posture? We asked security and container thought leaders for their views on the rise in incidents—and their advice for improving container security architecture.

Understanding the Root of Container Security Incidents

Like any business, cybercriminals look for new revenue streams—and bigger returns on their investments. As containers play a larger role in the software development life cycle, attackers see multiple opportunities—something Dan Moore, solutions architect at FusionAuth, calls “a natural outgrowth.”

“As more workloads are shifted to containers to take advantage of their deployment benefits, more focus is being placed on escaping the security boundaries,” Moore explains. “After all, most containers are built on Linux primitives such as namespaces and cgroups. It’s software all the way down.”

One of the chief container benefits also presents an interesting security challenge, according to Nunnikhoven.

“Each container is built on layers of other containers. This code reuse helps optimize developer efficiency,” Nunnikhoven said. “It also puts a lot of trust in the layers you’re building on. Lacework Labs has seen a number of container images with malicious intent in public registries. It’s important to take action to ensure that each layer does what is intended and only what is intended.”

As the de facto standard in container orchestration platforms, Kubernetes offers several built-in security features such as admission controllers, network policies and role-based access control (RBAC) (usually enabled by default on later versions of Kubernetes.) But its high level of complexity—including the number of engineers, APIs and cloud accounts—can increase the likelihood of misconfigurations and security gaps. An engineer may feel confident in containerizing an application but lack the expertise to build security into that process.

“Understanding how Kubernetes operates, its security controls and where you need to apply third-party tooling is critical to success here,” said Nunnikhoven. “This is a dynamic environment with a lot of moving parts.”

But without a firm grasp of container security best practices, many engineers are leaving their apps open to exploitation. “Often, teams will simply hack together a Dockerfile and call it a day once they see it working in production,” said Robert Brennan, vice president of product development at Fairwinds. “Maybe a security team follows up to make sure it’s not running as the root user, that the filesystem is read-only—but even security folks are struggling to adapt to the containerized world.”

Bridging Security Culture Differences
One of the most foundational challenges with container security is its inapplicability to older security tooling. These tools are often built for a group of single, static machines. But Kubernetes is a fluctuating, distributed environment that defies that paradigm.

“Nodes are constantly being created and destroyed as applications scale up and down dynamically,” Brennan pointed out. “And you have many different applications sharing CPU, memory and disk resources This means a flaw in one application has the potential to affect all the others in your environment.”

Brennan noted that the cultural “shift left” towards a DevOps model presents another challenge. The expanded circle of engineers responsible for configuring and deploying workloads can boost both productivity and application reliability. But it also increases the number of people who need to be educated about secure deployment configuration. “Developers are used to looking for SQL injection or code execution flaws, but they’re not used to thinking about the nuances of containerization and deployment configuration,” Brennan said.

It’s clear that container security expertise needs to catch up to the current threat level around containerized apps. In part two of this series, we’ll review specific practices to secure these apps and bridge the gap with new security tooling that can meet this moment in the containerized world.