Preparing Your Incident Response Team for Container Incidents
The use of containers—and orchestration platforms like Kubernetes—is increasing rapidly around the globe. Analysts predict that by 2023, more than 70% of global organizations will be running more than two containerized applications in production, up from less than 20% in 2019. The downside of this rapid growth is that many organizations have fallen behind when it comes to securing all of this new technology. This is especially true when it comes to responding to any incidents which may occur in containers.
This article will give you five practical steps to ensure that, when a security incident involving a container happens, your incident response team will be prepared. It is very easy to think that existing policies, methods and tools will cover containers, but they are actually very different and must be accounted for during an incident response effort.
1. Update Incident Response Plan
An incident response plan (IRP) isn’t the most exciting aspect of security, but it is critical in the event of an incident. One of the most important goals of an IRP is to provide a shared understanding of the incident response process among all stakeholders. Without that shared understanding, a lot of time will be lost in trying to figure out who owns what and who should be taking which action.
Containers and their supporting infrastructure often fall under the purview of DevOps (but not always). Are they listed in your incident response plan? If not, they need to be added. This would include primary and secondary contacts who would be brought into the incident response effort. DevOps is an entire discipline of its own with different technologies than other areas of the organization, including the security group. Their help will be invaluable when trying to investigate an incident.
2. Implement Workload Threat Detection
Containers are not like virtual machines (VMs) or traditional servers at all. The monitoring tools, such as endpoint detection and response (EDR) and network detection and response (NDR), may not protect containers in any way. Most containers are Linux-based, as are the hosts they run on. Containers can be run on Windows, but it is relatively rare. Even if you have a Linux EDR or an endpoint protection platform (EPP) agent, does it understand containers? To agents that are blind to containers, everything will just look like a process. Should an incident occur, a lot of context will be unavailable and the source of the incident may be obscured.
Workload detection and response (WDR) tools are a new breed of agents that understand containers as well as what is happening on the host itself. These tools record all of the container metadata that is available should a security event occur. They can even record EDR-type data about process, network and file operations for analysis should a malicious event occur. Since containers are often short-lived, this is invaluable so an investigator can understand what happened and which container was responsible.
3. Update DFIR Tools For Containers
After your threat detection tools have done their job and an incident is declared, it is time to begin the in-depth investigation. Traditionally, this may involve imaging drives, gathering artifacts and combing through logs. However, if containers are involved it won’t be the same process that we have been doing for decades now. Containers are often ephemeral—44% live less than five mnutes—so by the time you start your investigation the container involved may be long dead. This is why container-aware monitoring is so important; the activities that now-dead container performed will still be recorded.
In some cases, a malicious container image might be the source of the incident, but these images can’t just be dropped in a sandbox for analysis. Instead, they need to be analyzed with specialized tools which understand how containers work. Then artifacts can be retrieved and analyzed. There are a number of open source tools that can assist in this process.
4. Update Playbooks
Playbooks are an important aspect of incident response (IR). They allow for a repeatable set of instructions for dealing with specific types of incidents. This lowers the number of potential errors during the investigation, but also helps newer IR team members learn and participate. Playbooks are often made for a given incident type, such as dealing with ransomware or malware. This wouldn’t change because containers are involved as the type of incident is usually the same.
The problem is most playbooks are written with legacy networks and tools in mind. These playbooks should be updated to include instructions about what to do if the incident involves containerized environments. This will save a lot of time should an incident occur, possibly saving your company a significant amount of money.
5. Practice!
You have now updated your IRP, deployed monitoring for your containers, updated your DFIR tools and updated your playbooks. Now you need to make sure it all works. There are several ways to go about this. The most common is to rely on internal testing, but that is often subject to internal assumptions that may not accurately reflect what happens during an incident.
Bring in a third party to help as they will lack any assumptions or prior knowledge that could represent a blind spot in your organization. The first step in this process is to conduct a tabletop exercise (TTX). It should be a scenario involving your container environment. This is a purely exploratory exercise that will go over your IRP, security infrastructure and playbooks. It is great for finding gaps or other missing elements.
If the TTX goes well, the next step would be to have a third party conduct a purple team exercise. This is a combination of a red team and a blue team exercise. Their experts will sit with your blue team as the red team does their thing. The goal of this is to understand where your blue team can improve with their training, technology and processes. Finally, if you are confident, a pure red team engagement can be done which will leave it all up to your blue team.
The order of these exercises is actually very important. You need to be sure your program is mature enough to make the purple and red teams worth the cost. The TTX will give you a good indication of where you are so you don’t waste money with exercises you are not ready for yet.
Conclusion
Containers bring many benefits to your organization, but they can also cause major problems should a security incident occur and your incident response capabilities are not ready. An updated incident response plan will ensure all of the needed stakeholders are involved. Extending threat detection to your container environments is critical not just to know if an incident has occurred, but for all of the data it can provide to assist in the investigation. Forensics is still alive and well in the world of containers, but it is different. Knowing the right tools to use will make the processes much quicker. Making playbooks that include containers and then using tabletop exercises or purple teams to ensure they are accurate is the final step to making sure your IR team is ready.