Docker Hub Breach: What to Do Now
Last week, it was disclosed that a breach had occurred in the Docker Hub repository for container images. In an email to customers, Kent Lamb, director of Docker Support, wrote: “During a brief period of unauthorized access to a Docker Hub database, sensitive data from approximately 190,000 accounts may have been exposed (less than 5% of Hub users). Data includes usernames and hashed passwords for a small percentage of these users, as well as Github and Bitbucket tokens for Docker autobuilds.”
As a result of this breach, it’s possible that images in your Docker Hub repository may have been tampered with or overwritten. To mitigate this possibility, you should take immediate action. You can follow the recommendations below for any deployment.
Rotate Your Secrets
Despite the passwords being hashed, you should change your passwords. Docker has already revoked the GitHub access keys and tokens that may have been used.
For Docker Hub user accounts, rotating secrets is a fairly straightforward process:
Browse to https://hub.docker.com/ and login to your account.
Open the drop down under your username and select “Account Settings.”
Scroll down to Password and update.
If you use Docker Hub with Kubernetes environments, you’ll also need to roll your ImagePullSecrets, which can be more involved. The first step is to identify all of the ImagePullSecrets in your clusters, which can be done by running the following kubectl commands:
kubectl get secret –field-selector=type=kubernetes.io/dockerconfigjson –all-namespaces
kubectl get secret –field-selector=type=kubernetes.io/dockercfg –all-namespaces
Ideally, the secrets are named in such a way that you can identify to which registry they correspond. If not, you can parse the secret as a last resort to determine the corresponding registry. The following kubectl command will print out all the registries referenced in your image pull secrets:
kubectl get secret –field-selector=type=kubernetes.io/dockerconfigjson –all-namespaces -o json | jq -r ‘.items[] | {namespace: .metadata.namespace, name: .metadata.name, registries: .data[“.dockerconfigjson”] | @base64d | fromjson.auths | keys[]}’
Sample output:
{
“namespace”: “default”,
“name”: “gcr”,
“registries”: “https://us.gcr.io”
}
{
“namespace”: “default”,
“name”: “stackrox”,
“registries”: “https://docker.io”
}
Now delete the image pull secrets referencing docker.io and recreate them using kubectl (don’t forget to put them in the proper namespaces!).
Audit Your Images
Because Docker didn’t provide a specific timeline for this breach, no one knows how long ago the unauthorized access occurred. As with most breaches, the perpetrators may have had access to compromised resources significantly longer than just last week. To be safe, you should verify recently pushed images going back over the past several weeks. Doing this audit can be difficult as not every registry will let you filter the data by image age.
To help pull manifests and check the age of your images, you can use an open source tool such as the registry CLI available here. Here’s an example output:
repo=registry-1.docker.io/library/nginx
tag=latest
v1digest=$(reg manifest “${repo}:${tag}” | jq -r .config.digest)
reg layer “${repo}@${v1digest}” | jq ‘{created: .created, history: .history}’
Sample output:
{
“created”: “2017-03-27T19:48:34.817993292Z”,
“history”: [
{
“created”: “2017-03-21T18:28:51.055495122Z”,
“created_by”: “/bin/sh -c #(nop) ADD file:4eedf861fb567fffb2694b65ebdd58d5e371a2c28c3863f363f333cb34e5eb7b in / “
},
{
“created”: “2017-03-21T18:29:05.091744235Z”,
“created_by”: “/bin/sh -c #(nop) CMD [\”/bin/bash\”]”,
“empty_layer”: true
},
…
}
Inspect Your Environment for Unexpected Activity
Attacks on the build pipeline can have serious downstream effects on what is currently running inside your infrastructure. Tainted images can be difficult to detect, and the containers launched from them may even run as expected, except with a malicious process in the background. Backdoored images with cryptominers also have been discovered on Docker Hub.
Unexpected changes in images will have an effect on application behavior, making runtime detection and application baselining critical. Characterizing the behaviors of individual Kubernetes deployments will highlight deviations in network connectivity, file access and process executions. These deviations are all indicators that malicious activity is taking place within a container. You need the ability to quickly inspect runtime activity within your containers to verify they are running only expected processes.
In these breach scenarios, tooling that enables an effective triage process is essential. Being able to quickly identify what resources are affected, determine a solution and execute efficiently across your entire container environment are key enablers to reducing risks to your business.
Rotating Secrets
Based on our ability to collect and index Kubernetes secret metadata allowing for quick accounting of the different secrets being managed, a search for ImagePullSecrets that reference docker.io and are older than one day yields a queue of passwords that need to be updated and the environments in which they are located:
Auditing Images
Examining Dockerfiles across your infrastructure, as seen in the following screenshot, returns all of the images that are using docker.io that have been built in the last 30 days. This allows you to inspect the content of those images and determine whether they contain any suspicious layers.
Inspecting Deployment Behavior
Runtime detection provides visibility into all executed processes and will alert on known malicious processes. In conjunction with image auditing, runtime detection lets users quickly analyze the deployments that may have been compromised and verify no process irregularities are present.
Summary
After events such as this Docker Hub breach, you need tools and processes that help you rapidly identify where you’re vulnerable to the security lapse. Security platforms purpose-built for containers and Kubernetes can make these tasks easier by combining configuration checks, runtime detection, powerful search features and other capabilities to make it easier for you to triage issues and identify problem spots throughout your environment.