Docker Hub Breach: What to Do Now

Last week, it was disclosed that a breach had occurred in the Docker Hub repository for container images. In an email to customers, Kent Lamb, director of Docker Support, wrote: “During a brief period of unauthorized access to a Docker Hub database, sensitive data from approximately 190,000 accounts may have been exposed (less than 5% of Hub users). Data includes usernames and hashed passwords for a small percentage of these users, as well as Github and Bitbucket tokens for Docker autobuilds.”

As a result of this breach, it’s possible that images in your Docker Hub repository may have been tampered with or overwritten. To mitigate this possibility, you should take immediate action. You can follow the recommendations below for any deployment.

Rotate Your Secrets

Despite the passwords being hashed, you should change your passwords. Docker has already revoked the GitHub access keys and tokens that may have been used.

For Docker Hub user accounts, rotating secrets is a fairly straightforward process:

Browse to https://hub.docker.com/ and login to your account.

Open the drop down under your username and select “Account Settings.”

Scroll down to Password and update.

If you use Docker Hub with Kubernetes environments, you’ll also need to roll your ImagePullSecrets, which can be more involved. The first step is to identify all of the ImagePullSecrets in your clusters, which can be done by running the following kubectl commands:

kubectl get secret –field-selector=type=kubernetes.io/dockerconfigjson –all-namespaces

kubectl get secret –field-selector=type=kubernetes.io/dockercfg –all-namespaces

Ideally, the secrets are named in such a way that you can identify to which registry they correspond. If not, you can parse the secret as a last resort to determine the corresponding registry. The following kubectl command will print out all the registries referenced in your image pull secrets:

kubectl get secret –field-selector=type=kubernetes.io/dockerconfigjson –all-namespaces -o json | jq -r ‘.items[] | {namespace: .metadata.namespace, name: .metadata.name, registries: .data[“.dockerconfigjson”] | @base64d | fromjson.auths | keys[]}’

Sample output:

{

“namespace”: “default”,

“name”: “gcr”,

“registries”: “https://us.gcr.io”

}

{

“namespace”: “default”,

“name”: “stackrox”,

“registries”: “https://docker.io”

}

Now delete the image pull secrets referencing docker.io and recreate them using kubectl (don’t forget to put them in the proper namespaces!).

Audit Your Images

Because Docker didn’t provide a specific timeline for this breach, no one knows how long ago the unauthorized access occurred. As with most breaches, the perpetrators may have had access to compromised resources significantly longer than just last week. To be safe, you should verify recently pushed images going back over the past several weeks. Doing this audit can be difficult as not every registry will let you filter the data by image age.

To help pull manifests and check the age of your images, you can use an open source tool such as the registry CLI available here. Here’s an example output:

repo=registry-1.docker.io/library/nginx

tag=latest

v1digest=$(reg manifest “${repo}:${tag}” | jq -r .config.digest)

reg layer “${repo}@${v1digest}” | jq ‘{created: .created, history: .history}’

Sample output:

 

{

“created”: “2017-03-27T19:48:34.817993292Z”,

“history”: [

{

“created”: “2017-03-21T18:28:51.055495122Z”,

“created_by”: “/bin/sh -c #(nop) ADD file:4eedf861fb567fffb2694b65ebdd58d5e371a2c28c3863f363f333cb34e5eb7b in / “

},

{

“created”: “2017-03-21T18:29:05.091744235Z”,

“created_by”: “/bin/sh -c #(nop) CMD [\”/bin/bash\”]”,

“empty_layer”: true

},

}

 

Inspect Your Environment for Unexpected Activity

Attacks on the build pipeline can have serious downstream effects on what is currently running inside your infrastructure. Tainted images can be difficult to detect, and the containers launched from them may even run as expected, except with a malicious process in the background. Backdoored images with cryptominers also have been discovered on Docker Hub.

Unexpected changes in images will have an effect on application behavior, making runtime detection and application baselining critical. Characterizing the behaviors of individual Kubernetes deployments will highlight deviations in network connectivity, file access and process executions. These deviations are all indicators that malicious activity is taking place within a container. You need the ability to quickly inspect runtime activity within your containers to verify they are running only expected processes.

In these breach scenarios, tooling that enables an effective triage process is essential. Being able to quickly identify what resources are affected, determine a solution and execute efficiently across your entire container environment are key enablers to reducing risks to your business.

Rotating Secrets

Based on our ability to collect and index Kubernetes secret metadata allowing for quick accounting of the different secrets being managed, a search for ImagePullSecrets that reference docker.io and are older than one day yields a queue of passwords that need to be updated and the environments in which they are located:

Auditing Images

Examining Dockerfiles across your infrastructure, as seen in the following screenshot, returns all of the images that are using docker.io that have been built in the last 30 days. This allows you to inspect the content of those images and determine whether they contain any suspicious layers.

Inspecting Deployment Behavior

Runtime detection provides visibility into all executed processes and will alert on known malicious processes. In conjunction with image auditing, runtime detection lets users quickly analyze the deployments that may have been compromised and verify no process irregularities are present.

Summary

After events such as this Docker Hub breach, you need tools and processes that help you rapidly identify where you’re vulnerable to the security lapse. Security platforms purpose-built for containers and Kubernetes can make these tasks easier by combining configuration checks, runtime detection, powerful search features and other capabilities to make it easier for you to triage issues and identify problem spots throughout your environment.

Wei Lien Dang

Wei Lien Dang

Wei Lien Dang is StackRox’s container security expert and VP of product, responsible for product management and strategy. Previously, he was Head of Product at CoreOS and held senior product management roles for security and cloud infrastructure at Amazon Web Services, Splunk, and Bracket Computing. He was also part of the investment team at the venture capital firm Andreessen Horowitz. Wei Lien holds an MBA with high distinction from Harvard Business School and a BS in applied physics with honors from Caltech.

Wei Lien Dang has 1 posts and counting. See all posts by Wei Lien Dang