IPFS Emerges as Tool to Distribute Container Images
Contributors to the InterPlanetary File System (IPFS) have come up with a more efficient way to distribute container images.
IPFS is a distributed file system that makes use of a global namespace to connect all computing devices. The fundamental difference between IPFS and other distributed file systems is a decentralized system of operators who hold a portion of the overall data, which serves to create a highly resilient system for storing and sharing files. Any operator on the network can serve a file by its content address, and IT teams can find and request content from any node using a distributed hash table (DHT).
Molly Mackinlay, project lead for IPFS and a senior product manager for Protocol Labs, which provides protocols, systems and tools to improve how the internet works, says the IPFS community in collaboration with Netflix has been looking into a way to distribute containers across a large-scale, multi-region environment more efficiently than relying on application programming interface (API) calls to, for example, DockerHub.
As part of that effort, contributors to the IPFS community have been focusing on Bitswap, the mechanism IPFS uses to transfer pieces of a file between two or more peers. The contributors added a new way to fetch blocks to the Bitswap protocol that cut transfer time in half for benchmarked use cases around container distribution. In the previous version of Bitswap, it took on average 9.08 seconds to pull a 300 MiB image to 32 leeching peers. The optimized branch brought this down to 3.16 seconds—20% faster than DockerHub (3.93 seconds).
The issue at Netflix that IPFS researchers are trying to address revolves around Titus, the container management platform Netflix created to distribute millions of container images around the world using the Amazon Web Services (AWS) cloud platform. When DevOps teams push a release image to production, the image needs to be replicated to Docker Registries in other regions. Otherwise, deployments will experience slow transfer speeds and higher cross-region data costs. In effect, the goal is to leverage IPFS as a content delivery network (CDN) for container images.
Mackinlay says these new capabilities are scheduled to be made available when the next updates to Bitswap are merged into the next release of IPFS.
Beyond large webscale companies such as Netflix, it’s not clear how many traditional enterprise IT organizations could benefit from employing IPFS to distribute containers. However, as more applications span global cloud computing environments the number is surely growing.
In the meantime, the number of container images in IT environments continues to explode, which means container management issues are starting to raise their head. In theory, platforms such as DockerHub and orchestration engines such as Kubernetes are meant to address those issues. However, it’s also clear there is still plenty of room for additional innovation when it comes to container images.
The challenge many IT organizations soon will face is determining precisely what their container management strategy will be as the number of options and alternatives continue to expand.
