Sylabs Arrives to Advance High-Performance Computing Container Adoption
Sylabs came out of stealth today to provide support for container software designed specifically for high-performance computing (HPC) environments that are being employed to process scientific applications based on, for example, machine and deep learning algorithms.
Company CEO Greg Kurtzer says the open source Singularity container he developed for HPC environments is unique in that it is based on a single file (SIF) that encapsulates the runtime environment. That means that all user mobility, security controls compliance, archive, reproducibility and cryptographic signing of the runtime container are enabled via a single file versus requiring organizations to deploy containers such as Docker that are made up of multiple layers of services.
Kurtzer says that capability is critical in HPC environments that are designed to process jobs versus microservices. Singularity is compatible with images based on the Open Container Initiative (OCI) standard, while at the same time providing access to direct host IO and seamlessly integrating with resource managers, Message Passing Interface (MPI), batch job workflows and other utilities commonly employed in HPC environments.
Singularity also enables containers to without the security implications of granting users control of a root-owned daemon process or kernel features, adds Kurtzer.
Sylabs plans to make its development efforts available via a public GitHub repository for Singularity, which will provide the foundation for a commercially supported Singularity Pro edition that will be made available via a subscription license. That offering is available now as part of a private beta.
Kurtzer notes that HPC platforms are being employed to run an expanding class of workloads focused more on traditional enterprise use case. Those applications are ideal for Singularity because they typically involve data-intensive workloads running on graphical processor units (GPUs). Singularity was designed to natively run on GPUs, says Kurtzer.
Singularity was created in late 2015. Now in its 13th release, the scientific community regularly deploys more than a million Singularity containers a day via the Open Science Grid, a consortium that provides distributed computing resources for scientific research. It is estimated that more than 25,000 organizations already make use of Singularity containers.
Sylabs is coming into being at the time when many enterprise IT organizations are just starting to employ GPUs to drive advanced analytics applications infused with AI. Most of the algorithms being employed have been around for decades. But the widespread availability of massive amounts of compute power in the form of GPUs typically accessed via a cloud service coupled with inexpensive storage makes it economically feasible for more organizations to develop AI applications. The Singularity container provides a mechanism for running those applications on GPUs without having to become locked into a specific cloud service.
Many of the teams developing AI applications for enterprise IT organizations have extensive histories with scientific applications. Kurtzer created Singularity while working for the U.S. Department of Energy (DOE). In fact, Singularity is another example of how research and development partially funded by the U.S. government benefits the larger IT community. It’s not clear to what degree enterprise IT organizations will embrace Singularity, but the one thing that is for certain is that as concepts and ideas continue cross-pollinate across the container community, Singularity has already made a significant contribution.
