7 Jun 2017 Berkeley - Scientists used to come to Gregory Kurtzer of Lawrence Berkeley National Laboratory's IT department a lot, asking for a better way to use software containers in a high-performance computing (HPC) environment. After a while he got tired of saying, "Sorry, not possible". So he invented a solution and named it Singularity.
Within a few months of its release last year, Singularity took off. Computing-heavy scientific institutions worldwide - from Stanford University to the Massachusetts Institute of Technology to various sites on the European Grid e-Infrastructure - flocked to the software.
"Singularity has been making huge strides in the computing community", Gregory Kurtzer stated, with some surprise, adding that Open Science Grid, a consortium that provides distributed computing resources for scientific research, has served over 20 million containers with Singularity.
It's now on its seventh release (version 2.2.1) and has caught on so quickly that Gregory Kurtzer has launched SingularityWare LLC to further develop and support the open-source software. The company is being funded by RStor Inc., a start-up based in Saratoga, California. Gregory Kurtzer, the long-time technical lead and architect for the HPC Services group at Berkeley Lab with a joint appointment at UC Berkeley, will shift to an advisory role at the Lab in order to focus on Singularity.
"Berkeley Lab makes some Lab-developed software available at no cost to maximize its impact and to participate in the open-source software community", stated Elsie Quaite-Randall, Berkeley Lab's Chief Technology Transfer Officer. "Singularity fosters innovation as open-source software, and now SingularityWare LLC - like other Berkeley Lab start-ups - will set out to expand the reach and adoption of an important technology."
A typical case where users might need Singularity is if they want to run an application such as Google's TensorFlow. "They may need a very specific version of Tensor Flow installed", Gregory Kurtzer stated. "They can create a container to do that in about five minutes. Then they can take that container, bring it to our environment and run it, even if we dont have that version of Tensor Flow installed."
Software containers make it possible to take your entire computing environment, including your files and all the applications you want to run, and encapsulate it so it can be easily replicated on another machine without worrying whether the new machine has a compatible operating system, libraries, applications, and so forth.
"Containers share some of the use cases of virtual machines but without the code redundancy and performance hit associated with virtualization", Gregory Kurtzer stated. "Singularity containers allow a user to encapsulate an entire operating system (OS) environment and use it on a shared HPC system like any other programme, without an admin doing anything."
Another example where Singularity would be useful would be allowing other scientists to reproduce experiments. "Say you just published an article. Wouldn't it be nice to have a location you can cite where someone can download the Singularity container and replicate all the experiments?" Gregory Kurtzer asked. "Someone can enter the container, and now theyre sitting in the exact same environment as you were."
Containerization was developed for enterprise environments, where it has become very popular, especially with the rise of Docker's container technology. "Docker's container solution is for the enterprise. But the scientific use case is quite different", Gregory Kurtzer stated. "Our goal isn't to run as many containers as we can on a single host, with each having the illusion of sole occupancy and isolation, but to run maybe one, and enable it to utilize all the resources on that host. It's kind of the opposite of isolation."
So Gregory Kurtzer started working on his own solution, and four months later, the first version was released last spring. "When I started working on it, I asked, what do scientists really need from containers? They need reproducibility, mobility, and also freedom¾the ability to install their own applications and run in their own environment, and store it just like any other data file", Gregory Kurtzer stated. "That's what Singularity solves for scientific computing."
Gregory Kurtzer chose the name Singularity for its meaning in astronomy. "As I understand it, it's the culmination of a whole bunch of matter in the universe forming a single infinitely dense point", he stated. "That's what I was thinking when I was creating Singularity - taking everything necessary to create a reproducible scientific environment and putting it in one file."
Singularity also enables users to run legacy workflows easily. Grefory Kurtzer cites one example of how his group saved an 18-year-old workflow from failing hardware and was able to convert it to a Singularity container that is still being used in production today.
Gregory Kurtzer believes Singularity will benefit scientists who may not even know they need it. "We're trying to reach out to more scientists and engage with additional groups, especially those who are not traditional HPC users, also known as the computational 'long tail of science'", Gregory Kurtzer stated. "We have a lot of users that are running computationally intensive jobs on their laptops and workstations and not making use of the dedicated computational cycles that are designed specifically for computing and available to them. With Singularity we can easily make these large computing resources tangible."