Uppsala Multidisciplinary Center for Advanced Computational Science

Singularity User Guide

Singularity (http://singularity.lbl.gov/) provide tools for running containters that are more suitable to traditional HPC environments when some other tools such as Docker or lxc. These containers can be portable and could be run both on your desktop machine and our clusters.

One of the ways in which Singularity is more suitable for HPC is that it very actively restricts permissions so that you do not gain access to addtitional resources while inside the container. One consequence of this is that some common tools like ping or sudo do not work when run within a container (as a regular user).

Singularity is installed and usable to run custom container images on the clusters irma, milou and rackham.

Running an existing image

It's possible to download and run pre-built images from the Singularity hub (https://singularity-hub.org/) using the singularity pull sub command such as:

singularity pull shub://singularityhub/ubuntu

Which will download the requested image and place it in the current directory. You can also upload and run image directly your self.

Once you have an image, you can "run" it with a command such as

singularity run singularityhub-ubuntu-14.04.img

which will try to execute a "run" target in the container. There are also the shell and exec subcommands for starting a shell and running a specific command respectively.

Access to UPPMAX file systems.

By default, singularity will try to help and map the UPPMAX file systems from the current cluster so that they can be accessed from within the container. For CentOS7 based clusters (irma, rackham), this works as expected.

For milou however, this does not work except for the home directory without make some changes to the containers used.

These changes are difficult and not supported to do at UPPMAX. It should however be fairly trivial on your desktop environment if you have singularity installed (see the Download / Installation page on http://singularity.lbl.gov). Once you have it, you should be able to run something like

sudo singularity exec -w myimagefile.img mkdir /sw
sudo singularity exec -w myimagefile.img mkdir /pica
sudo singularity exec -w myimagefile.img mkdir /proj

and upload the file normally (sftp/rsync) to UPPMAX. This modified image should be able to access the UPPMAX file systems.

Singularity is installed on the system (on each separate node) and does not require any module load to be available.

It's possible to run Docker containers (you can try to run

 singularity shell docker://debian:stretch

but note that Docker containers are typically designed to run with more privileges than are allowed with Singularity, so it's quite possible things do not work as expected.

Not all images may work everywhere!

Images run with the same linux kernel as the rest of the system. For HPC we systems, the kernel used tend to be quite old for stability reasons (with milou using very old kernels). This is not normally a problem, but can cause issues if the libraries of the images you try to run expects functionality added in newer kernels. How and what works is difficult to know without trying, but we have successfully started a shell in an image for the currently most recent Ubuntu release (17.04).

Creating your own images

If you have singularity on your own machine, you can create your own images and upload and run them from UPPMAX. Creating images on UPPMAX is not possible as it requires administrator (root) privileges.

Singularity provides funcitonality to create and bootstrap images, and the installation contains example definitions for bootstrapping various images you can use as a start (like Ubuntu, Scientific Linux and so on). Typical usage would be:

sudo singularity create myimage.img
sudo singularity bootstrap myimage.img examples/ubuntu.def

after bootstrapping, you can run 

sudo singularity shell -w testfil.img 

and perform additional fixes (although it would be recommended to implement these in the container definition file, see http://singularity.lbl.gov/docs-bootstrap for documentation on the file format).

If you will be running the container on milou, you probably need to create empty directories for /sw, /pica and /proj.

Running your container in jobs

A sample job script for running a tool provided in a container may look like

#!/bin/bash -l
#SBATCH -N 1
#SBATCH -n 1
#SBATCH -t 0:30:00
#SBATCH -A your-project
#SBATCH -p core
singularity exec ~/ubuntu.img echo "Hey, I'm running ubuntu"
singularity exec ~/ubuntu.img lsb_release -a