Skip to content

Using Apptainer (Singularity) on our clusters§

Myriad has Apptainer installed, which will be rolled out to our other clusters at a later date. The other clusters still have Singularity. You can use containers you have downloaded in your space.

Apptainer§

Run apptainer --version to see which version we currently have installed.

Set up cache locations§

Use this command to load the apptainer module to set up a suitable environment:

module load apptainer

Or set the locations manually:

# Create a build directory to build your own containers.
mkdir -p $XDG_RUNTIME_DIR/$USER_apptainerbuild

# Set $APPTAINER_TMPDIR to this local build location.
export APPTAINER_TMPDIR=$XDG_RUNTIME_DIR/$USER_apptainerbuild

# Create a .apptainer directory in your Scratch
mkdir $HOME/Scratch/.apptainer

# Set the cache directory to your Scratch.
export APPTAINER_CACHEDIR=$HOME/Scratch/.apptainer

You probably want to add those export statements to your .bashrc under # User specific aliases and functions so those environment variables are always set when you log in.

Bind locations§

Your $HOME and Scratch directories are bound automatically so they are available from inside your containers.

For more information on these options, have a look at the Apptainer documentation:

Backwards compatibility§

The singularity command still exists as a symbolic link and will run apptainer.

Apptainer will use the older $SINGULARITY_ environment variables if the $APPTAINER_ versions are not set.

build --remote is no longer available§

If you were building remote containers on Sylabs using

singularity build --remote

this functionality is no longer available in Apptainer. You can still log in to Sylabs via the web interface, build containers on it and then pull them down with Apptainer.

Building your own containers§

With Apptainer you can build containers directly on our clusters without additional permissions, as long as they use a local filesystem and not home or Scratch.

Our default setup uses $XDG_RUNTIME_DIR on the local disk of the login nodes, or $TMPDIR on a compute node (local disk on the node, on clusters that are not diskless).

If you try to build a container on a parallel filesystem, it will fail with a number of permissions errors.

Singularity§

Run singularity --version to see which version we currently have installed.

Singularity update to Apptainer

On Myriad, we are updating to Singularity to Apptainer. This update will occur on 14th November during a planned outage

This update may affect any containers that are currently downloaded, so users will have to test them to check their workflow still functions correctly after the update. We expect most to work as before, but cannot confirm this.

A Singularity command that will no longer be available in Apptainer is singularity build --remote. If any of you have workflows that depend on this, please email rc-support@ucl.ac.uk. We are currently looking into how we would provide equivalent functionality.

Updates to the other clusters will follow, dates tbc.

Set up cache locations and bind directories§

The cache directories should be set to somewhere in your space so they don't fill up /tmp on the login nodes.

The bindpath mentioned below specifies what directories are made available inside the container - only your home is bound by default so you need to add Scratch.

You can either use the singularity-env environment module for this, or run the commands manually.

module load singularity-env

or:

# Create a .singularity directory in your Scratch
mkdir $HOME/Scratch/.singularity

# Create cache subdirectories we will use / export
mkdir $HOME/Scratch/.singularity/tmp
mkdir $HOME/Scratch/.singularity/localcache
mkdir $HOME/Scratch/.singularity/pull

# Set all the Singularity cache dirs to Scratch
export SINGULARITY_CACHEDIR=$HOME/Scratch/.singularity
export SINGULARITY_TMPDIR=$SINGULARITY_CACHEDIR/tmp
export SINGULARITY_LOCALCACHEDIR=$SINGULARITY_CACHEDIR/localcache
export SINGULARITY_PULLFOLDER=$SINGULARITY_CACHEDIR/pull

# Bind your Scratch directory so it is accessible from inside the container
#      and the temporary storage jobs are allocated
export SINGULARITY_BINDPATH=/scratch/scratch/$USER,/tmpdir

Different subdirectories are being set for each cache so you can tell which files came from where.

You probably want to add those export statements to your .bashrc under # User specific aliases and functions so those environment variables are always set when you log in.

For more information on these options, have a look at the Singularity documentation:

Downloading and running a container§

Assuming you want to run an existing container, first you need to pull it from somewhere online that provides it:

# make sure we set up apptainer as above
module load apptainer

# get image from location and call it hello-world.sif in our current directory
apptainer pull hello-world.sif shub://vsoch/hello-world

Run the container.

apptainer run hello-world.sif

Run a specific command within our container.

apptainer exec hello-world.sif /bin/echo Hello World!

You can run containers inside jobscripts in the same way.

Useful links:

Docker containers§

You can use Singularity to run Docker containers. Docker itself is not suitable for use on a multi-user HPC system, but Singularity can convert and run Docker containers for you.

apptainer pull python-3.9.6.sif docker://python:3.9.6-slim-buster

In this case, apptainer pull is downloading a Docker image, and also converting it into a format that Apptainer uses. You then use apptainer run or apptainer exec on the .sif image as above.

Graphical containers in interactive jobs§

If you are trying to run a graphical application from inside a container in an interactive job and it is failing with errors about not being able to open a display, you will need to:

  • ssh in to the cluster with X-forwarding on as normal
  • request an interactive job using qrsh
  • bind mount your $HOME/.Xauthority file inside the container

To do the bind mount, you could add it to your $APPTAINER_BINDPATH

export APPTAINER_BINDPATH=$APPTAINER_BINDPATH,$HOME/.Xauthority

or you can pass it in with the –-bind=$HOME/.Xauthority option to singularity shell or singularity exec.