Containers (Apptainer)

Introduction

What are containers?

Containers provide a way to package software together with all required dependencies so it can run in a reproducible environment. Containers use OS virtualization to isolate processes and control their access to system resources. A container is stored in a container image, which is simply a set of files that defines the runtime environment. A container image contains the applications, libraries, and runtime environment, while the host system provides the kernel.

Why Apptainer?

The best-known container implementation is Docker. However, Docker typically requires root privileges, which introduces security concerns on shared HPC systems. For this reason, Docker is generally not available on HPC clusters.

Apptainer (formerly Singularity) provides a container solution designed specifically for HPC environments. Containers can be executed by regular users and integrate well with cluster resources such as shared filesystems, GPUs, and high-performance networks.

Advantages of Apptainer:

• Containers can be executed without root privileges.

• Images are stored as a single portable and immutable file (SIF format).

• Integration with HPC resources such as GPUs, networks, and filesystems.

• Compatibility with Docker/OCI container images.

• Reproducible and verifiable images through immutable builds and optional image signing.

Apptainer can run containers built with Docker, and it can also build container images on top of base Docker images.

When should I use containers?

If the software you want to use is already available on the HPC system as a module, it is usually preferable to use the provided installation. HPC software is often compiled specifically for the hardware and may offer better performance. Containers are particularly useful in the following situations:

• Quickly testing a containerized application before starting a performance-optimized installation.

• Running software that is difficult to install on HPC systems.

• Using legacy applications that require outdated dependencies.

• Ensuring reproducibility of a workflow developed elsewhere.

• Running software built for a different (version of a) Linux distribution.

• Creating portable workflows that can run both on HPC systems and cloud platforms.

• Reducing the number of files (inodes) by packaging software into a single container image.

Limitations of containers

Containers are not always the best solution. Potential drawbacks include:

• Interaction with software outside the container can be difficult or infeasible (e.g. environment modules, Open OnDemand).

• MPI-based applications require compatible MPI libraries between the host and the container.

• To maximize portability, containers are often built for generic CPU architectures, which may reduce performance compared to optimized HPC builds.

• Containers built for one CPU architecture (e.g. x86-64, Intel/AMD CPUs) cannot run on another architecture (e.g. AArch64, Arm CPUs).

• Security risks arise when you don’t know who created a container image or exactly what software is inside it.

Apptainer on VSC clusters

Users can both run and build Apptainer containers on the VSC clusters.

Before starting, it is important to set environment variables for the Apptainer cache and temporary directories. We recommend adding the following code snippet to your ~/.bashrc so these are set automatically:

export APPTAINER_CACHEDIR=/tmp/$USER/apptainer_cachedir
export APPTAINER_TMPDIR=/tmp/$USER/apptainer_tmpdir
mkdir -p $APPTAINER_TMPDIR

APPTAINER_CACHEDIR stores data that can be reused across Apptainer runs (layers, images), while APPTAINER_TMPDIR is a temporary workspace for Apptainer.

Note

Make sure to always build and/or run your containers on a compute node with enough available RAM memory.

Also, images tend to be very large, so store them in a directory where you have sufficient quota, e.g. $VSC_DATA.

Pulling Apptainer images

Apptainer can download or pull images directly from several sources. However, it is important to understand where the image comes from and what it contains. Questions to consider include:

• Which software versions are included?

• How was the software built and configured?

• Is the source of the container trustworthy?

When pulling images from public registries such as Docker Hub, it is recommended to use images from verified publishers whenever possible. Here is a list of some well-known public registries:

Table 7 Public Container Registries

Container Registry	URI Prefix
Docker Hub	docker://
NVIDIA Container Registry	docker://nvcr.io/nvidia
AMD Infinity Hub	docker://rocm
GitHub Container Registry	docker://ghcr.io

The following example pulls an Ubuntu image from Docker Hub and saves it as an immutable Apptainer SIF image that can be executed on the cluster:

apptainer pull ubuntu-24.04.sif docker://ubuntu:24.04

Using Apptainer containers

Container startup behavior

Containers can be executed in several ways:

Table 8 Apptainer Primary Commands

Command	Action
apptainer shell	Start an interactive shell inside the container
apptainer exec	Execute a command inside the container
apptainer run	Run the container using its runscript

Containers contain startup scripts that define their environment and behavior:

Environment scripts

• located inside the container at /.singularity.d/env/

• always executed: apptainer shell, apptainer exec, apptainer run

Container runscript

• located inside the container at /.singularity.d/runscript

• only executed when using apptainer run

If the last line of the runscript contains exec "$@" (or equivalent), we can also use apptainer run to execute a command inside the container, which will execute after the runscript has run.

Inspecting containers

We can inspect the metadata and configuration of a container image with apptainer inspect:

# inspect labels
apptainer inspect ubuntu-24.04.sif
# inspect runscript
apptainer inspect --runscript ubuntu-24.04.sif
# inspect environment
apptainer inspect --environment ubuntu-24.04.sif

Running commands in a container

Executing a command inside a container can be done with apptainer exec:

apptainer exec ubuntu-24.04.sif <command>
# or (if the runscript supports it)
apptainer run ubuntu-24.04.sif <command>

# example: get info about the container OS
apptainer exec ubuntu-24.04.sif cat /etc/lsb-release

Starting an interactive shell inside the container can be done with apptainer shell. When inside the container, the prompt changes to Apptainer>, indicating that the container shell is ready to accept commands:

apptainer shell ubuntu-24.04.sif
Apptainer>

Bind mounts

By default, the user’s home directory and the current working directory are bind-mounted so they can be accessed from inside the container. Additional host directories can be mounted using the --bind or -B option. For example, to mount the $VSC_SCRATCH directory:

apptainer exec -B $VSC_SCRATCH ubuntu-24.04.sif <command>

If needed, we can specify a different path inside the container. For example, to mount $VSC_SCRATCH as /scratch:

apptainer exec -B $VSC_SCRATCH:/scratch ubuntu-24.04.sif <command>

VSC sites may have enabled additional default mount points, so you don’t always have to add them yourself. We can use the following command to see which paths are currently mounted inside the container:

apptainer exec ubuntu-24.04.sif findmnt -no TARGET

Environment variables

Environment variables defined on the host are automatically passed into the container (exceptions: $PATH and $LD_LIBRARY_PATH). The example below shows that $MY_VAR is defined inside the container:

export MY_VAR=my_value
apptainer run ubuntu-24.04.sif bash -c 'echo $MY_VAR'  # prints my_value

If needed, we can set a different value inside the container using either the $APPTAINERENV_*** environment variable or the --env option:

APPTAINERENV_MY_VAR=other_value apptainer run ubuntu-24.04.sif bash -c 'echo $MY_VAR'  # prints other_value
# or:
apptainer run ubuntu-24.04.sif --env MY_VAR=other_value bash -c 'echo $MY_VAR'  # prints other_value

Running containers in a job

Apptainer containers can be part of any workflow. The following job script runs an example tblite script with a tblite image that was created from an Apptainer definition file:

#!/bin/bash
#SBATCH --ntasks=1
#SBATCH --time=30:00

apptainer run tblite-0.4.0.sif python tblite-single-point-GFN2-xTB.py

Ensure that the container has access to all the required directories by providing additional bindings if necessary.

Common issues with Docker images

Although compatibility with Docker is high, users may experience issues when running Docker images with Apptainer. Below are some common problems and their possible workarounds:

Applications trying to write inside the image

Apptainer images are immutable, so writing to a directory inside the image will fail. There are two ways to work around this:

• Copy or move the directory from the image to the host, and bind-mount the copied directory back into the container:

apptainer run --bind <host-dir>:<container-dir>

• Use a temporary overlay which will be discarded on exit:

apptainer run --writable-tmpfs

Interference from the host environment

Environment variables such as $PYTHONPATH may interfere with container software. To avoid this, users can either override specific environment variables, or run the container with a clean environment:

apptainer run --cleanenv

Default bind mounts

Apptainer automatically binds directories such as $HOME into the container. In some cases, an application expects to find specific files installed in those directories inside the container, which it cannot access due to the mount. To avoid this, we can disable binding the home directory or disable all default binds and manually bind only what is needed:

apptainer run --no-home
# or
apptainer run --contain --bind <dir1> --bind <dir2>

Processes expecting root privileges

Some Docker images assume that applications run as root. Apptainer provides a --fakeroot option that can help in these situations:

apptainer run --fakeroot

Building Apptainer containers

Containers can be built in several ways:

• interactively from a base image

• from an Apptainer definition file

• from a Docker image via a Dockerfile

• using hpc-container-wrapper

• using remote build services

Many containers can be built on the VSC clusters without root privileges using the --fakeroot build option. However, due to inherent limitations of fakeroot, some containers may fail to build. In that case, you can build the image on a local machine where you have root privileges. Apptainer only runs under Linux, so you’ll need to use a virtual machine when using Windows (e.g. WSL) or macOS. For detailed instructions, see the Apptainer Quick Start guide. Once your image is built, you can transfer it to the VSC infrastructure.

Alternatively, you can use remote build services to build your images. The Sylabs Remote Builder builds images from a user-provided definition file. The Seqera platform allows to simply select a set of Conda or Python packages to be built into an image.

In the following sections, we will build the tblite software package with Python bindings on the VSC clusters. Before starting, ensure you have set the required environment variables as explained in the Apptainer on VSC clusters section.

Building interactively from base image

1. Download a base image and store it as a sandbox. A sandbox is a writable container directory structure. The following example creates a directory called my_sandbox and installs an Ubuntu container image in it:

   apptainer build --sandbox my_sandbox docker://ubuntu:24.04
   

2. Start an interactive shell in the container. The --fakeroot option is required when building as non-root user on the VSC clusters. Inside the container, the my_sandbox directory becomes the root directory (/). We can now make changes, such as installing packages:

   apptainer shell --writable --fakeroot my_sandbox
   Apptainer> apt-get update && apt-get install python3
   Apptainer> exit  # exit the container
   

3. Make changes from the host. From the host, we can also make changes by traversing the sandbox directory structure (e.g. updating the runscript):

   nano my_sandbox/.singularity.d/runscript
   

4. Create immutable SIF image from sandbox. When we’re finished making changes, we can convert the sandbox to a SIF image:

   apptainer build my_image.sif my_sandbox
   

Building from Apptainer definition file

For reproducibility, containers should ideally be built from Apptainer Definition Files. A definition file specifies the base image and the commands required to build the container. See also the Example Apptainer definition files section.

Another advantage of definition files is that they can be shared easily.

Note

A definition file does not guarantee reproducibility by itself. It is important to specify the exact versions for the base container and any installed software packages. Additionally, downloading files during installation risks breaking reproducibility if those external resources become unavailable.

In the example below, we’ll create a container for tblite. The definition file can be found in the gssi-training repo at tblite-0.4.0.def.

1. Create or download Apptainer definition file.

2. Create Apptainer image from the definition file. We can create the image in one step, or in two steps via a sandbox to verify the container before finalizing the SIF image file:

   # option1: in one step
   apptainer build --fakeroot tblite-0.4.0.sif tblite-0.4.0.def
   
   # option2: in two steps via sandbox
   apptainer build --fakeroot --sandbox my_sandbox tblite-0.4.0.def
   apptainer build tblite-0.4.0.sif my_sandbox
   

Building from Docker image via Dockerfile

Users familiar with Dockerfiles can create a Docker image from a Dockerfile and then convert it to Apptainer. This method requires a machine with Docker (and root privileges) or Podman. Windows users can use WSL.

1. Write Dockerfile.

2. Create Docker image from the Dockerfile.

   sudo docker build . -t my_docker_image
   

3. Create Docker archive from Docker image.

   sudo docker save my_docker_image -o my_docker_archive.tar
   sudo chown $USER:$USER my_docker_archive.tar
   

4. Create Apptainer image from Docker archive. We can create the image in one step, or in two steps via a sandbox to verify the container before finalizing the SIF image file:

   # option1: in one step
   apptainer build my_image.sif docker-archive:my_docker_archive.tar
   
   # option2: in two steps via sandbox
   apptainer build --sandbox my_sandbox docker-archive:my_docker_archive.tar
   apptainer build my_image.sif my_sandbox
   

Building with hpc-container-wrapper

hpc-container-wrapper is a tool that automates the creation of an Apptainer image and provides wrapper scripts to call executables within the container environment. It supports building both Conda and Pip packages.

In the example below, we’ll create a container for tblite with conda.

1. Create Conda environment file environment.yml:

   # environment.yaml
   name: tblite
   channels:
   - conda-forge
   dependencies:
   - tblite-python=0.4.0
   

2. Load hpc-container-wrapper environment module:

   module load hpc-container-wrapper/<VERSION>
   

3. Create Apptainer container and wrappers in new tblite-0.4.0 directory:

   conda-containerize new --prefix tblite-0.4.0 environment.yml
   

Example wrappers usage

# Add path to bin directory to $PATH
export PATH="$PWD/tblite-0.4.0/bin:$PATH"

# Verify that tblite and python executables from the image are used
which tblite  # $PWD/tblite-0.4.0/bin/tblite
which python  # $PWD/tblite-0.4.0/bin/python

# Verify that tblite python package from image is used
python -c 'import tblite; print(tblite.__file__)'  # /LOCAL_TYKKY_QcubNaZ/miniforge/envs/env1/lib/python3.13/site-packages/tblite/__init__.py

If needed, we can also update an image created with hpc-container-wrapper. The following example adds the beautifulsoup4 conda package:

1. Write update.sh script:

   # update.sh
   conda install -c conda-forge beautifulsoup4
   

2. Update image installed in tblite-0.4.0 directory:

   conda-containerize update --post-install update.sh tblite-0.4.0
   

Example Apptainer definition files

• Minimal example of an Apptainer definition file:

  Bootstrap: docker
  From: ubuntu:22.04
  
  %post
      apt-get update
      apt-get install -y grace
  
  %runscript
      /usr/bin/xmgrace
  

  The resulting image will be based on Ubuntu 22.04. Once bootstrapped, the commands in the %post section are executed to install the Grace plotting package.

  Note

  This example is intended to illustrate that very old software that is no longer maintained can successfully be run on modern infrastructure. It is not intended to encourage you to use Grace in this container.

• Example definition file using Conda environment file conda_env.yml to create a Conda environment in an Apptainer container:

  Bootstrap: docker
  From: condaforge/miniforge3
  
  %files
      conda_env.yml
  
  %post
      /opt/conda/bin/conda env create -n conda_env -f conda_env.yml
  
  %runscript
      . /opt/conda/etc/profile.d/conda.sh
      conda activate conda_env
      exec "$@"
  

  The exec "$@" line at the bottom of the runscript allows apptainer run to accept user commands, such as python --version.

Sylabs Remote Builder

We can build images on the Sylabs cloud website and download them to the VSC infrastructure. This requires a Sylabs account. Once created, use the Sylabs Remote Builder to generate an image from an Apptainer definition file. This service uses SingularityCE, which is highly compatible with Apptainer.

If the build succeeds, pull the image using the library URI,

apptainer pull library://<username>/<project>/<image_name>:<tag>

Remote builds offer several advantages:

• They are platform-independent and only require a web browser.

• They can be easily shared with other users.

However, local builds offer more flexibility, particularly when interactive setup is required.

GPU-enabled containers

Apptainer can run GPU-enabled containers by exposing GPU devices, drivers, and libraries (CUDA/ROCm) from the host system.

• NVIDIA GPUs: apptainer run --nv

• AMD GPUs: apptainer run --rocm

Requirements when building or running GPU containers:

• The GPU applications in the container must match the host GPU and driver’s supported CUDA/ROCm version and compute capability.

• The container OS should be from a similar generation as the host OS.

Recommendations for VSC clusters:

• Use a prebuilt CUDA or ROCm container image as a base image, which can be obained by pulling from a suitable container registry.

• Select a container built for a CUDA version that matches one of the available CUDA environment modules on the cluster.

By default, all GPUs visible on the host node are also visible inside the container.

MPI-enabled containers (advanced)

Running MPI applications in containers requires compatibility between the MPI implementation in the container and on the host. Two common approaches are:

Hybrid model

• The MPI library is installed inside the container.

• The MPI implementation inside the container and on the host must be compatible.

mpirun -n $SLURM_NTASKS apptainer exec <image_name> <executable>

Bind model

• The MPI library is not included in the container but is bind-mounted from the host system.

• The MPI library used to compile the application in the container must be compatible with the host library.

MPI_DIR=/path/to/MPI-libraries
mpirun -n $SLURM_NTASKS apptainer exec --bind "$MPI_DIR" <image_name> <executable>

For help with MPI-enabled containers, contact user support.