Skip to main content

Podman

What is Podman?

Podman is a container management system, similar to Docker. Containers are portable environments that allow you to run software with all required dependencies in a portable and reproducible way. Podman does not require a daemon, can run without root privileges, and is well suited for shared systems such as HPC clusters.

Getting Started

On systems that use SLURM, Podman must store its runtime and image data in locations that are writable and appropriate for the node type. Login nodes and compute nodes differ in filesystem layout and do not provide systemd user sessions inside SLURM jobs, so Podman must be configured explicitly.

This configuration allows users to:

  • Build containers on login nodes
  • Export containers for later use in SLURM jobs
  • Run containers inside jobs using job-local temporary storage
  • Avoid conflicts when logged into multiple nodes simultaneously

First, add the following to ~/.bashrc:

# set podman paths
if [ -n "${SLURM_JOBID}" -a -n "${TMPDIR}" ]
then
export XDG_RUNTIME_DIR=${TMPDIR}
export XDG_DATA_HOME=${TMPDIR}
export _PODMAN_GRAPHROOT=${XDG_DATA_HOME}/containers/storage
export _PODMAN_RUNROOT=${XDG_RUNTIME_DIR}/containers/run
else
export _PODMAN_GRAPHROOT=${HOME}/.local/share/containers/storage
export _PODMAN_RUNROOT=${XDG_RUNTIME_DIR}/run
fi

# create podman directories if they don't exist
[[ -d $_PODMAN_GRAPHROOT ]] || mkdir -p $_PODMAN_GRAPHROOT
[[ -d $_PODMAN_RUNROOT ]] || mkdir -p $_PODMAN_RUNROOT

This ensures Podman uses persistent storage on login nodes and job-local temporary storage on compute nodes. The runtime directory (runroot) is always local to the host, preventing conflicts across nodes.

Next, create the file ~/.config/containers/storage.conf with the following contents:

[storage]
driver = "overlay"
runroot = "${_PODMAN_RUNROOT}"
graphroot = "${_PODMAN_GRAPHROOT}"

[storage.options.overlay]
force_mask = "private"
mount_program = "/usr/bin/fuse-overlayfs"

Creating an Image

Users should build container images on the login nodes and export them for use in SLURM jobs.

Create a file named Containerfile.

Basic Containerfile format:

FROM <base image>
WORKDIR <working directory>
RUN <commands to be run during image building>
EXPOSE <port>
CMD <commands to be run when the container starts>

Example Containerfile:

FROM alpine:latest
CMD ["echo", "Hello World!"]

Build the image from the directory containing the Containerfile:

podman build -t helloworld ./helloworld/

Save the image to a tar file:

podman save -o helloworld.tar helloworld:latest

Compress the image to save space:

gzip helloworld.tar

The resulting archive can be copied to a working directory and loaded inside a SLURM job.

Note: Containers may be built for a different CPU architecture (for example, linux/arm64 for GH200 nodes) by using the --platform option to podman build or by specifying the platform in the FROM line of the Containerfile.

Running a Container in a SLURM Job

When running containers in unattended SLURM jobs, Podman paths must be set explicitly since systemd user sessions are not available.

Example SLURM job script:

#!/bin/bash
#SBATCH --job-name=helloworld
#SBATCH --output=helloworld-%j.out
#SBATCH --partition=cmp
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem-per-cpu=4G
#SBATCH --time=00:05:00
#SBATCH --mail-user=wavesupport@scu.edu

#SBATCH --mail-type=END

# set podman paths
export XDG_RUNTIME_DIR=${TMPDIR}
export XDG_DATA_HOME=${TMPDIR}
export _PODMAN_GRAPHROOT=${XDG_DATA_HOME}/containers/storage
export _PODMAN_RUNROOT=${XDG_RUNTIME_DIR}/containers/run

# create podman directories if they don't exist
[[ -d $_PODMAN_GRAPHROOT ]] || mkdir -p $_PODMAN_GRAPHROOT
[[ -d $_PODMAN_RUNROOT ]] || mkdir -p $_PODMAN_RUNROOT

# load the container image
podman load --cgroup-manager cgroupfs --input ./helloworld.tar.gz

# run the container
podman run --cgroup-manager cgroupfs --userns keep-id helloworld

When running inside SLURM jobs, the cgroup manager must be set to cgroupfs because systemd is not available.

A warning similar to the following may appear in job output:

Failed to add pause process to systemd sandbox cgroup: dial unix /run/user/<UID>/bus: connect: no such file or directory

This warning is expected and does not affect container execution.

Mounting Directories into a Container

Mounting allows a container to access files from the host system.

Basic mount syntax:

podman run -v <host path>:<container path> <image>

Example:

podman run -v $HOME:/home/user helloworld