Containers with Enroot and Pyxis
A TIR Slurm Cluster can run any Docker or OCI container image inside a Slurm job. You do not build
a custom cluster image, and you do not install anything: the cluster ships with
Enroot as an unprivileged container runtime and
Pyxis as the Slurm SPANK plugin that wires Enroot into srun
and sbatch.
The result is one extra flag on your job:
srun --container-image=nvcr.io/nvidia/pytorch:25.09-py3 python train.py
Everything else stays normal Slurm — same partitions, same --gres, same accounting, same
Jobs tab.
Why Enroot and Pyxis
Distributed training environments are hard to reproduce. Enroot and Pyxis let you take the exact image your team already builds — the one from NGC, from your CI, from Docker Hub — and run it as the job environment, without giving up Slurm scheduling.
| Property | What it means for you |
|---|---|
| Unprivileged | Enroot runs containers as your own user. There is no Docker daemon on the nodes and you do not need root inside the job |
| No cluster rebuild | The image is chosen per job, not per cluster. Two jobs on the same cluster can use completely different CUDA and framework versions |
| Native GPU access | The container sees the node's GPUs directly; nvidia-smi and CUDA work with no device plumbing from you |
| Slurm-native | --container-image is just another srun/sbatch flag. sbatch, job arrays, dependencies, --gres, accounting and preemption all behave normally |
Multi-node in one srun | A single srun launches the same container on every node of the allocation, with PMIx available for MPI and NCCL ranks |
| Reusable | --container-name keeps an imported container around so later steps and jobs start instantly instead of re-importing |
Your job is not a Kubernetes pod and there is no docker run. Enroot converts the image into a flat
root filesystem and runs your command in it with user namespaces. Practical differences: no
--privileged, no port publishing, no container restart policy, and the container's lifetime is the
job step's lifetime.
How It Fits Together
you login node worker nodes (slinky-0 … slinky-N)
─── ────────── ─────────────────────────────────
sbatch job.sh ────────► slurmctld schedules ──────► slurmd starts the job step
│
├─ Pyxis reads --container-* flags
├─ Enroot imports / reuses the image
└─ your command runs inside the container
with the node's GPUs and your
PFS / SFS mounts visible
Three pieces make it work, and all three are managed for you:
| Piece | What it is | Who owns it |
|---|---|---|
| Pyxis SPANK plugin | Registers --container-image, --container-mounts, --container-name and friends on srun/sbatch. Enabled by plugstack.conf containing include /usr/share/pyxis/* | Platform-managed. There is no console setting, and plugstack.conf is not editable |
| Enroot runtime | Imports images and starts containers, unprivileged | Baked into the cluster's login and compute images |
| Slurm prolog | The platform runs a prolog on every node that writes an Enroot mount rule so Slurm's own config cache is readable inside containers | Platform-managed; it lands in its own file and never interferes with your prolog scripts |
Because of that platform mount rule, every container on the cluster has
/var/spool/slurmd/conf-cache bind-mounted read-only. It is harmless — you will only notice it if you
list /var/spool/slurmd inside a job.
Confirm Pyxis Is Available
Pyxis needs both the plugin configuration and a Pyxis-enabled cluster image. Both are standard on GPU Slurm Clusters, but it costs nothing to check before you build a job script around it.
ssh root@<cluster-ip>
# If Pyxis is loaded, srun advertises the container options
srun --help | grep -i container
You should see --container-image, --container-mounts, --container-name and --container-workdir
among the options.
# Enroot itself
enroot version
--container-image is not recognisedsrun: unrecognized option '--container-image' means the plugin is not loaded on that cluster — most
often because the selected cluster image version is not a Pyxis build. Check the Image Version on
the Details tab, try
Update Image to move to a current version,
and contact support if it persists. Do not try to install Pyxis yourself — plugstack.conf is
platform-managed and any local edit is overwritten the next time the cluster is re-rendered.
Check the exact flag set your image supports
Enroot and Pyxis versions differ between cluster image versions, and the flags below marked standard Pyxis may or may not be present on yours. The authoritative answer is always on your own cluster:
srun --help | grep -i container
cat /etc/enroot/enroot.conf 2>/dev/null; enroot start --help | head -40
That last command is also how you find out which paths Enroot uses for its cache and data on your cluster — which matters for caching images.
Your First Container Job
This runs nvidia-smi inside an NGC PyTorch container on one GPU. It is the fastest way to prove the
whole path works.
ssh root@<cluster-ip>
Create container-hello.sh on a shared filesystem — for example your PFS mount at /pfs:
#!/bin/bash
#SBATCH --job-name=container-hello
#SBATCH --partition=all
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --gres=gpu:1
#SBATCH --time=00:10:00
#SBATCH --output=/pfs/logs/%x-%j.out
#SBATCH --error=/pfs/logs/%x-%j.err
srun --container-image=nvcr.io/nvidia/pytorch:25.09-py3 \
--container-mounts=/pfs:/pfs \
--container-workdir=/pfs \
bash -lc 'nvidia-smi && python -c "import torch; print(torch.__version__, torch.cuda.device_count())"'
Submit it and watch it run:
mkdir -p /pfs/logs
sbatch container-hello.sh
squeue
The first run spends a minute or two importing the image. Later runs against the same image are much faster — and faster still if you cache the image as a squash file.
The image is fetched from the node running the job, so a registry pull is outbound traffic subject to the attached security group. The default SSH group allows it. With a group that restricts egress, either allow what the registry needs or cache the image as a squash file so jobs never pull.
--container-mounts=/pfs:/pfs is doing real work here. Your PFS, SFS and dataset mounts exist on the
node, but a container starts from the image's own filesystem — pass the paths you need. See
Run containers in jobs.
Where to Go Next
| Page | What it covers |
|---|---|
| Run containers in jobs | Every --container-* flag, mounting storage, environment variables, interactive sessions, worked sbatch examples |
| Cache and manage images | enroot import, squash files on shared storage, named containers, private registries, avoiding repeated pulls |
| Multi-node container training | PMIx, NCCL, InfiniBand, torchrun across nodes, a full 8-GPU-per-node example |
| Troubleshoot container jobs | Import failures, missing GPUs, NCCL errors, permission problems |