Run containers in jobs

Run a Slurm job inside a Docker or OCI container image of your choice, with the GPUs Slurm allocated to the job and your storage visible inside the container. The cluster ships with Enroot, an unprivileged container runtime, and Pyxis, the Slurm plugin that adds the --container-* options to srun and sbatch. Running a container takes one extra flag on an otherwise normal job:

srun --container-image=nvcr.io/nvidia/pytorch:25.09-py3 python train.py

Everything else stays normal Slurm: the same partitions, the same --gres, the same accounting, and the same Jobs tab.

Before you begin

• A cluster that is Running, and a shell on its login node. See Connect to a cluster.
• A storage volume mounted on the cluster. These commands use /shared as the mount path; use the one you set.

How a container job behaves

Enroot and Pyxis come installed and ready to use, and Enroot takes the place of a Docker daemon. It converts the image into a flat root filesystem and runs your command in it using Linux user namespaces, so the job runs unprivileged inside the container. Inside the job:

• nvidia-smi shows the GPUs Slurm allocated to the job.
• Two jobs on the same cluster can use different images, so two teams can run different framework versions at the same time.
• The container runs the same way on every node of a multi-node allocation.
• Only the paths you pass with --container-mounts are visible inside the container.

A container job is a step of a Slurm job rather than a separate server. The container lives as long as the job step and leaves out Docker’s server features: docker run, --privileged, port publishing, and restart policies.

The Image you chose when you created the cluster is a different thing. It selects the cluster’s own Slurm, drivers, and operating system. Your job’s container is chosen for each job.

E2E manages the Pyxis plugin’s configuration file, plugstack.conf, so it is ready to use. The next change to the cluster, such as a scale, an image update, a storage mount, or a Slurm configuration change, overwrites any hand edit of that file.

Check that Pyxis is loaded

Before you build a job script around the container flags, check that they are there. On the login node, run:

srun --help | grep -i container
enroot version

The output lists --container-image, --container-mounts, --container-name, and --container-workdir, and enroot version prints the Enroot version, which confirms the runtime is installed. If the flags are missing, Pyxis is missing on the cluster, and srun answers srun: unrecognized option '--container-image'. Check the image version on the Details tab and try Update Image to move to a current version. If the cluster still lacks the flags, contact support with the cluster name and image version. See Update a cluster.

Use the container flags

Pass these flags to srun. In a batch script, put srun --container-image=... in the script body rather than in an #SBATCH line. It shows which step runs in the container, and it lets you run setup commands on the host first.

Flag	What it does
--container-image=<ref>	Names the image to run: a registry reference such as nvcr.io/nvidia/pytorch:25.09-py3, or the path to a squash file such as /shared/images/pytorch.sqsh. A path that starts with / is read as a file instead of pulled.
--container-name=<name>	Keeps the imported container on that node, so a later step or job on the same node starts at once instead of importing again. The name exists only on that node.
--container-mounts=<src>:<dst>[,...]	Bind-mounts host paths into the container. Separate several mounts with commas.
--container-workdir=<path>	Sets the working directory inside the container.
--container-writable, --container-readonly	Allows or forbids writes to the container’s root filesystem.
--container-remap-root, --no-container-remap-root	Sets whether your user appears as root inside the container.
--container-save=<path.sqsh>	Writes the container’s filesystem back to a squash file after the step.
--container-env=<VAR[,VAR...]>	Passes the named variables through, instead of the default environment handling.
--container-mount-home, --no-container-mount-home	Mounts your home directory, or skips it.
--container-entrypoint, --no-container-entrypoint	Runs the image’s declared entrypoint, or skips it.
--mpi=pmix	Uses the PMIx plugin to launch processes, which is what connects MPI ranks across nodes inside containers. Use it for MPI ranks, and it is recommended for any multi-node container job.

Which of the last six flags a cluster has depends on its Pyxis version. Before you use a flag in a production script, confirm that it exists on your own cluster with srun --help | grep -i container.

Mount your data

A container starts from the image’s own filesystem. Your storage is on the node, and you pass in the paths the job needs:

srun --container-image=nvcr.io/nvidia/pytorch:25.09-py3 \
     --container-mounts=/shared:/shared \
     --container-workdir=/shared/project \
     python train.py

Forgetting --container-mounts is the most common mistake in a container job. The job starts, runs, and reports that your dataset does not exist: the mount is fine on the node, and the job lacks only the flag that passes it in.

You can remap a path. --container-mounts=/shared/datasets/imagenet:/data makes the dataset appear at /data inside the container, which helps when your code expects a fixed path.

You mount and unmount storage on the Volumes tab. See Mount and unmount volumes. This is what each kind of storage looks like on a node:

Storage	Writable	Shared across nodes
Parallel File System or Shared File System mount	Yes	Yes. The same filesystem at the same path on the login node and every worker.
Dataset mount	Read-only	Yes.
/dev/shm	Yes	One node only. It is memory-backed and lost on restart.
Node-local scratch such as /tmp	Yes	One node only. It is erased when the node is recreated.

Write checkpoints, logs, and model output to shared storage. The container’s filesystem is discarded when the step ends, and a node-local path stays on one node and is erased on a restart. Writing to either of them is the commonest way to lose a training run.

For a job that uses RDMA, which includes most multi-node NCCL training, mount the device nodes explicitly, because a container includes /dev/infiniband only when you mount it:

--container-mounts=/shared:/shared,/dev/infiniband:/dev/infiniband

For the full multi-node picture, see Run multi-node training.

Set environment variables

Variables you export before srun are visible to the process inside the container. Slurm’s own job variables, such as the job ID, node index, and task count, are available there too, so you can derive ranks and build per-job output paths from them. Run multi-node training lists them:

export NCCL_DEBUG=INFO
srun --container-image=nvcr.io/nvidia/pytorch:25.09-py3 python train.py

NCCL and UCX tuning is left to you: the defaults come from the image, so a multi-node run over RDMA normally needs you to set several variables. Set environment for a job in the job script itself, or build it into the image. A variable set in a prolog does not reach the job.

If your job needs a secret at runtime, read it from a file on mounted storage, and keep it out of job scripts and images.

Use the GPUs inside the container

Ask Slurm for GPUs the usual way, and the container inherits them:

#SBATCH --gres=gpu:8            # 8 GPUs on the node
#SBATCH --gpus-per-node=8       # the same request, written another way

--gres counts GPUs per node, so on a four-node job --gres=gpu:8 means eight GPUs on each node. Inside the container, nvidia-smi shows only the GPUs the job was allocated:

srun --gres=gpu:2 --container-image=nvcr.io/nvidia/pytorch:25.09-py3 nvidia-smi -L

The cluster advertises its GPUs to Slurm as a GRES with the card type, so you can see what is available before you submit. Use the GRES column of this command, or scontrol show node:

sinfo -o "%20N %10c %10m %25G %10T"

scontrol show node slinky-0

Choose an image whose CUDA runtime matches the GPU generation you are using. An image built for an older CUDA version may run poorly, or fail to start, on newer hardware. NGC tags state the CUDA version they carry.

Try the worked examples

Each example runs on the default partition.

Train on one node with eight GPUs

#!/bin/bash
#SBATCH --job-name=train-8gpu
#SBATCH --partition=all
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --gpus-per-node=8
#SBATCH --cpus-per-task=64
#SBATCH --time=08:00:00
#SBATCH --output=/shared/logs/%x-%j.out
#SBATCH --error=/shared/logs/%x-%j.err

IMAGE=nvcr.io/nvidia/pytorch:25.09-py3

export NCCL_DEBUG=WARN

srun --container-image="$IMAGE" \
     --container-name=pytorch-train \
     --container-mounts=/shared:/shared \
     --container-workdir=/shared/project \
     torchrun --standalone --nproc_per_node=8 train.py \
       --data /shared/datasets/imagenet \
       --checkpoint-dir /shared/checkpoints/${SLURM_JOB_ID}

Run a one-off command in a container

srun --gres=gpu:1 --time=00:15:00 \
     --container-image=nvcr.io/nvidia/pytorch:25.09-py3 \
     --container-mounts=/shared:/shared \
     python /shared/scripts/validate_dataset.py

Run a hyperparameter sweep as a job array

#!/bin/bash
#SBATCH --job-name=sweep
#SBATCH --partition=all
#SBATCH --array=0-15
#SBATCH --nodes=1
#SBATCH --gpus-per-node=1
#SBATCH --time=02:00:00
#SBATCH --output=/shared/logs/%x-%A_%a.out

LRS=(0.1 0.03 0.01 0.003)
BSS=(32 64 128 256)
LR=${LRS[$((SLURM_ARRAY_TASK_ID / 4))]}
BS=${BSS[$((SLURM_ARRAY_TASK_ID % 4))]}

srun --container-image=/shared/images/pytorch-25.09.sqsh \
     --container-mounts=/shared:/shared \
     --container-workdir=/shared/project \
     python train.py --lr "$LR" --batch-size "$BS" \
       --out /shared/runs/lr${LR}_bs${BS}

Using a squash file instead of a registry reference means the 16 array tasks read the same file on shared storage instead of each pulling the image. See Cache container images.

Run a CPU-only step

srun --nodes=1 --ntasks=8 --time=01:00:00 \
     --container-image=python:3.12-slim \
     --container-mounts=/shared:/shared \
     python /shared/scripts/shard_dataset.py --workers 8

Minimal images such as python:3.12-slim can lack bash. Use sh -c instead of bash -lc, or choose a fuller base image.

Start an interactive session

To debug, allocate a node and get a shell inside the container:

srun --partition=all --nodes=1 --gres=gpu:1 --time=01:00:00 --pty \
     --container-image=nvcr.io/nvidia/pytorch:25.09-py3 \
     --container-mounts=/shared:/shared \
     --container-workdir=/shared \
     bash

You are inside the container on a compute node with a GPU, and nvidia-smi, python, and your data are all there. Exit the shell to release the allocation.

For a longer session, allocate first and attach steps to the allocation:

salloc --partition=all --nodes=1 --gres=gpu:1 --time=04:00:00

srun --container-image=nvcr.io/nvidia/pytorch:25.09-py3 --container-mounts=/shared:/shared --pty bash

An idle salloc keeps its GPUs out of the queue for other jobs. Always set --time and exit when you are done. squeue -u $USER shows what you are holding.

Pull from a registry

The node that runs the job pulls the image, so a registry pull is outbound traffic from that node, governed by the security group attached to the cluster. A group that restricts outbound traffic can break image pulls, and the failure looks like a plain network timeout. Either allow what the registry needs, or import the image once to a squash file that jobs read from shared storage. See Cache container images.

To stop every job from pulling the same image, cache container images. To train across nodes, see Run multi-node training. For import failures, missing GPUs, or NCCL errors, see Troubleshooting containers on TIR Slurm Cluster.