--- title: "Run containers in jobs" description: "Run any Docker or OCI image inside a Slurm job on a TIR Slurm Cluster with one extra flag, mounting your storage and using the allocated GPUs." content_type: "how-to" product: "tir-slurm-cluster" platform: "TIR" url: "/tir/tir-slurm-cluster/how-to/run-containers-in-jobs/" topic: ["run-jobs"] --- # Run containers in jobs Run a Slurm job inside a Docker or OCI container image of your choice, with the GPUs Slurm allocated to the job and your storage visible inside the container. The cluster ships with Enroot, an unprivileged container runtime, and Pyxis, the Slurm plugin that adds the `--container-*` options to `srun` and `sbatch`. Running a container takes one extra flag on an otherwise normal job: ```bash srun --container-image=nvcr.io/nvidia/pytorch:25.09-py3 python train.py ``` Everything else stays normal Slurm: the same partitions, the same `--gres`, the same accounting, and the same **Jobs** tab. ## Before you begin - A cluster that is **Running**, and a shell on its login node. See [Connect to a cluster](/tir/tir-slurm-cluster/how-to/connect-to-a-cluster/). - A storage volume mounted on the cluster. These commands use `/shared` as the mount path; use the one you set. ## How a container job behaves Enroot and Pyxis come installed and ready to use, and Enroot takes the place of a Docker daemon. It converts the image into a flat root filesystem and runs your command in it using Linux user namespaces, so the job runs unprivileged inside the container. Inside the job: - `nvidia-smi` shows the GPUs Slurm allocated to the job. - Two jobs on the same cluster can use different images, so two teams can run different framework versions at the same time. - The container runs the same way on every node of a multi-node allocation. - Only the paths you pass with `--container-mounts` are visible inside the container. A container job is a step of a Slurm job rather than a separate server. The container lives as long as the job step and leaves out Docker's server features: `docker run`, `--privileged`, port publishing, and restart policies. The **Image** you chose when you created the cluster is a different thing. It selects the cluster's own Slurm, drivers, and operating system. Your job's container is chosen for each job. E2E manages the Pyxis plugin's configuration file, `plugstack.conf`, so it is ready to use. The next change to the cluster, such as a scale, an image update, a storage mount, or a Slurm configuration change, overwrites any hand edit of that file. ## Check that Pyxis is loaded Before you build a job script around the container flags, check that they are there. On the login node, run: ```bash srun --help | grep -i container enroot version ``` The output lists `--container-image`, `--container-mounts`, `--container-name`, and `--container-workdir`, and `enroot version` prints the Enroot version, which confirms the runtime is installed. If the flags are missing, Pyxis is missing on the cluster, and `srun` answers `srun: unrecognized option '--container-image'`. Check the image version on the **Details** tab and try **Update Image** to move to a current version. If the cluster still lacks the flags, contact support with the cluster name and image version. See [Update a cluster](/tir/tir-slurm-cluster/how-to/update-a-cluster/). ## Use the container flags Pass these flags to `srun`. In a batch script, put `srun --container-image=...` in the script body rather than in an `#SBATCH` line. It shows which step runs in the container, and it lets you run setup commands on the host first. | Flag | What it does | | ----------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--container-image=` | Names the image to run: a registry reference such as `nvcr.io/nvidia/pytorch:25.09-py3`, or the path to a squash file such as `/shared/images/pytorch.sqsh`. A path that starts with `/` is read as a file instead of pulled. | | `--container-name=` | Keeps the imported container on that node, so a later step or job on the same node starts at once instead of importing again. The name exists only on that node. | | `--container-mounts=:[,...]` | Bind-mounts host paths into the container. Separate several mounts with commas. | | `--container-workdir=` | Sets the working directory inside the container. | | `--container-writable`, `--container-readonly` | Allows or forbids writes to the container's root filesystem. | | `--container-remap-root`, `--no-container-remap-root` | Sets whether your user appears as `root` inside the container. | | `--container-save=` | Writes the container's filesystem back to a squash file after the step. | | `--container-env=` | Passes the named variables through, instead of the default environment handling. | | `--container-mount-home`, `--no-container-mount-home` | Mounts your home directory, or skips it. | | `--container-entrypoint`, `--no-container-entrypoint` | Runs the image's declared entrypoint, or skips it. | | `--mpi=pmix` | Uses the PMIx plugin to launch processes, which is what connects MPI ranks across nodes inside containers. Use it for MPI ranks, and it is recommended for any multi-node container job. | Which of the last six flags a cluster has depends on its Pyxis version. Before you use a flag in a production script, confirm that it exists on your own cluster with `srun --help | grep -i container`. ## Mount your data A container starts from the image's own filesystem. Your storage is on the node, and you pass in the paths the job needs: ```bash srun --container-image=nvcr.io/nvidia/pytorch:25.09-py3 \ --container-mounts=/shared:/shared \ --container-workdir=/shared/project \ python train.py ``` Forgetting `--container-mounts` is the most common mistake in a container job. The job starts, runs, and reports that your dataset does not exist: the mount is fine on the node, and the job lacks only the flag that passes it in. You can remap a path. `--container-mounts=/shared/datasets/imagenet:/data` makes the dataset appear at `/data` inside the container, which helps when your code expects a fixed path. You mount and unmount storage on the **Volumes** tab. See [Mount and unmount volumes](/tir/tir-slurm-cluster/how-to/mount-and-unmount-volumes/). This is what each kind of storage looks like on a node: | Storage | Writable | Shared across nodes | | ------------------------------------------------ | --------- | ----------------------------------------------------------------------------- | | Parallel File System or Shared File System mount | Yes | Yes. The same filesystem at the same path on the login node and every worker. | | Dataset mount | Read-only | Yes. | | `/dev/shm` | Yes | One node only. It is memory-backed and lost on restart. | | Node-local scratch such as `/tmp` | Yes | One node only. It is erased when the node is recreated. | Write checkpoints, logs, and model output to shared storage. The container's filesystem is discarded when the step ends, and a node-local path stays on one node and is erased on a restart. Writing to either of them is the commonest way to lose a training run. For a job that uses RDMA, which includes most multi-node NCCL training, mount the device nodes explicitly, because a container includes `/dev/infiniband` only when you mount it: ```bash --container-mounts=/shared:/shared,/dev/infiniband:/dev/infiniband ``` For the full multi-node picture, see [Run multi-node training](/tir/tir-slurm-cluster/how-to/run-multi-node-training/). ## Set environment variables Variables you export before `srun` are visible to the process inside the container. Slurm's own job variables, such as the job ID, node index, and task count, are available there too, so you can derive ranks and build per-job output paths from them. [Run multi-node training](/tir/tir-slurm-cluster/how-to/run-multi-node-training/) lists them: ```bash export NCCL_DEBUG=INFO srun --container-image=nvcr.io/nvidia/pytorch:25.09-py3 python train.py ``` NCCL and UCX tuning is left to you: the defaults come from the image, so a multi-node run over RDMA normally needs you to set several variables. Set environment for a job in the job script itself, or build it into the image. A variable set in a prolog does not reach the job. If your job needs a secret at runtime, read it from a file on mounted storage, and keep it out of job scripts and images. ## Use the GPUs inside the container Ask Slurm for GPUs the usual way, and the container inherits them: ```bash #SBATCH --gres=gpu:8 # 8 GPUs on the node #SBATCH --gpus-per-node=8 # the same request, written another way ``` `--gres` counts GPUs per node, so on a four-node job `--gres=gpu:8` means eight GPUs on each node. Inside the container, `nvidia-smi` shows only the GPUs the job was allocated: ```bash srun --gres=gpu:2 --container-image=nvcr.io/nvidia/pytorch:25.09-py3 nvidia-smi -L ``` The cluster advertises its GPUs to Slurm as a GRES with the card type, so you can see what is available before you submit. Use the GRES column of this command, or `scontrol show node`: ```bash sinfo -o "%20N %10c %10m %25G %10T" ``` ```bash scontrol show node slinky-0 ``` Choose an image whose CUDA runtime matches the GPU generation you are using. An image built for an older CUDA version may run poorly, or fail to start, on newer hardware. NGC tags state the CUDA version they carry. ## Try the worked examples Each example runs on the default partition. ### Train on one node with eight GPUs ```bash #!/bin/bash #SBATCH --job-name=train-8gpu #SBATCH --partition=all #SBATCH --nodes=1 #SBATCH --ntasks=1 #SBATCH --gpus-per-node=8 #SBATCH --cpus-per-task=64 #SBATCH --time=08:00:00 #SBATCH --output=/shared/logs/%x-%j.out #SBATCH --error=/shared/logs/%x-%j.err IMAGE=nvcr.io/nvidia/pytorch:25.09-py3 export NCCL_DEBUG=WARN srun --container-image="$IMAGE" \ --container-name=pytorch-train \ --container-mounts=/shared:/shared \ --container-workdir=/shared/project \ torchrun --standalone --nproc_per_node=8 train.py \ --data /shared/datasets/imagenet \ --checkpoint-dir /shared/checkpoints/${SLURM_JOB_ID} ``` ### Run a one-off command in a container ```bash srun --gres=gpu:1 --time=00:15:00 \ --container-image=nvcr.io/nvidia/pytorch:25.09-py3 \ --container-mounts=/shared:/shared \ python /shared/scripts/validate_dataset.py ``` ### Run a hyperparameter sweep as a job array ```bash #!/bin/bash #SBATCH --job-name=sweep #SBATCH --partition=all #SBATCH --array=0-15 #SBATCH --nodes=1 #SBATCH --gpus-per-node=1 #SBATCH --time=02:00:00 #SBATCH --output=/shared/logs/%x-%A_%a.out LRS=(0.1 0.03 0.01 0.003) BSS=(32 64 128 256) LR=${LRS[$((SLURM_ARRAY_TASK_ID / 4))]} BS=${BSS[$((SLURM_ARRAY_TASK_ID % 4))]} srun --container-image=/shared/images/pytorch-25.09.sqsh \ --container-mounts=/shared:/shared \ --container-workdir=/shared/project \ python train.py --lr "$LR" --batch-size "$BS" \ --out /shared/runs/lr${LR}_bs${BS} ``` Using a squash file instead of a registry reference means the 16 array tasks read the same file on shared storage instead of each pulling the image. See [Cache container images](/tir/tir-slurm-cluster/how-to/cache-container-images/). ### Run a CPU-only step ```bash srun --nodes=1 --ntasks=8 --time=01:00:00 \ --container-image=python:3.12-slim \ --container-mounts=/shared:/shared \ python /shared/scripts/shard_dataset.py --workers 8 ``` Minimal images such as `python:3.12-slim` can lack `bash`. Use `sh -c` instead of `bash -lc`, or choose a fuller base image. ## Start an interactive session To debug, allocate a node and get a shell inside the container: ```bash srun --partition=all --nodes=1 --gres=gpu:1 --time=01:00:00 --pty \ --container-image=nvcr.io/nvidia/pytorch:25.09-py3 \ --container-mounts=/shared:/shared \ --container-workdir=/shared \ bash ``` You are inside the container on a compute node with a GPU, and `nvidia-smi`, `python`, and your data are all there. Exit the shell to release the allocation. For a longer session, allocate first and attach steps to the allocation: ```bash salloc --partition=all --nodes=1 --gres=gpu:1 --time=04:00:00 ``` ```bash srun --container-image=nvcr.io/nvidia/pytorch:25.09-py3 --container-mounts=/shared:/shared --pty bash ``` An idle `salloc` keeps its GPUs out of the queue for other jobs. Always set `--time` and exit when you are done. `squeue -u $USER` shows what you are holding. ## Pull from a registry The node that runs the job pulls the image, so a registry pull is outbound traffic from that node, governed by the security group attached to the cluster. A group that restricts outbound traffic can break image pulls, and the failure looks like a plain network timeout. Either allow what the registry needs, or import the image once to a squash file that jobs read from shared storage. See [Cache container images](/tir/tir-slurm-cluster/how-to/cache-container-images/). To stop every job from pulling the same image, [cache container images](/tir/tir-slurm-cluster/how-to/cache-container-images/). To train across nodes, see [Run multi-node training](/tir/tir-slurm-cluster/how-to/run-multi-node-training/). For import failures, missing GPUs, or NCCL errors, see [Troubleshooting containers on TIR Slurm Cluster](/tir/tir-slurm-cluster/troubleshooting/containers/).