--- title: Containers with Enroot and Pyxis sidebar_label: Overview --- import { Box, Package, Layers, Terminal, HelpCircle, Shield } from 'react-feather'; # Containers with Enroot and Pyxis A TIR Slurm Cluster can run **any Docker or OCI container image inside a Slurm job**. You do not build a custom cluster image, and you do not install anything: the cluster ships with [**Enroot**](https://github.com/NVIDIA/enroot) as an unprivileged container runtime and [**Pyxis**](https://github.com/NVIDIA/pyxis) as the Slurm SPANK plugin that wires Enroot into `srun` and `sbatch`. The result is one extra flag on your job: ```bash srun --container-image=nvcr.io/nvidia/pytorch:25.09-py3 python train.py ``` Everything else stays normal Slurm — same partitions, same `--gres`, same accounting, same [Jobs tab](/docs/tir/SlurmCluster/manage/jobs). }, { href: '#how-it-fits-together', label: 'How it fits together', icon: }, { href: '#confirm-pyxis-is-available', label: 'Confirm availability', icon: }, { href: '#your-first-container-job', label: 'First container job', icon: }, { href: '#where-to-go-next', label: 'Where to go next', icon: }, ]} /> --- ## Why Enroot and Pyxis Distributed training environments are hard to reproduce. Enroot and Pyxis let you take the exact image your team already builds — the one from NGC, from your CI, from Docker Hub — and run it as the job environment, without giving up Slurm scheduling. | Property | What it means for you | |----------|-----------------------| | **Unprivileged** | Enroot runs containers as your own user. There is no Docker daemon on the nodes and you do not need root inside the job | | **No cluster rebuild** | The image is chosen per job, not per cluster. Two jobs on the same cluster can use completely different CUDA and framework versions | | **Native GPU access** | The container sees the node's GPUs directly; `nvidia-smi` and CUDA work with no device plumbing from you | | **Slurm-native** | `--container-image` is just another `srun`/`sbatch` flag. `sbatch`, job arrays, dependencies, `--gres`, accounting and preemption all behave normally | | **Multi-node in one `srun`** | A single `srun` launches the same container on every node of the allocation, with PMIx available for MPI and NCCL ranks | | **Reusable** | `--container-name` keeps an imported container around so later steps and jobs start instantly instead of re-importing | :::info This is not Docker, and not Kubernetes Your job is not a Kubernetes pod and there is no `docker run`. Enroot converts the image into a flat root filesystem and runs your command in it with user namespaces. Practical differences: no `--privileged`, no port publishing, no container restart policy, and the container's lifetime is the job step's lifetime. ::: --- ## How It Fits Together ``` you login node worker nodes (slinky-0 … slinky-N) ─── ────────── ───────────────────────────────── sbatch job.sh ────────► slurmctld schedules ──────► slurmd starts the job step │ ├─ Pyxis reads --container-* flags ├─ Enroot imports / reuses the image └─ your command runs inside the container with the node's GPUs and your PFS / SFS mounts visible ``` Three pieces make it work, and all three are managed for you: | Piece | What it is | Who owns it | |-------|------------|-------------| | **Pyxis SPANK plugin** | Registers `--container-image`, `--container-mounts`, `--container-name` and friends on `srun`/`sbatch`. Enabled by `plugstack.conf` containing `include /usr/share/pyxis/*` | Platform-managed. There is no console setting, and `plugstack.conf` is not editable | | **Enroot runtime** | Imports images and starts containers, unprivileged | Baked into the cluster's login and compute images | | **Slurm prolog** | The platform runs a prolog on every node that writes an Enroot mount rule so Slurm's own config cache is readable inside containers | Platform-managed; it lands in its own file and never interferes with [your prolog scripts](/docs/tir/SlurmCluster/slurm-configuration/prolog-epilog) | :::note One visible side effect Because of that platform mount rule, every container on the cluster has `/var/spool/slurmd/conf-cache` bind-mounted read-only. It is harmless — you will only notice it if you list `/var/spool/slurmd` inside a job. ::: --- ## Confirm Pyxis Is Available Pyxis needs both the plugin configuration **and** a Pyxis-enabled cluster image. Both are standard on GPU Slurm Clusters, but it costs nothing to check before you build a job script around it. ```bash ssh root@ ``` ```bash # If Pyxis is loaded, srun advertises the container options srun --help | grep -i container ``` You should see `--container-image`, `--container-mounts`, `--container-name` and `--container-workdir` among the options. ```bash # Enroot itself enroot version ``` :::warning If `--container-image` is not recognised `srun: unrecognized option '--container-image'` means the plugin is not loaded on that cluster — most often because the selected cluster image version is not a Pyxis build. Check the **Image Version** on the [Details tab](/docs/tir/SlurmCluster/manage/), try [**Update Image**](/docs/tir/SlurmCluster/manage/actions#update-image) to move to a current version, and contact support if it persists. Do not try to install Pyxis yourself — `plugstack.conf` is platform-managed and any local edit is overwritten the next time the cluster is re-rendered. ::: ### Check the exact flag set your image supports Enroot and Pyxis versions differ between cluster image versions, and the flags below marked *standard Pyxis* may or may not be present on yours. The authoritative answer is always on your own cluster: ```bash srun --help | grep -i container ``` ```bash cat /etc/enroot/enroot.conf 2>/dev/null; enroot start --help | head -40 ``` That last command is also how you find out which paths Enroot uses for its cache and data on your cluster — which matters for [caching images](/docs/tir/SlurmCluster/containers/image-cache). --- ## Your First Container Job This runs `nvidia-smi` inside an NGC PyTorch container on one GPU. It is the fastest way to prove the whole path works. ```bash ssh root@ ``` Create `container-hello.sh` on a shared filesystem — for example your PFS mount at `/pfs`: ```bash #!/bin/bash #SBATCH --job-name=container-hello #SBATCH --partition=all #SBATCH --nodes=1 #SBATCH --ntasks=1 #SBATCH --gres=gpu:1 #SBATCH --time=00:10:00 #SBATCH --output=/pfs/logs/%x-%j.out #SBATCH --error=/pfs/logs/%x-%j.err srun --container-image=nvcr.io/nvidia/pytorch:25.09-py3 \ --container-mounts=/pfs:/pfs \ --container-workdir=/pfs \ bash -lc 'nvidia-smi && python -c "import torch; print(torch.__version__, torch.cuda.device_count())"' ``` Submit it and watch it run: ```bash mkdir -p /pfs/logs sbatch container-hello.sh squeue ``` The first run spends a minute or two importing the image. Later runs against the same image are much faster — and faster still if you [cache the image as a squash file](/docs/tir/SlurmCluster/containers/image-cache). :::note Registry pulls go out through the cluster's security group The image is fetched **from the node running the job**, so a registry pull is outbound traffic subject to the [attached security group](/docs/tir/SlurmCluster/manage/network-security#security-groups). The default SSH group allows it. With a group that restricts egress, either allow what the registry needs or [cache the image as a squash file](/docs/tir/SlurmCluster/containers/image-cache) so jobs never pull. ::: :::tip Mount your storage explicitly `--container-mounts=/pfs:/pfs` is doing real work here. Your PFS, SFS and dataset mounts exist on the node, but a container starts from the image's own filesystem — pass the paths you need. See [Run containers in jobs](/docs/tir/SlurmCluster/containers/run-containers#mount-your-data). ::: --- ## Where to Go Next | Page | What it covers | |------|----------------| | [Run containers in jobs](/docs/tir/SlurmCluster/containers/run-containers) | Every `--container-*` flag, mounting storage, environment variables, interactive sessions, worked `sbatch` examples | | [Cache and manage images](/docs/tir/SlurmCluster/containers/image-cache) | `enroot import`, squash files on shared storage, named containers, private registries, avoiding repeated pulls | | [Multi-node container training](/docs/tir/SlurmCluster/containers/multi-node-training) | PMIx, NCCL, InfiniBand, `torchrun` across nodes, a full 8-GPU-per-node example | | [Troubleshoot container jobs](/docs/tir/SlurmCluster/troubleshoot/containers) | Import failures, missing GPUs, NCCL errors, permission problems | --- ## Related Resources - [Submit your first job](/docs/tir/SlurmCluster/getting-started/first-job) - [Prolog and epilog scripts](/docs/tir/SlurmCluster/slurm-configuration/prolog-epilog) - [Storage and volumes](/docs/tir/SlurmCluster/manage/storage) - [Pyxis on GitHub](https://github.com/NVIDIA/pyxis) · [Enroot on GitHub](https://github.com/NVIDIA/enroot)