--- title: Container Job Failures sidebar_label: Containers --- import { Box, HardDrive, Cpu, Share2, Lock } from 'react-feather'; # Container Job Failures Problems specific to running container images inside Slurm jobs with Enroot and Pyxis. }, { href: '#the-image-will-not-import', label: 'Image will not import', icon: }, { href: '#the-container-cannot-see-my-files', label: 'Cannot see my files', icon: }, { href: '#no-gpus-inside-the-container', label: 'No GPUs', icon: }, { href: '#a-multi-node-job-hangs', label: 'Multi-node hangs', icon: }, { href: '#permission-problems', label: 'Permission problems', icon: }, ]} /> --- ## `--container-image` Is Not Recognised ``` srun: unrecognized option '--container-image' ``` The Pyxis plugin is not loaded on this cluster, so the container flags do not exist. 1. Confirm: ```bash srun --help | grep -i container ``` Nothing back means the plugin is absent. 2. Check the **Image Version** on the [Details tab](/docs/tir/SlurmCluster/manage/). Container support depends on the cluster image being a Pyxis-enabled build. 3. Try [**Actions → Update Image**](/docs/tir/SlurmCluster/manage/actions#update-image) to move to a current version. This is a rolling restart, so drain your queue first. 4. If a current image still does not offer the flags, contact support with the cluster name and the image version. :::danger Do not try to install Pyxis or edit `plugstack.conf` yourself `plugstack.conf` is platform-managed and regenerated whenever the cluster changes — a scale, an image update, a volume mount. Any local edit is silently reverted, and you will spend an afternoon concluding the flags "randomly stopped working". ::: --- ## The Image Will Not Import ### It hangs or times out ```bash enroot import -o /pfs/images/test.sqsh docker://nvcr.io/nvidia/pytorch:25.09-py3 ``` | Cause | Fix | |-------|-----| | Large image, slow first pull | Expect several minutes and several GB. Import from the login node, not inside a job with a short `--time` | | No outbound network from a worker | Import on the login node and pass the resulting `.sqsh` file to jobs | | Registry rate limit | You have pulled too often. This is exactly what caching solves | | Outbound traffic blocked by the security group | See the note below | :::danger A restrictive security group breaks image pulls Pulling `docker://…` happens **from the node running the job**, and that outbound traffic is governed by the [security group attached to the cluster](/docs/tir/SlurmCluster/manage/network-security#security-groups). The default SSH security group permits outbound traffic, so pulls work — but a group written to restrict egress blocks them, and the failure looks like a plain network timeout with no mention of the security group. Two ways out: - Allow the outbound traffic the registry needs in the attached group, or - **Import the image once on the login node** to a squash file on shared storage, so jobs never pull at all. That is the better answer either way. ::: :::tip Import once, on the login node, to shared storage ```bash mkdir -p /pfs/images enroot import -o /pfs/images/pytorch-25.09.sqsh docker://nvcr.io/nvidia/pytorch:25.09-py3 ``` Then `--container-image=/pfs/images/pytorch-25.09.sqsh`. Every node reads the same file — no per-node pulls, no rate limits, and predictable start-up. See [Cache and manage images](/docs/tir/SlurmCluster/containers/image-cache). ::: ### Authentication failures A private registry needs credentials in Enroot's per-user credentials file: ```bash cat /etc/enroot/enroot.conf 2>/dev/null | grep -i config ``` ```bash mkdir -p ~/.config/enroot cat > ~/.config/enroot/.credentials <<'EOF' machine nvcr.io login $oauthtoken password machine registry.example.com login password EOF chmod 600 ~/.config/enroot/.credentials ``` :::tip Import once with credentials, then jobs need none Do the authenticated `enroot import` on the login node and have jobs reference the squash file. No job script ever needs the token, which is both safer and simpler. ::: :::danger Never put registry credentials in a job script or a prolog Both are stored with the cluster configuration and carried into clones. Use a scoped read-only registry token, keep the credentials file at mode `600`, and rotate it on your normal schedule. ::: ### It runs out of space ```bash df -h /pfs du -sh /pfs/images/* ``` Squash files are commonly 5–20 GB each. Prune old ones. Also check where Enroot's own cache is pointing, since it defaults under `$HOME`: ```bash echo "HOME=$HOME"; cat /etc/enroot/enroot.conf 2>/dev/null ``` --- ## The Container Cannot See My Files Almost always a missing `--container-mounts`. A container starts from the image's filesystem. Your PFS and SFS mounts exist on the node but are **not** inside the container unless you pass them: ```bash srun --container-image=/pfs/images/pytorch.sqsh \ --container-mounts=/pfs:/pfs \ ls -la /pfs ``` | Symptom | Fix | |---------|-----| | `No such file or directory` for a path you know exists | Add it to `--container-mounts` | | The dataset is there but the code cannot find it | The path inside the container differs. Remap: `--container-mounts=/pfs/datasets/imagenet:/data` | | Checkpoints write "successfully" but are gone afterwards | They went into the container's own filesystem, which is discarded. Point the output at a mounted path | | `Permission denied` writing | It is a **dataset** mount, which is read-only. Use PFS or SFS | :::danger Output written inside the container is discarded when the step ends The container's filesystem is not persistent. A training run that appears to complete and saves nothing has almost always written to a path that was never a real mount. Verify with a one-liner before the real run: ```bash srun --container-image=... --container-mounts=/pfs:/pfs \ bash -c 'touch /pfs/mount-check && echo OK && rm /pfs/mount-check' ``` ::: ### Confirm the mount is shared across nodes ```bash echo "written from $(hostname)" > /pfs/shared-check.txt srun --nodes=4 --ntasks-per-node=1 bash -lc 'echo "$(hostname): $(cat /pfs/shared-check.txt)"' ``` Every node must print the same line. If one cannot read it, the volume is not mounted there — check the [Volumes tab](/docs/tir/SlurmCluster/manage/storage). --- ## No GPUs Inside the Container ```bash srun --gres=gpu:1 --container-image=... nvidia-smi ``` | Symptom | Cause | Fix | |---------|-------|-----| | `No devices were found` | The job asked for no GPUs | Add `--gres=gpu:N` or `--gpus-per-node=N` | | `nvidia-smi: command not found` | The image has no CUDA stack | Use a CUDA-based image such as an NGC one | | Fewer GPUs than expected | Slurm allocated fewer | Check `--gres`, and `scontrol show job ` | | `torch.cuda.is_available()` is `False` while `nvidia-smi` works | The image's CUDA build does not match the driver | Use an image built for your GPU generation | | CUDA errors only on some nodes | Hardware | Check the [Nodes tab](/docs/tir/SlurmCluster/manage/nodes#xid-errors) for XID errors | :::warning `--gres` is per node `--gres=gpu:8` on a 4-node job means 8 GPUs **on each** node, not 8 in total. Use `--gpus-per-node` to be explicit about intent. ::: :::tip Match the image to the hardware A container built for an older CUDA release may run poorly or not at all on newer GPUs such as H200 or B200. NGC tags state their CUDA version — check the [Nodes tab](/docs/tir/SlurmCluster/manage/nodes) for your GPU model and pick accordingly. ::: --- ## A Multi-Node Job Hangs Typically at NCCL initialisation, before any training output appears. ### 1. Diagnose the transport ```bash export NCCL_DEBUG=INFO ``` Resubmit and read the log: | What you see | Meaning | |--------------|---------| | `NET/IB` | Using InfiniBand. Correct | | `NET/Socket` | Fell back to TCP. Throughput will be a fraction of the hardware's | | It stops before naming a transport | The ranks never connected | ### 2. The three usual causes | Cause | Fix | |-------|-----| | RDMA devices not in the container | Add `--container-mounts=...,/dev/infiniband:/dev/infiniband` | | Wrong `NCCL_IB_HCA` names | List them per node with `ls /sys/class/infiniband` and confirm they are identical everywhere | | Rendezvous host unreachable | Derive it, do not hard-code it: `MASTER_ADDR=$(scontrol show hostnames "$SLURM_JOB_NODELIST" \| head -n1)` | ### 3. Other checks ```bash # Do all nodes agree on device names? srun --nodes=4 --ntasks-per-node=1 bash -lc 'echo "$(hostname): $(ls /sys/class/infiniband | tr "\n" " ")"' ``` | Check | Why | |-------|-----| | Is the image a squash file on shared storage? | A `--container-name` created on one node does not exist on another | | Did every node get the container? | A node that failed to import stalls the whole collective | | Is `--mpi=pmix` set? | Needed for MPI ranks, and recommended for any multi-node container job | | Are all nodes healthy? | One `UNKNOWN` or drained node silently shrinks the allocation | Full guidance and a working example: [Multi-node container training](/docs/tir/SlurmCluster/containers/multi-node-training). :::danger Node-local caches break multi-node jobs `/tmp`, `/dev/shm` and anything under `$HOME` on a non-persistent login are private to one node. A container name, a squash file or an `XDG_DATA_HOME` pointed at `/tmp` cannot be shared — and the symptom is a hang, not an error. ::: ### Benchmark before you blame your code Run a NCCL all-reduce at the scale you intend to use: [Validate the fabric first](/docs/tir/SlurmCluster/containers/multi-node-training#validate-the-fabric-first). If the benchmark is healthy, the problem is in your job; if it is not, it is in the fabric configuration or the hardware. --- ## Permission Problems Enroot runs containers unprivileged, so some things a `docker run` would allow do not apply. | Symptom | Cause | Fix | |---------|-------|-----| | `Permission denied` writing inside the container | The container root filesystem is not writable | Write to a mounted volume instead — which you should be doing anyway | | `apt install` fails | You are not root inside the container, and the filesystem is not writable | Build the dependency into the image | | A script expects to be root | Enroot's user mapping | Restructure the script, or investigate `--container-remap-root` if your Pyxis version offers it (`srun --help \| grep container`) | | A file created by a job is owned by an unexpected user | Your host UID maps through | Expected. On shared directories, use group-writable permissions | :::tip Bake dependencies into the image, do not install them at job time `pip install` at the start of every job is slow, fragile, and different on every node when a package version moves. Build an image, [cache it as a squash file](/docs/tir/SlurmCluster/containers/image-cache), and every rank on every node gets a byte-identical environment. ::: --- ## Checklist for a New Container Workload 1. `srun --help | grep -i container` — the flags exist. 2. `enroot import` the image once to shared storage. 3. Run a one-node smoke test: `nvidia-smi` plus your framework's version and device count. 4. Verify the mounts with a `touch` inside the container. 5. For multi-node, run the NCCL benchmark at the target scale. 6. Confirm GPU utilisation is where you expect on the [Monitoring tab](/docs/tir/SlurmCluster/manage/monitoring). 7. Then start the real run — with checkpointing to shared storage. --- ## Related Resources - [Containers with Enroot and Pyxis](/docs/tir/SlurmCluster/containers/) - [Run containers in jobs](/docs/tir/SlurmCluster/containers/run-containers) - [Cache and manage images](/docs/tir/SlurmCluster/containers/image-cache) - [Multi-node container training](/docs/tir/SlurmCluster/containers/multi-node-training) - [Jobs will not run](/docs/tir/SlurmCluster/troubleshoot/jobs)