Container Job Failures

Problems specific to running container images inside Slurm jobs with Enroot and Pyxis.

Flag not recognisedImage will not importCannot see my filesNo GPUsMulti-node hangsPermission problems

---

--container-image Is Not Recognised

srun: unrecognized option '--container-image'

The Pyxis plugin is not loaded on this cluster, so the container flags do not exist.

1. Confirm:

   srun --help | grep -i container

   Nothing back means the plugin is absent.

2. Check the Image Version on the Details tab. Container support depends on the cluster image being a Pyxis-enabled build.

3. Try Actions → Update Image to move to a current version. This is a rolling restart, so drain your queue first.

4. If a current image still does not offer the flags, contact support with the cluster name and the image version.

Do not try to install Pyxis or edit plugstack.conf yourself

plugstack.conf is platform-managed and regenerated whenever the cluster changes — a scale, an image update, a volume mount. Any local edit is silently reverted, and you will spend an afternoon concluding the flags "randomly stopped working".

---

The Image Will Not Import

It hangs or times out

enroot import -o /pfs/images/test.sqsh docker://nvcr.io/nvidia/pytorch:25.09-py3

Cause	Fix
Large image, slow first pull	Expect several minutes and several GB. Import from the login node, not inside a job with a short --time
No outbound network from a worker	Import on the login node and pass the resulting .sqsh file to jobs
Registry rate limit	You have pulled too often. This is exactly what caching solves
Outbound traffic blocked by the security group	See the note below

A restrictive security group breaks image pulls

Pulling docker://… happens from the node running the job, and that outbound traffic is governed by the security group attached to the cluster. The default SSH security group permits outbound traffic, so pulls work — but a group written to restrict egress blocks them, and the failure looks like a plain network timeout with no mention of the security group.

Two ways out:

• Allow the outbound traffic the registry needs in the attached group, or
• Import the image once on the login node to a squash file on shared storage, so jobs never pull at all. That is the better answer either way.

Import once, on the login node, to shared storage

mkdir -p /pfs/images
enroot import -o /pfs/images/pytorch-25.09.sqsh docker://nvcr.io/nvidia/pytorch:25.09-py3

Then --container-image=/pfs/images/pytorch-25.09.sqsh. Every node reads the same file — no per-node pulls, no rate limits, and predictable start-up. See Cache and manage images.

Authentication failures

A private registry needs credentials in Enroot's per-user credentials file:

cat /etc/enroot/enroot.conf 2>/dev/null | grep -i config

mkdir -p ~/.config/enroot
cat > ~/.config/enroot/.credentials <<'EOF'
machine nvcr.io login $oauthtoken password <NGC_API_KEY>
machine registry.example.com login <username> password <token>
EOF
chmod 600 ~/.config/enroot/.credentials

Import once with credentials, then jobs need none

Do the authenticated enroot import on the login node and have jobs reference the squash file. No job script ever needs the token, which is both safer and simpler.

Never put registry credentials in a job script or a prolog

Both are stored with the cluster configuration and carried into clones. Use a scoped read-only registry token, keep the credentials file at mode 600, and rotate it on your normal schedule.

It runs out of space

df -h /pfs
du -sh /pfs/images/*

Squash files are commonly 5–20 GB each. Prune old ones. Also check where Enroot's own cache is pointing, since it defaults under $HOME:

echo "HOME=$HOME"; cat /etc/enroot/enroot.conf 2>/dev/null

---

The Container Cannot See My Files

Almost always a missing --container-mounts.

A container starts from the image's filesystem. Your PFS and SFS mounts exist on the node but are not inside the container unless you pass them:

srun --container-image=/pfs/images/pytorch.sqsh \
     --container-mounts=/pfs:/pfs \
     ls -la /pfs

Symptom	Fix
No such file or directory for a path you know exists	Add it to --container-mounts
The dataset is there but the code cannot find it	The path inside the container differs. Remap: --container-mounts=/pfs/datasets/imagenet:/data
Checkpoints write "successfully" but are gone afterwards	They went into the container's own filesystem, which is discarded. Point the output at a mounted path
Permission denied writing	It is a dataset mount, which is read-only. Use PFS or SFS

Output written inside the container is discarded when the step ends

The container's filesystem is not persistent. A training run that appears to complete and saves nothing has almost always written to a path that was never a real mount. Verify with a one-liner before the real run:

srun --container-image=... --container-mounts=/pfs:/pfs \
     bash -c 'touch /pfs/mount-check && echo OK && rm /pfs/mount-check'

Confirm the mount is shared across nodes

echo "written from $(hostname)" > /pfs/shared-check.txt
srun --nodes=4 --ntasks-per-node=1 bash -lc 'echo "$(hostname): $(cat /pfs/shared-check.txt)"'

Every node must print the same line. If one cannot read it, the volume is not mounted there — check the Volumes tab.

---

No GPUs Inside the Container

srun --gres=gpu:1 --container-image=... nvidia-smi

Symptom	Cause	Fix
No devices were found	The job asked for no GPUs	Add --gres=gpu:N or --gpus-per-node=N
nvidia-smi: command not found	The image has no CUDA stack	Use a CUDA-based image such as an NGC one
Fewer GPUs than expected	Slurm allocated fewer	Check --gres, and scontrol show job <jobid>
torch.cuda.is_available() is False while nvidia-smi works	The image's CUDA build does not match the driver	Use an image built for your GPU generation
CUDA errors only on some nodes	Hardware	Check the Nodes tab for XID errors

--gres is per node

--gres=gpu:8 on a 4-node job means 8 GPUs on each node, not 8 in total. Use --gpus-per-node to be explicit about intent.

Match the image to the hardware

A container built for an older CUDA release may run poorly or not at all on newer GPUs such as H200 or B200. NGC tags state their CUDA version — check the Nodes tab for your GPU model and pick accordingly.

---

A Multi-Node Job Hangs

Typically at NCCL initialisation, before any training output appears.

1. Diagnose the transport

export NCCL_DEBUG=INFO

Resubmit and read the log:

What you see	Meaning
NET/IB	Using InfiniBand. Correct
NET/Socket	Fell back to TCP. Throughput will be a fraction of the hardware's
It stops before naming a transport	The ranks never connected

2. The three usual causes

Cause	Fix
RDMA devices not in the container	Add --container-mounts=...,/dev/infiniband:/dev/infiniband
Wrong NCCL_IB_HCA names	List them per node with ls /sys/class/infiniband and confirm they are identical everywhere
Rendezvous host unreachable	Derive it, do not hard-code it: MASTER_ADDR=$(scontrol show hostnames "$SLURM_JOB_NODELIST" | head -n1)

3. Other checks

# Do all nodes agree on device names?
srun --nodes=4 --ntasks-per-node=1 bash -lc 'echo "$(hostname): $(ls /sys/class/infiniband | tr "\n" " ")"'

Check	Why
Is the image a squash file on shared storage?	A --container-name created on one node does not exist on another
Did every node get the container?	A node that failed to import stalls the whole collective
Is --mpi=pmix set?	Needed for MPI ranks, and recommended for any multi-node container job
Are all nodes healthy?	One UNKNOWN or drained node silently shrinks the allocation

Full guidance and a working example: Multi-node container training.

Node-local caches break multi-node jobs

/tmp, /dev/shm and anything under $HOME on a non-persistent login are private to one node. A container name, a squash file or an XDG_DATA_HOME pointed at /tmp cannot be shared — and the symptom is a hang, not an error.

Benchmark before you blame your code

Run a NCCL all-reduce at the scale you intend to use: Validate the fabric first. If the benchmark is healthy, the problem is in your job; if it is not, it is in the fabric configuration or the hardware.

---

Permission Problems

Enroot runs containers unprivileged, so some things a docker run would allow do not apply.

Symptom	Cause	Fix
Permission denied writing inside the container	The container root filesystem is not writable	Write to a mounted volume instead — which you should be doing anyway
apt install fails	You are not root inside the container, and the filesystem is not writable	Build the dependency into the image
A script expects to be root	Enroot's user mapping	Restructure the script, or investigate --container-remap-root if your Pyxis version offers it (srun --help | grep container)
A file created by a job is owned by an unexpected user	Your host UID maps through	Expected. On shared directories, use group-writable permissions

Bake dependencies into the image, do not install them at job time

pip install at the start of every job is slow, fragile, and different on every node when a package version moves. Build an image, cache it as a squash file, and every rank on every node gets a byte-identical environment.

---

Checklist for a New Container Workload

1. srun --help | grep -i container — the flags exist.
2. enroot import the image once to shared storage.
3. Run a one-node smoke test: nvidia-smi plus your framework's version and device count.
4. Verify the mounts with a touch inside the container.
5. For multi-node, run the NCCL benchmark at the target scale.
6. Confirm GPU utilisation is where you expect on the Monitoring tab.
7. Then start the real run — with checkpointing to shared storage.

---

Related Resources

• Containers with Enroot and Pyxis
• Run containers in jobs
• Cache and manage images
• Multi-node container training
• Jobs will not run