Troubleshooting containers on TIR Slurm Cluster

Fix problems with running container images inside Slurm jobs, with Enroot and Pyxis. For the steps to run containers, see Run containers in jobs. For other problems, see Troubleshooting TIR Slurm Cluster, Troubleshooting connecting to TIR Slurm Cluster, and Troubleshooting jobs on TIR Slurm Cluster.

My results are missing after a container job ends

Output written only inside the container is discarded when the step ends. A run that appears to finish and saves nothing has almost always written to a path inside the container instead of a real mount. Write checkpoints, logs, and model output to a mounted PFS or SFS path, and check the mount with a small test before the real run:

srun --container-image=<image> --container-mounts=/pfs:/pfs \
     bash -c 'touch /pfs/mount-check && echo OK && rm /pfs/mount-check'

For the cause, see The container cannot see my files.

The container cannot see my files

This is almost always a missing --container-mounts. The container starts from the image’s own filesystem, so the PFS and SFS mounts on the node appear inside it only when you pass them in:

srun --container-image=/pfs/images/pytorch.sqsh \
     --container-mounts=/pfs:/pfs \
     ls -la /pfs

What you see	What to do
No such file or directory for a path you know exists	Add the path to --container-mounts.
The data is there but the code cannot find it	The path inside the container differs. Remap it: --container-mounts=/pfs/datasets/imagenet:/data.
Checkpoints save “successfully” but are gone afterwards	They went into the container’s own filesystem. Point the output at a mounted path.
Permission denied when writing	The path is a dataset mount, which is read-only. Use a PFS or SFS mount.

To confirm a mount is shared across nodes, write a file on the login node and read it back from every node:

echo "written from $(hostname)" > /pfs/shared-check.txt
srun --nodes=4 --ntasks-per-node=1 bash -lc 'echo "$(hostname): $(cat /pfs/shared-check.txt)"'

Every node must print the same line. If one node fails to read it, that node is missing the volume. Check the Volumes tab. See Mount and unmount volumes.

srun: unrecognized option ‘–container-image’

This error means the Pyxis plugin, which adds the container flags, is missing on this cluster.

1. Confirm it. An empty result means the plugin is absent:

   srun --help | grep -i container

2. Check the image version on the Details tab. Container support depends on the cluster image.

3. Select Actions, then Update Image to move to a current version. It is a rolling restart, so drain your queue first. See Update a cluster.

4. If a current image still lacks the flags, contact support with the cluster name and the image version.

Leave Pyxis and plugstack.conf to the platform. It manages plugstack.conf and rebuilds it on every cluster change, such as a scale, an image update, or a volume mount, so a hand edit is silently reverted.

An image will not import

The node running the job pulls the image, and the traffic goes out through the security group attached to the cluster. Import an image once on the login node into a squash file on shared storage, so that jobs read it from there. This avoids a slow first pull on every node, registry rate limits, and a security group that blocks the pull:

mkdir -p /pfs/images
enroot import -o /pfs/images/pytorch-25.09.sqsh docker://nvcr.io/nvidia/pytorch:25.09-py3

Then use --container-image=/pfs/images/pytorch-25.09.sqsh. Every node reads the same file. See Cache container images.

Cause	What to do
A large image and a slow first pull	Expect several minutes and several GB. Import from the login node rather than in a job with a short --time.
A worker that lacks outbound network access	Import on the login node and pass the resulting .sqsh file to jobs.
A registry rate limit	You have pulled too often. Caching the image solves this.
A security group that blocks outbound traffic	See the next paragraph.

A restrictive security group can break docker:// pulls, and the failure looks like a plain network timeout. Either allow the outbound traffic the registry needs in the attached group, or import the image once to a squash file. See Manage network and security.

The import fails to authenticate

A private registry needs credentials in Enroot’s per-user credentials file. The location is set by /etc/enroot/enroot.conf on the cluster image and can differ between image versions, so check that file instead of assuming a path:

cat /etc/enroot/enroot.conf 2>/dev/null | grep -i config

mkdir -p ~/.config/enroot
cat > ~/.config/enroot/.credentials <<'EOF'
machine nvcr.io login $oauthtoken password <NGC_API_KEY>
machine registry.example.com login <username> password <token>
EOF
chmod 600 ~/.config/enroot/.credentials

In the file, replace each value in angle brackets with your own: the registry key, and the login and token for registry.example.com. Do the authenticated import once on the login node and have jobs use the squash file, so that the token stays out of job scripts. Keep registry credentials out of job scripts and prologs, because both are stored with the cluster configuration and carried into clones. Use a read-only registry token with the narrowest scope.

The import runs out of space

df -h /pfs
du -sh /pfs/images/*

Squash files are commonly 5 to 20 GB each, so prune old ones. Also check where Enroot’s own cache points, since it defaults under $HOME:

echo "HOME=$HOME"; cat /etc/enroot/enroot.conf 2>/dev/null

There are no GPUs inside the container

srun --gres=gpu:1 --container-image=<image> nvidia-smi

What you see	Cause	What to do
No devices were found	The job left out a GPU request	Add --gres=gpu:N or --gpus-per-node=N.
nvidia-smi: command not found	The image lacks a CUDA stack	Use a CUDA-based image, such as an NGC image.
Fewer GPUs than expected	Slurm allocated fewer	Check --gres and scontrol show job <jobid>.
torch.cuda.is_available() is False while nvidia-smi works	The image’s CUDA build is incompatible with the driver	Use an image built for your GPU generation.
CUDA errors only on some nodes	Hardware	Check the Nodes tab for XID errors.

--gres counts GPUs per node. --gres=gpu:8 on a four-node job means 8 GPUs on each node, 32 in total. Use --gpus-per-node to say what you mean. Match the image to the hardware: a container built for an older CUDA release may run poorly or fail to start on newer GPUs such as H200 or B200, and NGC tags state their CUDA version. The Nodes tab shows the GPU model your cluster has.

A multi-node job hangs

A multi-node job that hangs at NCCL initialisation, before any training output, usually traces to three causes. Set NCCL_DEBUG=INFO, resubmit, and read the log:

export NCCL_DEBUG=INFO

What you see	What it means
NET/IB	InfiniBand is in use, which is correct.
NET/Socket	NCCL fell back to TCP, and throughput will be far lower.
No transport named	The ranks failed to connect.

Check the three usual causes:

Cause	What to do
RDMA devices are missing from the container	Add --container-mounts=...,/dev/infiniband:/dev/infiniband.
NCCL_IB_HCA names differ between nodes	List them per node with ls /sys/class/infiniband and confirm they are identical everywhere.
The rendezvous host is hard-coded	Derive it: MASTER_ADDR=$(scontrol show hostnames “$SLURM_JOB_NODELIST” | head -n1).

Then check that all nodes agree on device names:

srun --nodes=4 --ntasks-per-node=1 bash -lc 'echo "$(hostname): $(ls /sys/class/infiniband | tr "\n" " ")"'

Also check these:

• Is the image a squash file on shared storage? A --container-name is local to the node that created it.
• Did every node get the container? A node that failed to import stalls the whole collective.
• Is --mpi=pmix set? It is needed for MPI ranks, and recommended for any multi-node container job.
• Are all nodes healthy? One Unknown or drained node quietly shrinks the allocation.

/tmp, /dev/shm, and anything under $HOME while Login User Management is off are private to one node. A container name, a squash file, or an XDG_DATA_HOME pointed at /tmp stays on one node, and the symptom is a hang rather than an error.

Run an NCCL all-reduce benchmark at the scale you intend to use before you blame your code. If it is healthy, the problem is in your job. If it is slow or fails, the problem is in the fabric configuration or the hardware. See Run multi-node training.

I get permission errors inside the container

Enroot runs containers unprivileged, so a few things a Docker run allows work differently.

What you see	Cause	What to do
Permission denied writing inside the container	The container’s root filesystem is read-only	Write to a mounted volume.
apt install fails	You run as your own user inside the container	Build the dependency into the image.
A script expects to be root	Enroot’s user mapping	Restructure the script, or look at --container-remap-root if your Pyxis version offers it (srun --help | grep container).
A file a job created has an unexpected owner	Your host user ID carries through	This is expected. On shared directories, use group-writable permissions.

Bake dependencies into the image instead of installing them at job time. A pip install at the start of every job is slow, fragile, and different on every node when a package version moves. Build an image and cache it as a squash file, and every rank on every node gets an identical environment.

Checklist for a new container job

1. srun --help | grep -i container shows the flags.
2. enroot import the image once to shared storage.
3. Run a one-node smoke test: nvidia-smi plus your framework’s version and device count.
4. Verify the mounts with a touch inside the container.
5. For multi-node, run the NCCL benchmark at the target scale.
6. Confirm GPU utilization on the Monitoring tab.
7. Start the real run, checkpointing to shared storage.

If the job still fails, contact support with the cluster name and the image version. For a fabric slower than expected, include your NCCL benchmark output and the NCCL_* variables you set. See Troubleshooting TIR Slurm Cluster.