---
title: "Troubleshooting containers on TIR Slurm Cluster"
description: "Fix container jobs on a TIR Slurm Cluster when flags are missing, an image fails to import, files or GPUs are out of reach, or a multi-node start hangs."
content_type: "troubleshooting"
product: "tir-slurm-cluster"
platform: "TIR"
url: "/tir/tir-slurm-cluster/troubleshooting/containers/"
relationships: {"troubleshooting-for":["/tir/tir-slurm-cluster/how-to/run-containers-in-jobs/","/tir/tir-slurm-cluster/how-to/run-multi-node-training/","/tir/tir-slurm-cluster/how-to/cache-container-images/"]}
---

# Troubleshooting containers on TIR Slurm Cluster

<!-- cspell:ignore sacct nodename RDMA sqsh nvcr DCGM sstat sawtoothing oauthtoken jobname FSDP -->

Fix problems with running container images inside Slurm jobs, with Enroot and Pyxis. For the steps
to run containers, see [Run containers in
jobs](/tir/tir-slurm-cluster/how-to/run-containers-in-jobs/). For other problems, see
[Troubleshooting TIR Slurm Cluster](/tir/tir-slurm-cluster/troubleshooting/tir-slurm-cluster/),
[Troubleshooting connecting to TIR Slurm Cluster](/tir/tir-slurm-cluster/troubleshooting/connecting/), and [Troubleshooting jobs on TIR Slurm Cluster](/tir/tir-slurm-cluster/troubleshooting/jobs/).

## My results are missing after a container job ends

Output written only inside the container is discarded when the step ends. A run that appears to
finish and saves nothing has almost always written to a path inside the container instead of a real mount. Write
checkpoints, logs, and model output to a mounted PFS or SFS path, and check the mount with a small
test before the real run:

```bash
srun --container-image=<image> --container-mounts=/pfs:/pfs \
     bash -c 'touch /pfs/mount-check && echo OK && rm /pfs/mount-check'
```

For the cause, see [The container cannot see my files](#the-container-cannot-see-my-files).

## The container cannot see my files

This is almost always a missing `--container-mounts`. The container starts from the image's own
filesystem, so the PFS and SFS mounts on the node appear inside it only when you pass them in:

```bash
srun --container-image=/pfs/images/pytorch.sqsh \
     --container-mounts=/pfs:/pfs \
     ls -la /pfs
```

| What you see                                            | What to do                                                                                          |
| ------------------------------------------------------- | --------------------------------------------------------------------------------------------------- |
| `No such file or directory` for a path you know exists  | Add the path to `--container-mounts`.                                                               |
| The data is there but the code cannot find it           | The path inside the container differs. Remap it: `--container-mounts=/pfs/datasets/imagenet:/data`. |
| Checkpoints save "successfully" but are gone afterwards | They went into the container's own filesystem. Point the output at a mounted path.                  |
| `Permission denied` when writing                        | The path is a dataset mount, which is read-only. Use a PFS or SFS mount.                            |

To confirm a mount is shared across nodes, write a file on the login node and read it back from every
node:

```bash
echo "written from $(hostname)" > /pfs/shared-check.txt
srun --nodes=4 --ntasks-per-node=1 bash -lc 'echo "$(hostname): $(cat /pfs/shared-check.txt)"'
```

Every node must print the same line. If one node fails to read it, that node is missing the volume. Check
the **Volumes** tab. See
[Mount and unmount volumes](/tir/tir-slurm-cluster/how-to/mount-and-unmount-volumes/).

## srun: unrecognized option '--container-image'

This error means the Pyxis plugin, which adds the container flags, is missing on this cluster.

1. Confirm it. An empty result means the plugin is absent:

   ```bash
   srun --help | grep -i container
   ```

2. Check the image version on the **Details** tab. Container support depends on the cluster image.
3. Select **Actions**, then **Update Image** to move to a current version. It is a rolling restart,
   so drain your queue first. See [Update a cluster](/tir/tir-slurm-cluster/how-to/update-a-cluster/).
4. If a current image still lacks the flags, contact support with the cluster name and the image
   version.

Leave Pyxis and `plugstack.conf` to the platform. It manages `plugstack.conf` and rebuilds it on
every cluster change, such as a scale, an image update, or a volume mount, so a hand edit is
silently reverted.

## An image will not import

The node running the job pulls the image, and the traffic goes out through the security group
attached to the cluster. Import an image once on the login node into a squash file on shared
storage, so that jobs read it from there. This avoids a slow first pull on every node, registry rate
limits, and a security group that blocks the pull:

```bash
mkdir -p /pfs/images
enroot import -o /pfs/images/pytorch-25.09.sqsh docker://nvcr.io/nvidia/pytorch:25.09-py3
```

Then use `--container-image=/pfs/images/pytorch-25.09.sqsh`. Every node reads the same file. See
[Cache container images](/tir/tir-slurm-cluster/how-to/cache-container-images/).

| Cause                                         | What to do                                                                                                    |
| --------------------------------------------- | ------------------------------------------------------------------------------------------------------------- |
| A large image and a slow first pull           | Expect several minutes and several GB. Import from the login node rather than in a job with a short `--time`. |
| A worker that lacks outbound network access   | Import on the login node and pass the resulting `.sqsh` file to jobs.                                         |
| A registry rate limit                         | You have pulled too often. Caching the image solves this.                                                     |
| A security group that blocks outbound traffic | See the next paragraph.                                                                                       |

A restrictive security group can break `docker://` pulls, and the failure looks like a plain network
timeout. Either allow the outbound traffic the registry needs in
the attached group, or import the image once to a squash file. See
[Manage network and security](/tir/tir-slurm-cluster/how-to/manage-network-and-security/).

### The import fails to authenticate

A private registry needs credentials in Enroot's per-user credentials file. The location is set by
`/etc/enroot/enroot.conf` on the cluster image and can differ between image versions, so check that
file instead of assuming a path:

```bash
cat /etc/enroot/enroot.conf 2>/dev/null | grep -i config
```

```bash
mkdir -p ~/.config/enroot
cat > ~/.config/enroot/.credentials <<'EOF'
machine nvcr.io login $oauthtoken password <NGC_API_KEY>
machine registry.example.com login <username> password <token>
EOF
chmod 600 ~/.config/enroot/.credentials
```

In the file, replace each value in angle brackets with your own: the registry key, and the login
and token for `registry.example.com`. Do the authenticated import once on the login node and have
jobs use the squash file, so that the token stays out of job scripts. Keep registry credentials out of
job scripts and prologs, because both are stored with the cluster configuration and carried into
clones. Use a read-only registry token with the narrowest scope.

### The import runs out of space

```bash
df -h /pfs
du -sh /pfs/images/*
```

Squash files are commonly 5 to 20 GB each, so prune old ones. Also check where Enroot's own cache
points, since it defaults under `$HOME`:

```bash
echo "HOME=$HOME"; cat /etc/enroot/enroot.conf 2>/dev/null
```

## There are no GPUs inside the container

```bash
srun --gres=gpu:1 --container-image=<image> nvidia-smi
```

| What you see                                                  | Cause                                                  | What to do                                      |
| ------------------------------------------------------------- | ------------------------------------------------------ | ----------------------------------------------- |
| `No devices were found`                                       | The job left out a GPU request                         | Add `--gres=gpu:N` or `--gpus-per-node=N`.      |
| `nvidia-smi: command not found`                               | The image lacks a CUDA stack                           | Use a CUDA-based image, such as an NGC image.   |
| Fewer GPUs than expected                                      | Slurm allocated fewer                                  | Check `--gres` and `scontrol show job <jobid>`. |
| torch.cuda.is_available() is `False` while `nvidia-smi` works | The image's CUDA build is incompatible with the driver | Use an image built for your GPU generation.     |
| CUDA errors only on some nodes                                | Hardware                                               | Check the **Nodes** tab for XID errors.         |

`--gres` counts GPUs per node. `--gres=gpu:8` on a four-node job means 8 GPUs on each node, 32 in
total. Use `--gpus-per-node` to say what you mean. Match the image to the hardware: a container built
for an older CUDA release may run poorly or fail to start on newer GPUs such as H200 or B200, and NGC
tags state their CUDA version. The **Nodes** tab shows the GPU model your cluster has.

## A multi-node job hangs

A multi-node job that hangs at NCCL initialisation, before any training output, usually traces to
three causes. Set NCCL_DEBUG=INFO, resubmit, and read the log:

```bash
export NCCL_DEBUG=INFO
```

| What you see       | What it means                                            |
| ------------------ | -------------------------------------------------------- |
| `NET/IB`           | InfiniBand is in use, which is correct.                  |
| `NET/Socket`       | NCCL fell back to TCP, and throughput will be far lower. |
| No transport named | The ranks failed to connect.                             |

Check the three usual causes:

| Cause                                       | What to do                                                                                    |
| ------------------------------------------- | --------------------------------------------------------------------------------------------- |
| RDMA devices are missing from the container | Add `--container-mounts=...,/dev/infiniband:/dev/infiniband`.                                 |
| NCCL_IB_HCA names differ between nodes      | List them per node with `ls /sys/class/infiniband` and confirm they are identical everywhere. |
| The rendezvous host is hard-coded           | Derive it: MASTER_ADDR=$(scontrol show hostnames "$SLURM_JOB_NODELIST" \| head -n1).          |

Then check that all nodes agree on device names:

```bash
srun --nodes=4 --ntasks-per-node=1 bash -lc 'echo "$(hostname): $(ls /sys/class/infiniband | tr "\n" " ")"'
```

Also check these:

- Is the image a squash file on shared storage? A `--container-name` is local to the node that
  created it.
- Did every node get the container? A node that failed to import stalls the whole collective.
- Is `--mpi=pmix` set? It is needed for MPI ranks, and recommended for any multi-node container job.
- Are all nodes healthy? One **Unknown** or drained node quietly shrinks the allocation.

`/tmp`, `/dev/shm`, and anything under `$HOME` while Login User Management is off are private to one
node. A container name, a squash file, or an XDG_DATA_HOME pointed at `/tmp` stays on one node, and
the symptom is a hang rather than an error.

Run an NCCL all-reduce benchmark at the scale you intend to use before you blame your code. If it is
healthy, the problem is in your job. If it is slow or fails, the problem is in the fabric configuration or the
hardware. See [Run multi-node training](/tir/tir-slurm-cluster/how-to/run-multi-node-training/).

## I get permission errors inside the container

Enroot runs containers unprivileged, so a few things a Docker run allows work differently.

| What you see                                     | Cause                                         | What to do                                                                                                                     |
| ------------------------------------------------ | --------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ |
| `Permission denied` writing inside the container | The container's root filesystem is read-only  | Write to a mounted volume.                                                                                                     |
| `apt install` fails                              | You run as your own user inside the container | Build the dependency into the image.                                                                                           |
| A script expects to be root                      | Enroot's user mapping                         | Restructure the script, or look at `--container-remap-root` if your Pyxis version offers it (`srun --help \| grep container`). |
| A file a job created has an unexpected owner     | Your host user ID carries through             | This is expected. On shared directories, use group-writable permissions.                                                       |

Bake dependencies into the image instead of installing them at job time. A `pip install` at the start
of every job is slow, fragile, and different on every node when a package version moves. Build an
image and cache it as a squash file, and every rank on every node gets an identical environment.

## Checklist for a new container job

1. `srun --help | grep -i container` shows the flags.
2. `enroot import` the image once to shared storage.
3. Run a one-node smoke test: `nvidia-smi` plus your framework's version and device count.
4. Verify the mounts with a `touch` inside the container.
5. For multi-node, run the NCCL benchmark at the target scale.
6. Confirm GPU utilization on the **Monitoring** tab.
7. Start the real run, checkpointing to shared storage.

If the job still fails, contact support with the cluster name and the image version. For a fabric
slower than expected, include your NCCL benchmark output and the NCCL_* variables you set. See
[Troubleshooting TIR Slurm Cluster](/tir/tir-slurm-cluster/troubleshooting/tir-slurm-cluster/#contact-support).