---
title: "Troubleshooting TIR Slurm Cluster"
description: "Fix TIR Slurm Cluster problems from the symptom you see, starting with five checks, then clusters stuck in Creating, Failed clusters, and refused actions."
content_type: "troubleshooting"
product: "tir-slurm-cluster"
platform: "TIR"
url: "/tir/tir-slurm-cluster/troubleshooting/tir-slurm-cluster/"
relationships: {"troubleshooting-for":["/tir/tir-slurm-cluster/how-to/update-a-cluster/","/tir/tir-slurm-cluster/how-to/scale-a-cluster/","/tir/tir-slurm-cluster/how-to/clone-a-cluster/","/tir/tir-slurm-cluster/how-to/convert-a-cluster-to-committed/","/tir/tir-slurm-cluster/how-to/terminate-or-delete-a-cluster/","/tir/tir-slurm-cluster/how-to/check-cluster-and-node-health/"]}
---

# Troubleshooting TIR Slurm Cluster

<!-- cspell:ignore sacct nodename RDMA sqsh nvcr DCGM sstat sawtoothing oauthtoken jobname FSDP -->

Find the symptom you see, then follow the link. SSH, job, and container problems each have their own
troubleshooting page.

| Symptom                                                       | Where to look                                                                                                            |
| ------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ |
| Output or files are gone after a restart                      | [My output disappeared after a restart](#my-output-disappeared-after-a-restart)                                          |
| The cluster is stuck in **Creating**, or shows **Failed**     | [My cluster is stuck in Creating](#my-cluster-is-stuck-in-creating), [My cluster shows Failed](#my-cluster-shows-failed) |
| A node is stuck **Unknown**                                   | [A node is stuck Unknown](#a-node-is-stuck-unknown)                                                                      |
| An action is missing, or is refused with a message            | [An action is missing from the Actions menu](#an-action-is-missing-from-the-actions-menu), and the messages further down |
| `ssh` times out, is refused, or says `Permission denied`      | [Troubleshooting connecting to TIR Slurm Cluster](/tir/tir-slurm-cluster/troubleshooting/connecting/)                    |
| A named login user cannot log in                              | [Troubleshooting connecting to TIR Slurm Cluster](/tir/tir-slurm-cluster/troubleshooting/connecting/)                    |
| A job stays **Pending**, fails at once, or runs out of memory | [Troubleshooting jobs on TIR Slurm Cluster](/tir/tir-slurm-cluster/troubleshooting/jobs/)                                |
| A node keeps getting drained                                  | [Troubleshooting jobs on TIR Slurm Cluster](/tir/tir-slurm-cluster/troubleshooting/jobs/)                                |
| `srun: unrecognized option '--container-image'`               | [Troubleshooting containers on TIR Slurm Cluster](/tir/tir-slurm-cluster/troubleshooting/containers/)                    |
| A container cannot see your files or GPUs                     | [Troubleshooting containers on TIR Slurm Cluster](/tir/tir-slurm-cluster/troubleshooting/containers/)                    |
| A multi-node job hangs at start-up                            | [Troubleshooting containers on TIR Slurm Cluster](/tir/tir-slurm-cluster/troubleshooting/containers/)                    |
| A node shows **XID Error**                                    | [Check cluster and node health](/tir/tir-slurm-cluster/how-to/check-cluster-and-node-health/)                            |
| GPU use stays near zero while a job runs                      | [Monitor a cluster](/tir/tir-slurm-cluster/how-to/monitor-a-cluster/)                                                    |

For short answers on cost, limits, and settings, see the [TIR Slurm Cluster
FAQ](/tir/tir-slurm-cluster/faq/).

## My output disappeared after a restart

Work written outside a mounted volume is erased whenever the node is recreated. That covers `/root`,
`/tmp`, and `/home` while Login User Management is off, and it is the commonest reason output seems to have
disappeared. Restarting workers, the login service, or the cluster, scaling down, updating the
image, and mounting or unmounting a volume all recreate nodes. A PFS or SFS mount, and a dataset
mount, are the same at the same path on every node and keep their contents across restarts. Write
checkpoints, logs, and results there. Check where the output was written before you assume anything is broken.

## I am not sure what is wrong

Five checks answer most problems:

1. **Is the cluster Running?** The status shows on the **Details** tab. Most tabs and actions need
   **Running**.
2. **Are the nodes healthy?** Open **Cluster Overview**. An **Unknown** or **Down** node sits out of
   scheduling, which leaves jobs pending for reasons that are hard to spot. On the login node, `sinfo -N -o "%N %t %E"`
   shows why a node was drained.
3. **Is your storage mounted where you think?** On the login node, run the first command to list the
   mounts and the second to test a write:

   ```bash
   df -h | grep -E '/pfs|/shared'
   echo test > /pfs/write-check && rm /pfs/write-check && echo "writable"
   ```

4. **What does Slurm say?** The reason column of `squeue` is usually the answer:

   ```bash
   squeue -o "%.18i %.9P %.30j %.8T %.10M %.20R"
   scontrol show job <jobid>
   sacct -j <jobid> --format=JobID,State,ExitCode,Elapsed,NodeList
   ```

5. **What do the logs say?** Open the **Logs** tab for the worker the job landed on, and read your
   own `--output` file. See [See jobs and logs](/tir/tir-slurm-cluster/how-to/see-jobs-and-logs/).

## My cluster is stuck in Creating

Creating a cluster normally takes a few minutes.

1. Wait a little longer. The cluster list refreshes itself while a cluster is **Creating**.
2. Select **Actions**, then **Reconfigure Cluster**. It is available while the cluster is still
   being set up, and it applies the intended state again.
3. If the cluster stays in **Creating**, contact support with the cluster name and how long it has
   been in **Creating**.

Billing starts only once the create succeeds, so a failed create is free. See
[Cluster statuses and actions](/tir/tir-slurm-cluster/concepts/cluster-statuses-and-actions/).

## My cluster shows Failed

A **Failed** cluster stays **Failed**, and its compute is free, so you can take your time. Its
reserved nodes are released, and a clone gets you a **Running** cluster with the same setup.

1. Select **Actions**, then **Clone Cluster**. You get the same configuration in a fresh create form.
2. Change anything that might have caused the failure. The most common cause is a plan whose nodes are
   all taken: try a different GPU type, fewer nodes, or a TIR Private Cluster.
3. Create the new cluster and confirm it reaches **Running**.
4. Delete the failed record.

See [Clone a cluster](/tir/tir-slurm-cluster/how-to/clone-a-cluster/).

## A GPU plan shows Inventory not available

A GPU plan that shows "Inventory not available" cannot be selected at that moment,
because every node of that type is taken. Retry later, or reserve capacity with a TIR Private Cluster. When you ask
for more private-cluster worker nodes than are free, the form says "The requested node count exceeds
current availability. Please reduce the node count."

## A node is stuck Unknown

**Unknown** means the controller has yet to get a report from that node. It is normal for a few minutes after a
restart, a scale-up, or an image update.

1. Wait a few minutes.
2. Check that worker's log on the **Logs** tab.
3. Reboot the node from the **Nodes** tab. See
   [Check cluster and node health](/tir/tir-slurm-cluster/how-to/check-cluster-and-node-health/).
4. Select **Restart All Workers** only if several nodes are affected, because it ends every running
   job.

After a reboot, the node shows **Rebooting** for 5 minutes and then its real state. TIR makes one
reboot attempt. If a node still fails to register after a reboot, contact support and include
that worker's log.

## Capacity looks free but nothing schedules

The constraint can be a partition rather than capacity. On the login node, run:

```bash
sinfo
scontrol show partition <name>
```

`sinfo` shows whether a partition is `UP`, and `scontrol` shows its nodes, MaxTime, and state. A
partition in `DOWN` or `INACTIVE` keeps queued jobs waiting, and one in `DRAIN` refuses new submissions.
A partition whose nodes were all removed, for example by a scale-down, drops out of Slurm; edit it
to name nodes that still exist. A node in `drain` can also be the cause. See [Manage
partitions](/tir/tir-slurm-cluster/how-to/manage-partitions/) and [Troubleshooting jobs on TIR Slurm Cluster](/tir/tir-slurm-cluster/troubleshooting/jobs/#capacity-looks-free-but-nothing-schedules).

## An action is missing from the Actions menu

The cluster's status explains most missing actions. **Scale Cluster** missing on a **Running**
cluster means a committed plan. **Convert to Committed** missing means the cluster is already
committed, is on a TIR Private Cluster, or is **Failed**, **Terminated**, or **Terminating**. A page
with no actions at all means you lack update permission on Training Clusters in the project. See
[Cluster statuses and actions](/tir/tir-slurm-cluster/concepts/cluster-statuses-and-actions/).

## Scale cluster is not supported for committed cluster

A committed cluster keeps its size for the whole term, so size the cluster before you commit. To get
a different size, create a second hourly cluster. See
[Scale a cluster](/tir/tir-slurm-cluster/how-to/scale-a-cluster/).

## Cluster plan upgrade not allowed for committed clusters

The cluster is on a committed plan, and its plan stays until the term ends.

## Convert to committed is not supported for committed cluster

The cluster is already committed. See
[Convert a cluster to committed](/tir/tir-slurm-cluster/how-to/convert-a-cluster-to-committed/).

## Inventory not available for the selected configuration

A scale-up needs free nodes of the right type on the same rack as your existing nodes, and every one is
taken at the moment. Try again later. For guaranteed capacity, use a TIR Private Cluster.

## A message about credit appears when I confirm

Project credit is too low for the requested nodes, or for the upfront committed charge.
TIR checks credit before it reserves any node. Add credit and try again. See [Pay now in TIR](/tir/tir-billing/how-to/pay-now/).

## Slurm configuration can only be updated on a running cluster

**Slurm Configuration** works while the cluster is **Running**. Wait for **Running**, then apply the
change again. See
[Change the Slurm configuration](/tir/tir-slurm-cluster/how-to/change-the-slurm-configuration/).

## A slurm.conf line or a partition is rejected

TIR rejects a key the platform manages with "is managed by the platform and cannot be overridden",
and a partition with a reserved name with "is reserved by the platform and cannot be redefined". It
also rejects a line missing `=`, a duplicate key, or a block over 8 KB. Fix the line and apply
again. See
[Change the Slurm configuration](/tir/tir-slurm-cluster/how-to/change-the-slurm-configuration/) and
[Manage partitions](/tir/tir-slurm-cluster/how-to/manage-partitions/). You add partitions once the
cluster is running, because a partition names nodes that exist only then.

## Minimum one node required for cluster to remain in running

A scale-down must leave at least one node. Keep one, or terminate the cluster instead.

## Invalid request, Cluster does not exist

The cluster was deleted, or it belongs to another project. Check the project selector at the top of
the console.

## Terminate or delete is refused while a VPC IP is attaching

A VPC IP that is still attaching blocks both terminate and delete. Wait for the attach to finish,
then try again.

## Cannot remove storage. At least one storage volume must remain mounted on the cluster.

A cluster keeps at least one mounted volume. Mount a replacement first, then unmount. Unmounting a
volume keeps the volume and its data.

## Cannot unmount this storage as it is used for persistent login

The volume holds the home directories for Login User Management. Turn Login User Management off
first, then unmount the volume. See
[Manage login users](/tir/tir-slurm-cluster/how-to/manage-login-users/).

## At least one IP (reserved or floating) must be attached.

A cluster keeps at least one IP. Attach a reserved IP first, then detach the other. An error
that at least one security group is required means the same for security groups: attach a
replacement before you remove the last one. See
[Manage network and security](/tir/tir-slurm-cluster/how-to/manage-network-and-security/).

## Contact support

To contact support, raise a ticket from **Support** in the TIR sidebar, or email
cloud-platform@e2enetworks.com. For the ticket steps, see [Create a support
ticket](/tir/tir-support/how-to/create-a-support-ticket/). Give the cluster name and the project, plus the detail for the
problem:

| Problem                                    | Also include                                                                                        |
| ------------------------------------------ | --------------------------------------------------------------------------------------------------- |
| Stuck in **Creating** or **Terminating**   | How long it has been, and whether you tried **Reconfigure Cluster**                                 |
| Repeated XID errors                        | The XID code, the node's machine identifier (it starts with e2e_node_), and whether a reboot helped |
| A node that fails to register              | That worker's log from the **Logs** tab                                                             |
| Fabric performance far below expectation   | Your NCCL benchmark output and the NCCL_* variables you set                                         |
| A configuration change that fails to apply | The exact configuration and the error message                                                       |

Download a worker's log before you restart anything that recreates its node, because the earlier log
is gone once the node is replaced.