Troubleshooting TIR Slurm Cluster

View as Markdown

Find the symptom you see, then follow the link. SSH, job, and container problems each have their own troubleshooting page.

Symptom	Where to look
Output or files are gone after a restart	My output disappeared after a restart
The cluster is stuck in Creating, or shows Failed	My cluster is stuck in Creating, My cluster shows Failed
A node is stuck Unknown	A node is stuck Unknown
An action is missing, or is refused with a message	An action is missing from the Actions menu, and the messages further down
`ssh` times out, is refused, or says `Permission denied`	Troubleshooting connecting to TIR Slurm Cluster
A named login user cannot log in	Troubleshooting connecting to TIR Slurm Cluster
A job stays Pending, fails at once, or runs out of memory	Troubleshooting jobs on TIR Slurm Cluster
A node keeps getting drained	Troubleshooting jobs on TIR Slurm Cluster
`srun: unrecognized option '--container-image'`	Troubleshooting containers on TIR Slurm Cluster
A container cannot see your files or GPUs	Troubleshooting containers on TIR Slurm Cluster
A multi-node job hangs at start-up	Troubleshooting containers on TIR Slurm Cluster
A node shows XID Error	Check cluster and node health
GPU use stays near zero while a job runs	Monitor a cluster

For short answers on cost, limits, and settings, see the TIR Slurm Cluster FAQ.

My output disappeared after a restart

Work written outside a mounted volume is erased whenever the node is recreated. That covers /root, /tmp, and /home while Login User Management is off, and it is the commonest reason output seems to have disappeared. Restarting workers, the login service, or the cluster, scaling down, updating the image, and mounting or unmounting a volume all recreate nodes. A PFS or SFS mount, and a dataset mount, are the same at the same path on every node and keep their contents across restarts. Write checkpoints, logs, and results there. Check where the output was written before you assume anything is broken.

I am not sure what is wrong

Five checks answer most problems:

Is the cluster Running? The status shows on the Details tab. Most tabs and actions need Running.
Are the nodes healthy? Open Cluster Overview. An Unknown or Down node sits out of scheduling, which leaves jobs pending for reasons that are hard to spot. On the login node, sinfo -N -o "%N %t %E" shows why a node was drained.
Is your storage mounted where you think? On the login node, run the first command to list the mounts and the second to test a write:
```
df -h | grep -E '/pfs|/shared'
echo test > /pfs/write-check && rm /pfs/write-check && echo "writable"
```

What does Slurm say? The reason column of squeue is usually the answer:

squeue -o "%.18i %.9P %.30j %.8T %.10M %.20R"
scontrol show job <jobid>
sacct -j <jobid> --format=JobID,State,ExitCode,Elapsed,NodeList

What do the logs say? Open the Logs tab for the worker the job landed on, and read your own --output file. See See jobs and logs.

My cluster is stuck in Creating

Creating a cluster normally takes a few minutes.

Wait a little longer. The cluster list refreshes itself while a cluster is Creating.
Select Actions, then Reconfigure Cluster. It is available while the cluster is still being set up, and it applies the intended state again.
If the cluster stays in Creating, contact support with the cluster name and how long it has been in Creating.

Billing starts only once the create succeeds, so a failed create is free. See Cluster statuses and actions.

My cluster shows Failed

A Failed cluster stays Failed, and its compute is free, so you can take your time. Its reserved nodes are released, and a clone gets you a Running cluster with the same setup.

Select Actions, then Clone Cluster. You get the same configuration in a fresh create form.
Change anything that might have caused the failure. The most common cause is a plan whose nodes are all taken: try a different GPU type, fewer nodes, or a TIR Private Cluster.
Create the new cluster and confirm it reaches Running.
Delete the failed record.

See Clone a cluster.

A GPU plan shows Inventory not available

A GPU plan that shows “Inventory not available” cannot be selected at that moment, because every node of that type is taken. Retry later, or reserve capacity with a TIR Private Cluster. When you ask for more private-cluster worker nodes than are free, the form says “The requested node count exceeds current availability. Please reduce the node count.”

A node is stuck Unknown

Unknown means the controller has yet to get a report from that node. It is normal for a few minutes after a restart, a scale-up, or an image update.

Wait a few minutes.
Check that worker’s log on the Logs tab.
Reboot the node from the Nodes tab. See Check cluster and node health.
Select Restart All Workers only if several nodes are affected, because it ends every running job.

After a reboot, the node shows Rebooting for 5 minutes and then its real state. TIR makes one reboot attempt. If a node still fails to register after a reboot, contact support and include that worker’s log.

Capacity looks free but nothing schedules

The constraint can be a partition rather than capacity. On the login node, run:

sinfo
scontrol show partition <name>

sinfo shows whether a partition is UP, and scontrol shows its nodes, MaxTime, and state. A partition in DOWN or INACTIVE keeps queued jobs waiting, and one in DRAIN refuses new submissions. A partition whose nodes were all removed, for example by a scale-down, drops out of Slurm; edit it to name nodes that still exist. A node in drain can also be the cause. See Manage partitions and Troubleshooting jobs on TIR Slurm Cluster.

The cluster’s status explains most missing actions. Scale Cluster missing on a Running cluster means a committed plan. Convert to Committed missing means the cluster is already committed, is on a TIR Private Cluster, or is Failed, Terminated, or Terminating. A page with no actions at all means you lack update permission on Training Clusters in the project. See Cluster statuses and actions.

Scale cluster is not supported for committed cluster

A committed cluster keeps its size for the whole term, so size the cluster before you commit. To get a different size, create a second hourly cluster. See Scale a cluster.

Cluster plan upgrade not allowed for committed clusters

The cluster is on a committed plan, and its plan stays until the term ends.

Convert to committed is not supported for committed cluster

The cluster is already committed. See Convert a cluster to committed.

Inventory not available for the selected configuration

A scale-up needs free nodes of the right type on the same rack as your existing nodes, and every one is taken at the moment. Try again later. For guaranteed capacity, use a TIR Private Cluster.

A message about credit appears when I confirm

Project credit is too low for the requested nodes, or for the upfront committed charge. TIR checks credit before it reserves any node. Add credit and try again. See Pay now in TIR.

Slurm configuration can only be updated on a running cluster

Slurm Configuration works while the cluster is Running. Wait for Running, then apply the change again. See Change the Slurm configuration.

A slurm.conf line or a partition is rejected

TIR rejects a key the platform manages with “is managed by the platform and cannot be overridden”, and a partition with a reserved name with “is reserved by the platform and cannot be redefined”. It also rejects a line missing =, a duplicate key, or a block over 8 KB. Fix the line and apply again. See Change the Slurm configuration and Manage partitions. You add partitions once the cluster is running, because a partition names nodes that exist only then.

Minimum one node required for cluster to remain in running

A scale-down must leave at least one node. Keep one, or terminate the cluster instead.

Invalid request, Cluster does not exist

The cluster was deleted, or it belongs to another project. Check the project selector at the top of the console.

Terminate or delete is refused while a VPC IP is attaching

A VPC IP that is still attaching blocks both terminate and delete. Wait for the attach to finish, then try again.

Cannot remove storage. At least one storage volume must remain mounted on the cluster.

A cluster keeps at least one mounted volume. Mount a replacement first, then unmount. Unmounting a volume keeps the volume and its data.

The volume holds the home directories for Login User Management. Turn Login User Management off first, then unmount the volume. See Manage login users.

At least one IP (reserved or floating) must be attached.

A cluster keeps at least one IP. Attach a reserved IP first, then detach the other. An error that at least one security group is required means the same for security groups: attach a replacement before you remove the last one. See Manage network and security.

Contact support

To contact support, raise a ticket from Support in the TIR sidebar, or email cloud-platform@e2enetworks.com. For the ticket steps, see Create a support ticket. Give the cluster name and the project, plus the detail for the problem:

Problem	Also include
Stuck in Creating or Terminating	How long it has been, and whether you tried Reconfigure Cluster
Repeated XID errors	The XID code, the node’s machine identifier (it starts with e2e_node_), and whether a reboot helped
A node that fails to register	That worker’s log from the Logs tab
Fabric performance far below expectation	Your NCCL benchmark output and the NCCL_* variables you set
A configuration change that fails to apply	The exact configuration and the error message

Download a worker’s log before you restart anything that recreates its node, because the earlier log is gone once the node is replaced.