Check cluster and node health

Find out whether your TIR Slurm Cluster is healthy and busy, which node or GPU is at fault when something goes wrong, and how to reboot one node while the rest keep running. The Cluster Overview tab answers “is the cluster healthy, and is it busy?” at a glance, and the Nodes tab goes down to individual GPUs.

Before you begin

• A cluster to inspect. Cluster Overview opens only on a Running cluster. Nodes opens as soon as the workers start joining, during Initializing Worker Nodes. See Cluster statuses and actions.
• Update permission on Training Clusters in the project, if you want to reboot a node.

Read the Cluster Overview tab

Under Training Cluster in the TIR sidebar, open the cluster, then open Cluster Overview. The tab is a snapshot; select refresh in the page header to reload it. For trends over time, use Monitor a cluster.

Six totals head the tab:

• Total GPUs, as nodes times GPUs per node.
• Allocated GPUs, which counts the nodes in the Slurm state Allocated or Mixed. Compare it with the node count in Total GPUs. For per-GPU allocation, use the Nodes tab or scontrol show nodes.
• GPU Util, the average GPU utilization.
• GPU Memory, the average GPU memory used over the per-GPU capacity.
• Running Jobs.
• Idle Nodes.

Idle Nodes and GPU Util together show whether the cluster is under-used. Idle nodes while jobs are pending point to a request that needs more than the free nodes offer. Every node busy with low GPU utilization means jobs are holding GPUs and leaving them idle. Open Monitoring to see the pattern.

Read the node summaries

Below the totals are summaries of up to four nodes; the rest are on the Nodes tab. Each summary gives the node’s Slurm state:

State	Meaning
Idle	Up and free for jobs.
Allocated	Fully allocated to jobs.
Mixed	Partly allocated.
Down	Slurm has marked the node out of service.
Unknown	Slurm has no report for the node. Usually the worker is still registering, or the controller has lost contact with it.

A node shows Unknown for a short while after a restart, a scale-up, or an image update. If it stays Unknown after a few minutes, check that worker’s log on the Logs tab, then reboot the node with Reboot Node. Use Restart All Workers only if several nodes are affected, because it ends every running job.

Each node summary also gives GPU Util, VRAM, and Temp / Power. VRAM here is the average memory used on each GPU against one GPU’s capacity. Temperature is highlighted from 75 °C and again from 85 °C, and power from 300 W and again from 400 W. One node that stays highlighted for temperature while its neighbors stay normal usually has a cooling or seating problem rather than a heavier job. Check the Nodes tab for XID errors, then note the node name before you contact support.

Slurm Partition Info has your partitions and the platform partition. See Manage partitions.

Watch for the GPU health banner

An XID error is an NVIDIA error code for a GPU fault. When any GPU reports one, a GPU / Node Health Issues Detected banner names the node and counts each fault. The banner covers XID faults only, so a node that is Down or Unknown for another reason appears only in the node summaries. Read those too.

Check before you submit a large job

This check takes 20 seconds:

1. Look for the GPU / Node Health Issues Detected banner. If it is there, go to the Nodes tab before you submit anything expensive.
2. Scan the node summaries for nodes that are Unknown or Down.
3. Check that Idle Nodes has room for what you are about to submit.
4. Check that the partition you target is UP and that its Max time allows your --time.
5. Expect to queue if GPU Util is already high.

Read the Nodes tab

The Nodes tab has every node with its Slurm state, a health verdict, and live GPU metrics. Four totals head it: Total Nodes, Allocated, which counts nodes running jobs, Avg GPU Util, and Total GPUs, which shows the GPU model and per-GPU memory as a subtitle. On a large cluster the node table splits into pages, and the totals can follow the page you are on, so take cluster-wide totals from Cluster Overview.

The filters are:

• All shows every node.
• Failed shows nodes whose Slurm state is Unknown, meaning the controller has no report. A node that Slurm marked DOWN or DRAIN is outside this group, so for the state of every node, run sinfo -N on the login node.
• XID Errors shows nodes with at least one GPU reporting an XID fault.
• Healthy shows everything else.

When nodes report XID codes, you can also filter by code.

For each node, the table gives its Slurm State, Health, GPU Util as a heat strip, VRAM, Temp, and Power. Slurm State reads Idle, Allocated, Mixed, or the raw state Slurm reports. Health reads Healthy, XID Error, or Rebooting. Each node is named by its machine identifier with the Slurm name in brackets, such as e2e_node_682 (slinky-0). Use the Slurm name in job scripts and partitions, and the machine identifier when you report a hardware problem. Here VRAM is the node total over the node’s capacity, while on Cluster Overview it is the per-GPU average.

The heat strip finds one bad GPU quickly. It has one small bar per GPU. A row where most bars are busy and one is flat, or marked for an XID fault, points to a single GPU rather than the whole node. That is the difference between rebooting one machine and rebuilding a cluster.

Look at one node’s GPUs

Select View GPUs on a node for its GPU Utilization, VRAM Used, Temperature, and Power Draw, with a heatmap of one cell per GPU and a breakdown for each GPU. In healthy data-parallel training, all GPUs on a node show similar utilization. One consistently lower GPU usually means a stuck rank, and one consistently higher often means uneven sharding.

The metrics come from NVIDIA DCGM. This tab has current values, and Monitoring has them over time.

Respond to XID errors

Each XID fault comes with its code, the node, the GPU index, and the error message. Some XID codes are a transient event, and some point to failing hardware.

1. Filter the table to XID Errors and note which nodes are affected.
2. Open each affected node’s drawer and see whether one GPU or all of them are involved.
3. Copy the XID code and message before you reboot. They are the first thing support asks for, and they are gone from the banner once the node reboots.
4. Choose a response:

Situation	Action
One node, with jobs already failing on it	Drain the node, then reboot that node.
Several nodes at once	Select Restart All Workers. See Update a cluster.
Errors return after a reboot	This is likely failing hardware. Note the XID code and the node’s machine identifier, which starts with e2e_node_, and contact support.
Errors began during one specific job	Suspect the job, because that code path may have triggered a driver-level fault.

Stop sending jobs to a node with XID errors. Jobs that land there fail, sometimes after hours of apparently normal training, and a fault on one rank takes down a whole multi-node job. Drain the node first, from the login node:

scontrol update nodename=slinky-3 state=drain reason="XID errors"

Running jobs finish, and Slurm schedules new jobs on other nodes. Put the node back after the reboot:

scontrol update nodename=slinky-3 state=resume

Reboot one node

Reboot Node is on a node’s drawer and in the XID dialog, rather than on the cluster’s Actions menu. Prefer it to Restart All Workers, which ends every running job on the cluster: it deals with one bad node out of eight and leaves the jobs on the other seven running.

The reboot goes ahead whatever is in the Slurm queue, so jobs on that node end. Drain the node and let its jobs finish first, unless the node is already broken enough that the jobs are lost anyway.

1. Open the node with View GPUs, or open the XID dialog from the banner, which lists each fault before it offers the reboot.
2. Select Reboot Node.

While the node comes back, its Health is Rebooting. After 5 minutes the node reports its current state again, whatever it is. TIR tries the reboot once, and that state is the only sign of how it went, so check it then. You can start another reboot once the current one has finished. If the node stays down, see Troubleshooting TIR Slurm Cluster.

Use the command line

The same checks work from the login node:

ssh root@<cluster-ip>

Replace <cluster-ip> with the cluster’s address from Connection Details.

sinfo -N -o "%N %P %t %C %G %E"
scontrol show node slinky-0

The first command prints the node, partition, state, CPUs, GPUs, and the reason a node was drained. That reason is often the most direct answer to “why is my job pending?” when capacity looks free. A reason you recognize from your own prolog script means the prolog drained the node; see Add prolog and epilog scripts. To read GPU state from a job:

srun --nodelist=slinky-3 --gres=gpu:8 nvidia-smi
srun --nodelist=slinky-3 nvidia-smi -q | grep -i -A2 xid

To see activity over time rather than a snapshot, continue to Monitor a cluster. For a node that stays Unknown, see Troubleshooting TIR Slurm Cluster. The NVIDIA XID error reference explains what each code means.