Nodes and GPU Health
The Nodes tab is where hardware problems surface. It shows every node with its Slurm state, a health verdict, and live DCGM metrics — down to individual GPUs — plus a per-node reboot for when one machine is bad and the rest are fine.
Summary Cards
| Card | Shows | Subtitle |
|---|---|---|
| TOTAL NODES | Nodes in the pool | In pool |
| ALLOCATED | Nodes running jobs | Running jobs |
| AVG GPU UTIL | Average GPU utilisation | Active nodes |
| TOTAL GPUS | Total GPUs | <GPU model> · <n> GB |
The node table is paginated, and TOTAL NODES and TOTAL GPUS reflect the page you are looking at, while ALLOCATED is cluster-wide. On a cluster bigger than one page, take the cluster-wide totals from the Cluster Overview tab instead.
Filters
| Filter | Selects |
|---|---|
| All | Every node |
| Failed | Nodes Slurm is not reporting — state UNKNOWN |
| XID Errors | Nodes with at least one GPU reporting an XID fault |
| Healthy | Everything else |
Counts appear beside the label when non-zero, so Failed and XID Errors tell you at a glance whether anything is wrong.
The Failed filter selects nodes in Slurm state UNKNOWN — the controller has no report for them.
A node that Slurm has explicitly marked DOWN or DRAIN is not in this bucket; it appears under
Healthy unless it also has an XID error.
For the authoritative state of every node, run sinfo -N on the login node.
When any XID codes are present a chip row appears — Filter by XID: with All GPUs and one chip per code. The chips highlight which code you are looking at; use the XID Errors filter to narrow the table itself.
The Node Table
Headed Node-Level Metrics (DCGM).
| Column | Shows |
|---|---|
| NODE | e2e_node_<id> (slinky-N) and, below it, the underlying pod name |
| SLURM STATE | Idle, Allocated, Mixed, or the raw state Slurm reports |
| HEALTH | Healthy, XID Error, or Rebooting |
| GPU UTIL | Average utilisation, a bar, and a per-GPU heat strip — one small bar per GPU, red for a GPU with an XID fault |
| VRAM | Total GPU memory in use on the node, over node capacity |
| TEMP | Average GPU temperature. Orange from 75 °C, red from 85 °C |
| POWER | Average GPU power draw. Orange from 300 W, red from 400 W |
| View GPUs — opens the per-node drawer |
Empty after filtering: No nodes match the selected filter.
Each small bar is one GPU. A row where seven bars are busy and one is flat — or red — points at a single GPU, not the node. That is the difference between rebooting one machine and rebuilding a cluster.
Here it is the node total over node capacity (e.g. 240 GB of 640 GB). On the
Cluster Overview node cards it is the per-GPU
average over per-GPU capacity. Same underlying metric, different aggregation.
Per-GPU Drilldown
View GPUs opens a drawer for one node:
| Section | Shows |
|---|---|
| Header | The node name, its pod name, and badges for Slurm state and health |
| Cards | GPU Utilization, VRAM Used (e.g. 0 / 640 GB), Temperature, Power Draw |
| GPU UTILIZATION HEATMAP | One cell per GPU — G0…G7 — on a 0–100% scale |
| PER-GPU BREAKDOWN (n GPUS) | One card per GPU with Utilisation, VRAM, Temp and Power |
This is the view that answers "is GPU 3 the problem?".
In healthy data-parallel training, all GPUs on a node should show similar utilisation. One consistently lower GPU usually means a stuck rank; one consistently higher often means uneven sharding. Both are visible here in seconds and invisible in an averaged chart.
The metrics come from NVIDIA DCGM, the same source as the Monitoring tab charts — this tab shows current values, Monitoring shows them over time.
XID Errors
An XID is an NVIDIA error code reported by the driver when a GPU hits a fault. They surface here and in the red banner on this tab and the Cluster Overview tab:
GPU / Node Health Issues Detected
<n> node(s) DOWN · <n> node(s) with XID errors · <n> total GPU fault events
Each fault row shows the code, the node, the GPU index, and the error message.
What to do
- Filter the table to XID Errors and note which nodes are affected.
- Open the affected node's drawer and see whether one GPU or all of them are involved.
- Decide:
| Situation | Action |
|---|---|
| A single node, jobs already failing on it | Drain it, then reboot that node |
| Several nodes at once | Restart All Workers |
| Errors return after a reboot | Likely failing hardware. Note the XID code and the e2e_node_<id> name and contact support |
| Errors appeared during one specific job | Suspect the workload — a driver-level fault triggered by that code path |
Jobs landing there will fail, sometimes after hours of apparently normal training, and a fault on one rank takes down a whole multi-node job. Drain the node first:
scontrol update nodename=slinky-3 state=drain reason="XID errors"
Running jobs finish; nothing new is scheduled there. Resume it after the reboot:
scontrol update nodename=slinky-3 state=resume
XID codes are diagnostic — some indicate a transient ECC event, others a failing board. The code and message are the first thing support will ask for, and they are gone from the banner once the node is rebooted. Screenshot or copy them first.
Reboot a Node
Rebooting one node is not in the Actions menu — it lives here.
- Open the node's drawer with View GPUs, or open the XID dialog from the banner.
- Click Reboot Node.
- The node shows Rebooting while it comes back.
The XID dialog is headed XID Errors - <n> detected and lists each fault with its code, title and GPU before offering the reboot.
Nothing checks the Slurm queue first. Drain the node and let its jobs finish before rebooting, unless the node is already broken enough that the jobs are lost anyway.
One bad node out of eight does not justify Restart All Workers, which ends every running job on the cluster. Reboot the one node and leave the other seven working.
Command-Line Equivalents
ssh root@<cluster-ip>
sinfo -N -o "%N %P %t %C %G %E" # node, partition, state, CPUs, GRES, reason
scontrol show node slinky-0 # everything Slurm knows about one node
# Take a node out of service / put it back
scontrol update nodename=slinky-3 state=drain reason="hardware check"
scontrol update nodename=slinky-3 state=resume
# GPU state on a node, from a job
srun --nodelist=slinky-3 --gres=gpu:8 nvidia-smi
srun --nodelist=slinky-3 nvidia-smi -q | grep -i -A2 xid
%E shows the drain reasonsinfo -N -o "%N %t %E" prints why a node was drained — often the most direct answer to "why is my
job pending?" when capacity looks available. A reason you recognise from your own
prolog script
means the prolog drained it.