--- title: Nodes and GPU Health sidebar_label: Nodes --- import { Cpu, Filter, Thermometer, AlertTriangle, RotateCcw, Activity } from 'react-feather'; # Nodes and GPU Health The **Nodes** tab is where hardware problems surface. It shows every node with its Slurm state, a health verdict, and live DCGM metrics — down to individual GPUs — plus a per-node reboot for when one machine is bad and the rest are fine. }, { href: '#filters', label: 'Filters', icon: }, { href: '#the-node-table', label: 'The node table', icon: }, { href: '#per-gpu-drilldown', label: 'Per-GPU drilldown', icon: }, { href: '#xid-errors', label: 'XID errors', icon: }, { href: '#reboot-a-node', label: 'Reboot a node', icon: , destructive: true }, ]} /> --- ## Summary Cards | Card | Shows | Subtitle | |------|-------|----------| | **TOTAL NODES** | Nodes in the pool | `In pool` | | **ALLOCATED** | Nodes running jobs | `Running jobs` | | **AVG GPU UTIL** | Average GPU utilisation | `Active nodes` | | **TOTAL GPUS** | Total GPUs | ` · GB` | :::note On a large cluster the counts are per page The node table is paginated, and **TOTAL NODES** and **TOTAL GPUS** reflect the page you are looking at, while **ALLOCATED** is cluster-wide. On a cluster bigger than one page, take the cluster-wide totals from the [Cluster Overview tab](/docs/tir/SlurmCluster/manage/overview-tab) instead. ::: --- ## Filters | Filter | Selects | |--------|---------| | **All** | Every node | | **Failed** | Nodes Slurm is not reporting — state `UNKNOWN` | | **XID Errors** | Nodes with at least one GPU reporting an XID fault | | **Healthy** | Everything else | Counts appear beside the label when non-zero, so **Failed** and **XID Errors** tell you at a glance whether anything is wrong. :::warning "Failed" means unreported, not Slurm DOWN The **Failed** filter selects nodes in Slurm state `UNKNOWN` — the controller has no report for them. A node that Slurm has explicitly marked `DOWN` or `DRAIN` is **not** in this bucket; it appears under **Healthy** unless it also has an XID error. For the authoritative state of every node, run `sinfo -N` on the login node. ::: When any XID codes are present a chip row appears — **Filter by XID:** with **All GPUs** and one chip per code. The chips highlight which code you are looking at; use the **XID Errors** filter to narrow the table itself. --- ## The Node Table Headed **Node-Level Metrics (DCGM)**. | Column | Shows | |--------|-------| | **NODE** | `e2e_node_ (slinky-N)` and, below it, the underlying pod name | | **SLURM STATE** | `Idle`, `Allocated`, `Mixed`, or the raw state Slurm reports | | **HEALTH** | `Healthy`, `XID Error`, or `Rebooting` | | **GPU UTIL** | Average utilisation, a bar, and a per-GPU heat strip — one small bar per GPU, red for a GPU with an XID fault | | **VRAM** | Total GPU memory in use on the node, over node capacity | | **TEMP** | Average GPU temperature. Orange from 75 °C, red from 85 °C | | **POWER** | Average GPU power draw. Orange from 300 W, red from 400 W | | | **View GPUs** — opens the per-node drawer | Empty after filtering: **No nodes match the selected filter.** :::tip The heat strip finds one bad GPU fast Each small bar is one GPU. A row where seven bars are busy and one is flat — or red — points at a single GPU, not the node. That is the difference between rebooting one machine and rebuilding a cluster. ::: :::note VRAM means different things on the two tabs Here it is the **node total** over node capacity (e.g. `240 GB of 640 GB`). On the [Cluster Overview](/docs/tir/SlurmCluster/manage/overview-tab) node cards it is the **per-GPU average** over per-GPU capacity. Same underlying metric, different aggregation. ::: --- ## Per-GPU Drilldown **View GPUs** opens a drawer for one node: | Section | Shows | |---------|-------| | Header | The node name, its pod name, and badges for Slurm state and health | | Cards | **GPU Utilization**, **VRAM Used** (e.g. `0 / 640 GB`), **Temperature**, **Power Draw** | | **GPU UTILIZATION HEATMAP** | One cell per GPU — `G0`…`G7` — on a 0–100% scale | | **PER-GPU BREAKDOWN (n GPUS)** | One card per GPU with Utilisation, VRAM, Temp and Power | This is the view that answers "is GPU 3 the problem?". :::tip Use it to diagnose imbalanced training In healthy data-parallel training, all GPUs on a node should show similar utilisation. One consistently lower GPU usually means a stuck rank; one consistently higher often means uneven sharding. Both are visible here in seconds and invisible in an averaged chart. ::: The metrics come from NVIDIA DCGM, the same source as the [Monitoring tab](/docs/tir/SlurmCluster/manage/monitoring) charts — this tab shows current values, Monitoring shows them over time. --- ## XID Errors An **XID** is an NVIDIA error code reported by the driver when a GPU hits a fault. They surface here and in the red banner on this tab and the Cluster Overview tab: > **GPU / Node Health Issues Detected** > ` node(s) DOWN · node(s) with XID errors · total GPU fault events` Each fault row shows the code, the node, the GPU index, and the error message. ### What to do 1. Filter the table to **XID Errors** and note which nodes are affected. 2. Open the affected node's drawer and see whether one GPU or all of them are involved. 3. Decide: | Situation | Action | |-----------|--------| | A single node, jobs already failing on it | Drain it, then [reboot that node](#reboot-a-node) | | Several nodes at once | [**Restart All Workers**](/docs/tir/SlurmCluster/manage/actions#restart-actions) | | Errors return after a reboot | Likely failing hardware. Note the XID code and the `e2e_node_` name and contact support | | Errors appeared during one specific job | Suspect the workload — a driver-level fault triggered by that code path | :::danger Do not keep submitting onto a node with XID errors Jobs landing there will fail, sometimes after hours of apparently normal training, and a fault on one rank takes down a whole multi-node job. Drain the node first: ```bash scontrol update nodename=slinky-3 state=drain reason="XID errors" ``` Running jobs finish; nothing new is scheduled there. Resume it after the reboot: ```bash scontrol update nodename=slinky-3 state=resume ``` ::: :::tip Note the code before you reboot XID codes are diagnostic — some indicate a transient ECC event, others a failing board. The code and message are the first thing support will ask for, and they are gone from the banner once the node is rebooted. Screenshot or copy them first. ::: --- ## Reboot a Node Rebooting one node is not in the Actions menu — it lives here. 1. Open the node's drawer with **View GPUs**, or open the XID dialog from the banner. 2. Click **Reboot Node**. 3. The node shows **Rebooting** while it comes back. The XID dialog is headed **XID Errors - <n> detected** and lists each fault with its code, title and GPU before offering the reboot. :::danger A node reboot ends the jobs on that node Nothing checks the Slurm queue first. Drain the node and let its jobs finish before rebooting, unless the node is already broken enough that the jobs are lost anyway. ::: :::tip Prefer a node reboot over a cluster restart One bad node out of eight does not justify [**Restart All Workers**](/docs/tir/SlurmCluster/manage/actions#restart-actions), which ends every running job on the cluster. Reboot the one node and leave the other seven working. ::: --- ## Command-Line Equivalents ```bash ssh root@ ``` ```bash sinfo -N -o "%N %P %t %C %G %E" # node, partition, state, CPUs, GRES, reason scontrol show node slinky-0 # everything Slurm knows about one node ``` ```bash # Take a node out of service / put it back scontrol update nodename=slinky-3 state=drain reason="hardware check" scontrol update nodename=slinky-3 state=resume ``` ```bash # GPU state on a node, from a job srun --nodelist=slinky-3 --gres=gpu:8 nvidia-smi srun --nodelist=slinky-3 nvidia-smi -q | grep -i -A2 xid ``` :::tip `%E` shows the drain reason `sinfo -N -o "%N %t %E"` prints why a node was drained — often the most direct answer to "why is my job pending?" when capacity looks available. A reason you recognise from your own [prolog script](/docs/tir/SlurmCluster/slurm-configuration/prolog-epilog#a-non-zero-exit-from-a-worker-prolog-drains-the-node) means the prolog drained it. ::: --- ## Related Resources - [Cluster Overview tab](/docs/tir/SlurmCluster/manage/overview-tab) - [Monitoring](/docs/tir/SlurmCluster/manage/monitoring) - [Cluster actions](/docs/tir/SlurmCluster/manage/actions) - [Troubleshoot jobs](/docs/tir/SlurmCluster/troubleshoot/jobs) - [NVIDIA XID error reference](https://docs.nvidia.com/deploy/xid-errors/index.html)