--- title: "Check cluster and node health" description: "Read the Cluster Overview and Nodes tabs of a TIR Slurm Cluster to see whether it is healthy and busy, find a bad GPU, and reboot a single node." content_type: "how-to" product: "tir-slurm-cluster" platform: "TIR" url: "/tir/tir-slurm-cluster/how-to/check-cluster-and-node-health/" topic: ["monitor"] --- # Check cluster and node health Find out whether your TIR Slurm Cluster is healthy and busy, which node or GPU is at fault when something goes wrong, and how to reboot one node while the rest keep running. The **Cluster Overview** tab answers "is the cluster healthy, and is it busy?" at a glance, and the **Nodes** tab goes down to individual GPUs. ## Before you begin - A cluster to inspect. **Cluster Overview** opens only on a **Running** cluster. **Nodes** opens as soon as the workers start joining, during **Initializing Worker Nodes**. See [Cluster statuses and actions](/tir/tir-slurm-cluster/concepts/cluster-statuses-and-actions/). - Update permission on Training Clusters in the project, if you want to reboot a node. ## Read the Cluster Overview tab Under **Training Cluster** in the TIR sidebar, open the cluster, then open **Cluster Overview**. The tab is a snapshot; select refresh in the page header to reload it. For trends over time, use [Monitor a cluster](/tir/tir-slurm-cluster/how-to/monitor-a-cluster/). Six totals head the tab: - **Total GPUs**, as nodes times GPUs per node. - **Allocated GPUs**, which counts the nodes in the Slurm state **Allocated** or **Mixed**. Compare it with the node count in **Total GPUs**. For per-GPU allocation, use the **Nodes** tab or `scontrol show nodes`. - **GPU Util**, the average GPU utilization. - **GPU Memory**, the average GPU memory used over the per-GPU capacity. - **Running Jobs**. - **Idle Nodes**. **Idle Nodes** and **GPU Util** together show whether the cluster is under-used. Idle nodes while jobs are pending point to a request that needs more than the free nodes offer. Every node busy with low GPU utilization means jobs are holding GPUs and leaving them idle. Open **Monitoring** to see the pattern. ### Read the node summaries Below the totals are summaries of up to four nodes; the rest are on the **Nodes** tab. Each summary gives the node's Slurm state: | State | Meaning | | ------------- | ---------------------------------------------------------------------------------------------------------------------- | | **Idle** | Up and free for jobs. | | **Allocated** | Fully allocated to jobs. | | **Mixed** | Partly allocated. | | **Down** | Slurm has marked the node out of service. | | **Unknown** | Slurm has no report for the node. Usually the worker is still registering, or the controller has lost contact with it. | A node shows **Unknown** for a short while after a restart, a scale-up, or an image update. If it stays **Unknown** after a few minutes, check that worker's log on the **Logs** tab, then reboot the node with **Reboot Node**. Use **Restart All Workers** only if several nodes are affected, because it ends every running job. Each node summary also gives **GPU Util**, **VRAM**, and **Temp / Power**. **VRAM** here is the average memory used on each GPU against one GPU's capacity. Temperature is highlighted from 75 °C and again from 85 °C, and power from 300 W and again from 400 W. One node that stays highlighted for temperature while its neighbors stay normal usually has a cooling or seating problem rather than a heavier job. Check the **Nodes** tab for XID errors, then note the node name before you contact support. **Slurm Partition Info** has your partitions and the platform partition. See [Manage partitions](/tir/tir-slurm-cluster/how-to/manage-partitions/). ### Watch for the GPU health banner An XID error is an NVIDIA error code for a GPU fault. When any GPU reports one, a **GPU / Node Health Issues Detected** banner names the node and counts each fault. The banner covers XID faults only, so a node that is Down or Unknown for another reason appears only in the node summaries. Read those too. ### Check before you submit a large job This check takes 20 seconds: 1. Look for the **GPU / Node Health Issues Detected** banner. If it is there, go to the **Nodes** tab before you submit anything expensive. 2. Scan the node summaries for nodes that are **Unknown** or **Down**. 3. Check that **Idle Nodes** has room for what you are about to submit. 4. Check that the partition you target is `UP` and that its **Max time** allows your `--time`. 5. Expect to queue if **GPU Util** is already high. ## Read the Nodes tab The **Nodes** tab has every node with its Slurm state, a health verdict, and live GPU metrics. Four totals head it: **Total Nodes**, **Allocated**, which counts nodes running jobs, **Avg GPU Util**, and **Total GPUs**, which shows the GPU model and per-GPU memory as a subtitle. On a large cluster the node table splits into pages, and the totals can follow the page you are on, so take cluster-wide totals from **Cluster Overview**. The filters are: - **All** shows every node. - **Failed** shows nodes whose Slurm state is **Unknown**, meaning the controller has no report. A node that Slurm marked `DOWN` or `DRAIN` is outside this group, so for the state of every node, run `sinfo -N` on the login node. - **XID Errors** shows nodes with at least one GPU reporting an XID fault. - **Healthy** shows everything else. When nodes report XID codes, you can also filter by code. For each node, the table gives its **Slurm State**, **Health**, **GPU Util** as a heat strip, **VRAM**, **Temp**, and **Power**. **Slurm State** reads **Idle**, **Allocated**, **Mixed**, or the raw state Slurm reports. **Health** reads **Healthy**, **XID Error**, or **Rebooting**. Each node is named by its machine identifier with the Slurm name in brackets, such as e2e_node_682 (slinky-0). Use the Slurm name in job scripts and partitions, and the machine identifier when you report a hardware problem. Here **VRAM** is the node total over the node's capacity, while on **Cluster Overview** it is the per-GPU average. The heat strip finds one bad GPU quickly. It has one small bar per GPU. A row where most bars are busy and one is flat, or marked for an XID fault, points to a single GPU rather than the whole node. That is the difference between rebooting one machine and rebuilding a cluster. ### Look at one node's GPUs Select **View GPUs** on a node for its **GPU Utilization**, **VRAM Used**, **Temperature**, and **Power Draw**, with a heatmap of one cell per GPU and a breakdown for each GPU. In healthy data-parallel training, all GPUs on a node show similar utilization. One consistently lower GPU usually means a stuck rank, and one consistently higher often means uneven sharding. The metrics come from NVIDIA DCGM. This tab has current values, and **Monitoring** has them over time. ## Respond to XID errors Each XID fault comes with its code, the node, the GPU index, and the error message. Some XID codes are a transient event, and some point to failing hardware. 1. Filter the table to **XID Errors** and note which nodes are affected. 2. Open each affected node's drawer and see whether one GPU or all of them are involved. 3. Copy the XID code and message before you reboot. They are the first thing support asks for, and they are gone from the banner once the node reboots. 4. Choose a response: | Situation | Action | | ----------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | | One node, with jobs already failing on it | Drain the node, then reboot that node. | | Several nodes at once | Select **Restart All Workers**. See [Update a cluster](/tir/tir-slurm-cluster/how-to/update-a-cluster/). | | Errors return after a reboot | This is likely failing hardware. Note the XID code and the node's machine identifier, which starts with e2e_node_, and contact support. | | Errors began during one specific job | Suspect the job, because that code path may have triggered a driver-level fault. | Stop sending jobs to a node with XID errors. Jobs that land there fail, sometimes after hours of apparently normal training, and a fault on one rank takes down a whole multi-node job. Drain the node first, from the login node: ```bash scontrol update nodename=slinky-3 state=drain reason="XID errors" ``` Running jobs finish, and Slurm schedules new jobs on other nodes. Put the node back after the reboot: ```bash scontrol update nodename=slinky-3 state=resume ``` ## Reboot one node **Reboot Node** is on a node's drawer and in the XID dialog, rather than on the cluster's **Actions** menu. Prefer it to **Restart All Workers**, which ends every running job on the cluster: it deals with one bad node out of eight and leaves the jobs on the other seven running. The reboot goes ahead whatever is in the Slurm queue, so jobs on that node end. Drain the node and let its jobs finish first, unless the node is already broken enough that the jobs are lost anyway. 1. Open the node with **View GPUs**, or open the XID dialog from the banner, which lists each fault before it offers the reboot. 2. Select **Reboot Node**. While the node comes back, its **Health** is **Rebooting**. After 5 minutes the node reports its current state again, whatever it is. TIR tries the reboot once, and that state is the only sign of how it went, so check it then. You can start another reboot once the current one has finished. If the node stays down, see [Troubleshooting TIR Slurm Cluster](/tir/tir-slurm-cluster/troubleshooting/tir-slurm-cluster/). ## Use the command line The same checks work from the login node: ```bash ssh root@ ``` Replace `` with the cluster's address from **Connection Details**. ```bash sinfo -N -o "%N %P %t %C %G %E" scontrol show node slinky-0 ``` The first command prints the node, partition, state, CPUs, GPUs, and the reason a node was drained. That reason is often the most direct answer to "why is my job pending?" when capacity looks free. A reason you recognize from your own prolog script means the prolog drained the node; see [Add prolog and epilog scripts](/tir/tir-slurm-cluster/how-to/add-prolog-and-epilog-scripts/). To read GPU state from a job: ```bash srun --nodelist=slinky-3 --gres=gpu:8 nvidia-smi srun --nodelist=slinky-3 nvidia-smi -q | grep -i -A2 xid ``` To see activity over time rather than a snapshot, continue to [Monitor a cluster](/tir/tir-slurm-cluster/how-to/monitor-a-cluster/). For a node that stays **Unknown**, see [Troubleshooting TIR Slurm Cluster](/tir/tir-slurm-cluster/troubleshooting/tir-slurm-cluster/). The [NVIDIA XID error reference](https://docs.nvidia.com/deploy/xid-errors/index.html) explains what each code means.