--- title: "Monitor a cluster" description: "Chart GPU and scheduler metrics over time for a TIR Slurm Cluster, read the patterns, and attach alerts that warn you while you are away from the charts." content_type: "how-to" product: "tir-slurm-cluster" platform: "TIR" url: "/tir/tir-slurm-cluster/how-to/monitor-a-cluster/" topic: ["monitor"] --- # Monitor a cluster See whether your TIR Slurm Cluster is doing work or only holding GPUs, with charts over time on the **Monitoring** tab, and attach alerts that tell you when something needs attention. ## Before you begin You need a cluster that is **Running**. See [Cluster statuses and actions](/tir/tir-slurm-cluster/concepts/cluster-statuses-and-actions/) for which tabs open in each status. To attach an alert, create the alert first under **Alert Management** in the TIR sidebar. ## Open the Monitoring tab Under **Training Cluster** in the TIR sidebar, open the cluster, then open **Monitoring**. Choose the nodes to chart under **Select Node**, at least one, and a time range: **5m**, the default, **15m**, **1h**, **6h**, or **1d**. The time labels on the charts are in India Standard Time, whatever the time zone of your computer. Allow for it when you compare a chart with the timestamps in a job's own log. The tab reloads when you change the node selection or the time range, or select refresh. For a live view while a job starts, choose **5m** and refresh. For a whole run, choose **6h** or **1d**. Four totals sit at the top: **RUNNING JOBS**, **PENDING JOBS**, **COMPLETED**, and **CPU ALLOC**, which counts the CPUs the scheduler has allocated. The small words under each total are fixed labels rather than counts. ## Read the GPU metrics The **GPU Metrics** section overlays one series per selected node, so you can compare nodes. It has four charts: **GPU Utilisation %**, **GPU Memory Utilisation %**, **GPU Temperature °C**, and **Power Draw W**. An empty chart reads **No data available**. ## Read the scheduler metrics The **Scheduler Metrics** section covers the whole cluster rather than each node. The **Job Count** chart compares running with pending jobs, and **CPU Allocation (jobs)** shows the CPUs allocated to jobs. A gap that stays between pending and running means demand exceeds capacity, which calls for more nodes or a short-jobs partition; see [Scale a cluster](/tir/tir-slurm-cluster/how-to/scale-a-cluster/) and [Manage partitions](/tir/tir-slurm-cluster/how-to/manage-partitions/). Pending at zero with low utilization means the cluster is larger than your jobs need. ## Read the patterns Check **GPU Utilisation** on the first run of any new job script. It is the cheapest optimization available on a GPU cluster: a job that completes at 15% utilization costs the same as one at 95% and takes far longer. | Pattern | What it usually means | | -------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | | Utilization high and steady | The GPUs are the bottleneck, which is the healthy case. | | Utilization sawtoothing between high and near zero | The data pipeline is the bottleneck. Add data-loading workers, prefetch, and keep the data on a PFS. | | Utilization near zero with jobs running | The GPU sits idle, for example because the model was left on the CPU, or a job is stuck starting up. | | Utilization zero with the queue empty | An idle cluster you are paying for. | | Power far below the GPU's rating during training | The GPUs are lightly loaded, whatever the utilization number says. | | One node much lower than the others | A straggler rank. Open the per-GPU view for that node on the **Nodes** tab. | | Temperature rising while utilization is flat | A cooling problem. Check the node for XID errors. | | Multi-node run much slower than a single node | Collective traffic may have fallen back to TCP. See [Troubleshooting jobs on TIR Slurm Cluster](/tir/tir-slurm-cluster/troubleshooting/jobs/). | To trace a problem across tabs, start on **Monitoring** to see when it changed and on which nodes. Then check **Jobs** for what was running, **Nodes** for XID errors on that node, and **Logs** for what the worker said. See [Check cluster and node health](/tir/tir-slurm-cluster/how-to/check-cluster-and-node-health/) and [See jobs and logs](/tir/tir-slurm-cluster/how-to/see-jobs-and-logs/). ## Attach alerts The **Alerts** tab attaches existing alerts to the cluster. The charts help when you look at them, and an alert tells you while you are away. An alert on low GPU utilization catches a job that runs for days at 15%, and an alert on high temperature catches a cooling problem early. Attach both to any long-lived cluster. 1. Create the alert under **Alert Management** in the TIR sidebar. The **Click here** link on the **Alerts** tab goes there. 2. Return to the cluster's **Alerts** tab. 3. Choose the alert in **Select Alerts** and select **Attach**. The attached alerts are listed with their severity, threshold, and status, and you can detach each one. For the node-level view of the same GPUs, see [Check cluster and node health](/tir/tir-slurm-cluster/how-to/check-cluster-and-node-health/).