Monitor a cluster

See whether your TIR Slurm Cluster is doing work or only holding GPUs, with charts over time on the Monitoring tab, and attach alerts that tell you when something needs attention.

Before you begin

You need a cluster that is Running. See Cluster statuses and actions for which tabs open in each status. To attach an alert, create the alert first under Alert Management in the TIR sidebar.

Open the Monitoring tab

Under Training Cluster in the TIR sidebar, open the cluster, then open Monitoring. Choose the nodes to chart under Select Node, at least one, and a time range: 5m, the default, 15m, 1h, 6h, or 1d.

The time labels on the charts are in India Standard Time, whatever the time zone of your computer. Allow for it when you compare a chart with the timestamps in a job’s own log.

The tab reloads when you change the node selection or the time range, or select refresh. For a live view while a job starts, choose 5m and refresh. For a whole run, choose 6h or 1d.

Four totals sit at the top: RUNNING JOBS, PENDING JOBS, COMPLETED, and CPU ALLOC, which counts the CPUs the scheduler has allocated. The small words under each total are fixed labels rather than counts.

Read the GPU metrics

The GPU Metrics section overlays one series per selected node, so you can compare nodes. It has four charts: GPU Utilisation %, GPU Memory Utilisation %, GPU Temperature °C, and Power Draw W. An empty chart reads No data available.

Read the scheduler metrics

The Scheduler Metrics section covers the whole cluster rather than each node. The Job Count chart compares running with pending jobs, and CPU Allocation (jobs) shows the CPUs allocated to jobs. A gap that stays between pending and running means demand exceeds capacity, which calls for more nodes or a short-jobs partition; see Scale a cluster and Manage partitions. Pending at zero with low utilization means the cluster is larger than your jobs need.

Read the patterns

Check GPU Utilisation on the first run of any new job script. It is the cheapest optimization available on a GPU cluster: a job that completes at 15% utilization costs the same as one at 95% and takes far longer.

Pattern	What it usually means
Utilization high and steady	The GPUs are the bottleneck, which is the healthy case.
Utilization sawtoothing between high and near zero	The data pipeline is the bottleneck. Add data-loading workers, prefetch, and keep the data on a PFS.
Utilization near zero with jobs running	The GPU sits idle, for example because the model was left on the CPU, or a job is stuck starting up.
Utilization zero with the queue empty	An idle cluster you are paying for.
Power far below the GPU’s rating during training	The GPUs are lightly loaded, whatever the utilization number says.
One node much lower than the others	A straggler rank. Open the per-GPU view for that node on the Nodes tab.
Temperature rising while utilization is flat	A cooling problem. Check the node for XID errors.
Multi-node run much slower than a single node	Collective traffic may have fallen back to TCP. See Troubleshooting jobs on TIR Slurm Cluster.

To trace a problem across tabs, start on Monitoring to see when it changed and on which nodes. Then check Jobs for what was running, Nodes for XID errors on that node, and Logs for what the worker said. See Check cluster and node health and See jobs and logs.

Attach alerts

The Alerts tab attaches existing alerts to the cluster. The charts help when you look at them, and an alert tells you while you are away. An alert on low GPU utilization catches a job that runs for days at 15%, and an alert on high temperature catches a cooling problem early. Attach both to any long-lived cluster.

1. Create the alert under Alert Management in the TIR sidebar. The Click here link on the Alerts tab goes there.
2. Return to the cluster’s Alerts tab.
3. Choose the alert in Select Alerts and select Attach.

The attached alerts are listed with their severity, threshold, and status, and you can detach each one.

For the node-level view of the same GPUs, see Check cluster and node health.