Monitoring and Alerts
The Monitoring tab charts GPU and scheduler metrics over time, so you can see whether the cluster is actually doing work rather than just holding GPUs. The Alerts tab attaches alerts to it.
Controls
| Control | What it does |
|---|---|
| Select Node | Choose which nodes to chart. Multi-select; at least one node is always selected |
| Time range | 5m, 15m, 1h, 6h, 1d. Default 5m |
| Refresh | Reload everything on the tab |
Nodes are listed as e2e_node_<id> (slinky-N) with a status dot, so you can pick a specific machine.
Data reloads when you change the node selection or the time range, or press refresh. For a live view while a job starts, pick 5m and press refresh; for a whole run, use 6h or 1d.
Chart X-axis labels are rendered in India Standard Time regardless of your machine's timezone. Account for that when correlating a chart with a job's own log timestamps.
Summary Cards
| Card | Shows | Subtitle |
|---|---|---|
| RUNNING JOBS | Jobs executing now | active |
| PENDING JOBS | Jobs queued | in queue |
| COMPLETED | Jobs finished | finished |
| CPU ALLOC | Latest CPU allocation reported by the scheduler | latest |
The value is the number of CPUs the scheduler has allocated, but it is rendered with a % suffix and
the matching chart is scaled to 0–100. On a cluster with more than 100 allocated CPUs the chart sits
at the top of its axis. Read it as "allocated CPUs", and use squeue -o "%C" or sinfo -o "%C" when
you need the real figure.
GPU Metrics
Section: GPU Metrics — All Nodes Overlay. One series per selected node, so you can compare nodes directly.
| Chart | What it plots |
|---|---|
| GPU Utilisation % | Average GPU utilisation on each node |
| GPU Memory Utilisation % | GPU framebuffer memory in use on each node, in GB |
| GPU Temperature °C | Average GPU temperature per node |
| Power Draw W | Average GPU power draw per node |
The selected time range appears in each chart title, e.g. GPU Temperature °C (15m).
Despite the % in the title and axis, GPU Memory Utilisation plots framebuffer memory used in
GB. A value of 40 means 40 GB, not 40%. Compare it against your GPU's capacity — 80 GB on an H100,
141 GB on an H200 — which the Nodes tab shows per node.
Each chart carries a clickable legend showing every node and its latest value; click a node to hide or show its series. Empty ranges read No data available.
Colours are assigned per render, so the same node is not a stable colour across refreshes or ranges, and colours do not match between the four charts. Identify a series from the legend, not its colour.
A CPU cluster has no DCGM metrics, so all four GPU charts read No data available. The scheduler charts still work.
Scheduler Metrics
Section: Scheduler Metrics. Cluster-wide, not per node.
| Chart | What it plots |
|---|---|
| Job Count — running vs pending | Two series: running (green) and pending (orange) |
| CPU Allocation (jobs) | CPUs allocated to jobs over time |
A persistent gap between pending and running means demand exceeds capacity — the case for
scaling up or adding a
short-jobs partition. pending at zero with
low utilisation means the cluster is oversized for the workload.
Reading the Charts
Is the cluster earning its cost?
| Pattern | What it means |
|---|---|
| GPU Utilisation high and steady | Healthy training |
| Utilisation sawtoothing between high and near-zero | Data pipeline bottleneck — the GPUs are waiting on I/O. Check your dataloader workers and whether your dataset is on PFS |
| Utilisation near zero with jobs running | The work is not on the GPU. Often a model left on CPU, or a job stuck in initialisation |
| Utilisation zero, no jobs running | An idle cluster you are paying for |
| One node much lower than the others | A straggler rank. Check the per-GPU drilldown |
| Temperature rising while utilisation is flat | Cooling problem. Check the node for XID errors |
| Power far below the GPU's rating during training | The GPUs are not under real load, whatever the utilisation number says |
The most expensive failure mode on a GPU cluster is not a crash — it is a job that runs to completion at 15% utilisation for three days. Watch GPU Utilisation for the first few minutes of a new script; it is the cheapest optimisation available.
Correlating a problem
- Monitoring — when did it change, and on which nodes?
- Jobs — what was running then?
- Nodes — is that node reporting XID errors?
- Logs — what did the controller or worker say?
Alerts
The Alerts tab attaches existing alerts to this cluster.
Choose alerts attached to this Cluster. Click here to create a new alert.
- Create the alert first, under Alert Management in the TIR sidebar (the Click here link goes there).
- Return to the cluster's Alerts tab.
- Pick it from Select Alerts and click Attach.
The table lists what is attached:
| Column | Shows |
|---|---|
| Name | The alert name |
| Severity | Its configured severity |
| Threshold | The value that triggers it |
| Metric Type | What it watches |
| Operator | The comparison, e.g. above or below |
| Status | Whether it is active |
| Action | Detach it |
Empty: No alerts found.
The tabs on this page are pull, not push. An alert on low GPU utilisation catches the expensive failure above; one on high temperature catches a cooling problem before it becomes an XID error. Attach both to any long-lived cluster.