--- title: Monitoring and Alerts sidebar_label: Monitoring --- import { Activity, Clock, BarChart2, Bell, Cpu } from 'react-feather'; # Monitoring and Alerts The **Monitoring** tab charts GPU and scheduler metrics over time, so you can see whether the cluster is actually doing work rather than just holding GPUs. The **Alerts** tab attaches alerts to it. }, { href: '#summary-cards', label: 'Summary cards', icon: }, { href: '#gpu-metrics', label: 'GPU metrics', icon: }, { href: '#scheduler-metrics', label: 'Scheduler metrics', icon: }, { href: '#reading-the-charts', label: 'Reading the charts', icon: }, { href: '#alerts', label: 'Alerts', icon: }, ]} /> --- ## Controls | Control | What it does | |---------|--------------| | **Select Node** | Choose which nodes to chart. Multi-select; at least one node is always selected | | Time range | **5m**, **15m**, **1h**, **6h**, **1d**. Default **5m** | | Refresh | Reload everything on the tab | Nodes are listed as `e2e_node_ (slinky-N)` with a status dot, so you can pick a specific machine. :::note The tab does not auto-refresh Data reloads when you change the node selection or the time range, or press refresh. For a live view while a job starts, pick **5m** and press refresh; for a whole run, use **6h** or **1d**. ::: :::tip Timestamps are IST Chart X-axis labels are rendered in India Standard Time regardless of your machine's timezone. Account for that when correlating a chart with a job's own log timestamps. ::: --- ## Summary Cards | Card | Shows | Subtitle | |------|-------|----------| | **RUNNING JOBS** | Jobs executing now | `active` | | **PENDING JOBS** | Jobs queued | `in queue` | | **COMPLETED** | Jobs finished | `finished` | | **CPU ALLOC** | Latest CPU allocation reported by the scheduler | `latest` | :::note CPU ALLOC is a count, shown with a percent sign The value is the number of CPUs the scheduler has allocated, but it is rendered with a `%` suffix and the matching chart is scaled to 0–100. On a cluster with more than 100 allocated CPUs the chart sits at the top of its axis. Read it as "allocated CPUs", and use `squeue -o "%C"` or `sinfo -o "%C"` when you need the real figure. ::: --- ## GPU Metrics Section: **GPU Metrics — All Nodes Overlay**. One series per selected node, so you can compare nodes directly. | Chart | What it plots | |-------|---------------| | **GPU Utilisation %** | Average GPU utilisation on each node | | **GPU Memory Utilisation %** | GPU framebuffer memory in use on each node, **in GB** | | **GPU Temperature °C** | Average GPU temperature per node | | **Power Draw W** | Average GPU power draw per node | The selected time range appears in each chart title, e.g. `GPU Temperature °C (15m)`. :::warning The memory chart is in gigabytes, not percent Despite the `%` in the title and axis, **GPU Memory Utilisation** plots framebuffer memory *used* in GB. A value of `40` means 40 GB, not 40%. Compare it against your GPU's capacity — 80 GB on an H100, 141 GB on an H200 — which the [Nodes tab](/docs/tir/SlurmCluster/manage/nodes) shows per node. ::: Each chart carries a clickable legend showing every node and its latest value; click a node to hide or show its series. Empty ranges read **No data available**. :::note Series colours change between refreshes Colours are assigned per render, so the same node is not a stable colour across refreshes or ranges, and colours do not match between the four charts. Identify a series from the legend, not its colour. ::: :::info No GPU charts on a CPU-only cluster A CPU cluster has no DCGM metrics, so all four GPU charts read **No data available**. The scheduler charts still work. ::: --- ## Scheduler Metrics Section: **Scheduler Metrics**. Cluster-wide, not per node. | Chart | What it plots | |-------|---------------| | **Job Count — running vs pending** | Two series: `running` (green) and `pending` (orange) | | **CPU Allocation (jobs)** | CPUs allocated to jobs over time | :::tip The job-count chart is your queue-pressure gauge A persistent gap between `pending` and `running` means demand exceeds capacity — the case for [scaling up](/docs/tir/SlurmCluster/manage/actions#scale-cluster) or adding a [short-jobs partition](/docs/tir/SlurmCluster/slurm-configuration/partitions). `pending` at zero with low utilisation means the cluster is oversized for the workload. ::: --- ## Reading the Charts ### Is the cluster earning its cost? | Pattern | What it means | |---------|---------------| | GPU Utilisation high and steady | Healthy training | | Utilisation sawtoothing between high and near-zero | Data pipeline bottleneck — the GPUs are waiting on I/O. Check your dataloader workers and whether your dataset is on PFS | | Utilisation near zero with jobs running | The work is not on the GPU. Often a model left on CPU, or a job stuck in initialisation | | Utilisation zero, no jobs running | An idle cluster you are paying for | | One node much lower than the others | A straggler rank. Check the [per-GPU drilldown](/docs/tir/SlurmCluster/manage/nodes#per-gpu-drilldown) | | Temperature rising while utilisation is flat | Cooling problem. Check the node for [XID errors](/docs/tir/SlurmCluster/manage/nodes#xid-errors) | | Power far below the GPU's rating during training | The GPUs are not under real load, whatever the utilisation number says | :::tip Check utilisation on the first run of any new job script The most expensive failure mode on a GPU cluster is not a crash — it is a job that runs to completion at 15% utilisation for three days. Watch **GPU Utilisation** for the first few minutes of a new script; it is the cheapest optimisation available. ::: ### Correlating a problem 1. **Monitoring** — when did it change, and on which nodes? 2. **[Jobs](/docs/tir/SlurmCluster/manage/jobs)** — what was running then? 3. **[Nodes](/docs/tir/SlurmCluster/manage/nodes)** — is that node reporting XID errors? 4. **[Logs](/docs/tir/SlurmCluster/manage/logs)** — what did the controller or worker say? --- ## Alerts The **Alerts** tab attaches existing alerts to this cluster. > Choose alerts attached to this Cluster. **Click here** to create a new alert. 1. Create the alert first, under **Alert Management** in the TIR sidebar (the **Click here** link goes there). 2. Return to the cluster's **Alerts** tab. 3. Pick it from **Select Alerts** and click **Attach**. The table lists what is attached: | Column | Shows | |--------|-------| | **Name** | The alert name | | **Severity** | Its configured severity | | **Threshold** | The value that triggers it | | **Metric Type** | What it watches | | **Operator** | The comparison, e.g. above or below | | **Status** | Whether it is active | | **Action** | Detach it | Empty: **No alerts found.** :::tip Alerts turn monitoring into something you do not have to watch The tabs on this page are pull, not push. An alert on low GPU utilisation catches the expensive failure above; one on high temperature catches a cooling problem before it becomes an XID error. Attach both to any long-lived cluster. ::: --- ## Related Resources - [Nodes and GPU health](/docs/tir/SlurmCluster/manage/nodes) - [Jobs tab](/docs/tir/SlurmCluster/manage/jobs) - [Cluster Overview tab](/docs/tir/SlurmCluster/manage/overview-tab) - [Logs](/docs/tir/SlurmCluster/manage/logs) - [Alert Management](/docs/tir/alert_management/)