Cluster Overview Tab
One screen answering "is the cluster healthy, and is it busy?" — summary cards, per-node health, and the Slurm partition table.
The tab requires a Running cluster. Otherwise it reads "Cluster overview is available when the cluster is running."
Summary Cards
| Card | What it shows | Subtitle |
|---|---|---|
| TOTAL GPUS | Every GPU across the cluster | <n> nodes × <gpus per node> |
| ALLOCATED GPUS | How much of the cluster is in use | <n> nodes active |
| GPU UTIL | Average GPU utilisation | avg across active nodes |
| GPU MEMORY | Average GPU memory in use | avg used / <n> GB — the per-GPU capacity |
| RUNNING JOBS | Jobs executing now | <n> pending in queue |
| IDLE NODES | Nodes with nothing allocated | available for workloads |
Despite the label, this card counts nodes that are allocated or partly allocated, not individual
GPUs. Compare it against TOTAL GPUS' node count rather than its GPU count. For exact per-GPU
allocation, use scontrol show node or the Nodes tab.
IDLE NODES and GPU UTIL together tell you whether the cluster is under-used. Idle nodes with jobs pending means a scheduling constraint — the queue wants something the free nodes cannot give. Zero idle nodes with low GPU utilisation means jobs are holding GPUs without using them, which is usually a dataloader bottleneck.
Node Health
One card per node, with a coloured dot for its Slurm state. At most four are shown; a Show More (N more nodes) link jumps to the Nodes tab.
States
| State | Dot | Meaning |
|---|---|---|
| IDLE | Green | Up, nothing allocated. Ready for work |
| ALLOCATED | Blue | Every allocatable resource is allocated to jobs |
| MIXED | Yellow | Partly allocated — some resources free |
| DOWN | Red | Slurm has marked the node unavailable |
| UNKNOWN | Grey | Slurm is not reporting this node — usually the worker has not registered yet, or the controller cannot reach it |
The legend above the cards lists IDLE, ALLOCATED, MIXED and UNKNOWN. A DOWN node shows a red dot
even though the legend does not include it, and other Slurm states (DRAIN, MAINT, RESERVED,
REBOOT_REQUESTED, …) render grey.
A worker shows UNKNOWN for a short while after a restart, a scale-up or an image update while it registers with the controller. If it is still UNKNOWN after a few minutes, that node is not joining — check the Logs tab for that worker, and consider Restart All Workers.
Card contents
| Row | Shows |
|---|---|
| Title | The node, as e2e_node_<id> (slinky-N) — the physical machine and the Slurm node name |
| GPU Util | Average across the node's GPUs, with a bar |
| VRAM | Average per-GPU memory used, over per-GPU capacity |
| Temp / Power | Average temperature and power draw per GPU, colour-coded |
Temperature turns orange at 75 °C and red at 85 °C. Power turns orange at 300 W and red at 400 W.
Sustained red on one node while its neighbours are green usually means a cooling or seating problem, not a workload difference. Note the node name and check the Nodes tab for XID errors before opening a support case.
Slurm Partition Info
Partitions you define are written to
slurm.conf. The platform partition covering every node is managed for you.
| Column | Shows |
|---|---|
| PARTITION | Name. * marks the default. not live means it is in your configuration but the controller is not reporting it |
| NODES | Node count |
| GPU TYPE | Not currently populated — see the note below |
| AVAIL / TOTAL | Not currently populated — see the note below |
| DEFAULT TIME | Time limit applied when a job omits --time |
| MAX TIME | Longest a job may request. ∞ means UNLIMITED |
| STATE | UP, DOWN, DRAIN, INACTIVE — or DISABLED for one of yours that is switched off |
| ACTIONS | ⋮ with Edit and Delete for your partitions. Platform partitions read Managed |
An Add Partition button sits at the top right. Full documentation: Manage partitions.
The footer records the source — Sourced from slurmrestd · GET /slurm/<version>/partitions — so the
table reflects the live controller.
These two columns are not fed by the partition data the console receives: GPU TYPE renders blank and AVAIL / TOTAL shows a placeholder rather than real availability. Do not rely on them.
For authoritative per-partition availability, use the login node:
sinfo -o "%20P %5a %10l %6D %6t %N"
The Nodes tab shows real per-node GPU models and health.
The GPU Health Banner
A red banner appears at the top when any GPU reports an XID error — an NVIDIA hardware or driver fault:
GPU / Node Health Issues Detected
<n> node(s) DOWN · <n> node(s) with XID errors · <n> total GPU fault events
Below it, one row per fault with the XID code, the node, the GPU index, and the error text.
A cluster with DOWN or UNKNOWN nodes and no XID errors shows no banner. Node state is on the node cards above, not here — do not treat a missing banner as "all nodes healthy".
XID errors mean the GPU or its driver hit a fault. Some are transient; some indicate failing hardware. Investigate on the Nodes tab, where a single affected node can be rebooted.
Reading the Tab Quickly
A 20-second health check before submitting a large job:
- Red banner? → XID errors. Go to the Nodes tab before you submit anything expensive.
- Node cards → any grey (UNKNOWN) or red (DOWN) dots? Those nodes are not usable.
- IDLE NODES → is there room for what you are about to submit?
- Partition table → is the partition you target
UP, and does its MAX TIME allow your--time? - GPU UTIL → if it is high, expect to queue.
Nothing on it auto-refreshes. Use the refresh icon in the header, and for trends over time use the Monitoring tab, which has selectable time ranges.