--- title: Cluster Overview Tab sidebar_label: Cluster Overview --- import { Grid, Cpu, Heart, Layers, AlertTriangle } from 'react-feather'; # Cluster Overview Tab One screen answering "is the cluster healthy, and is it busy?" — summary cards, per-node health, and the Slurm partition table. }, { href: '#node-health', label: 'Node health', icon: }, { href: '#slurm-partition-info', label: 'Partition table', icon: }, { href: '#the-gpu-health-banner', label: 'GPU health banner', icon: }, { href: '#reading-the-tab-quickly', label: 'Reading it quickly', icon: }, ]} /> The tab requires a **Running** cluster. Otherwise it reads *"Cluster overview is available when the cluster is running."* --- ## Summary Cards | Card | What it shows | Subtitle | |------|---------------|----------| | **TOTAL GPUS** | Every GPU across the cluster | ` nodes × ` | | **ALLOCATED GPUS** | How much of the cluster is in use | ` nodes active` | | **GPU UTIL** | Average GPU utilisation | `avg across active nodes` | | **GPU MEMORY** | Average GPU memory in use | `avg used / GB` — the per-GPU capacity | | **RUNNING JOBS** | Jobs executing now | ` pending in queue` | | **IDLE NODES** | Nodes with nothing allocated | `available for workloads` | :::note "Allocated GPUs" counts busy nodes Despite the label, this card counts **nodes** that are allocated or partly allocated, not individual GPUs. Compare it against **TOTAL GPUS**' node count rather than its GPU count. For exact per-GPU allocation, use `scontrol show node` or the [Nodes tab](/docs/tir/SlurmCluster/manage/nodes). ::: :::tip The two numbers that matter most **IDLE NODES** and **GPU UTIL** together tell you whether the cluster is under-used. Idle nodes with jobs pending means a scheduling constraint — the queue wants something the free nodes cannot give. Zero idle nodes with low GPU utilisation means jobs are holding GPUs without using them, which is usually a dataloader bottleneck. ::: --- ## Node Health One card per node, with a coloured dot for its Slurm state. At most four are shown; a **Show More (N more nodes)** link jumps to the [Nodes tab](/docs/tir/SlurmCluster/manage/nodes). ### States | State | Dot | Meaning | |-------|-----|---------| | **IDLE** | Green | Up, nothing allocated. Ready for work | | **ALLOCATED** | Blue | Every allocatable resource is allocated to jobs | | **MIXED** | Yellow | Partly allocated — some resources free | | **DOWN** | Red | Slurm has marked the node unavailable | | **UNKNOWN** | Grey | Slurm is not reporting this node — usually the worker has not registered yet, or the controller cannot reach it | The legend above the cards lists IDLE, ALLOCATED, MIXED and UNKNOWN. A **DOWN** node shows a red dot even though the legend does not include it, and other Slurm states (`DRAIN`, `MAINT`, `RESERVED`, `REBOOT_REQUESTED`, …) render grey. :::warning UNKNOWN right after a change is normal; UNKNOWN that persists is not A worker shows **UNKNOWN** for a short while after a restart, a scale-up or an image update while it registers with the controller. If it is still UNKNOWN after a few minutes, that node is not joining — check the [Logs tab](/docs/tir/SlurmCluster/manage/logs) for that worker, and consider [Restart All Workers](/docs/tir/SlurmCluster/manage/actions#restart-actions). ::: ### Card contents | Row | Shows | |-----|-------| | Title | The node, as `e2e_node_ (slinky-N)` — the physical machine and the Slurm node name | | **GPU Util** | Average across the node's GPUs, with a bar | | **VRAM** | Average per-GPU memory used, over per-GPU capacity | | **Temp / Power** | Average temperature and power draw per GPU, colour-coded | Temperature turns orange at 75 °C and red at 85 °C. Power turns orange at 300 W and red at 400 W. :::tip Temperature colour is a real signal Sustained red on one node while its neighbours are green usually means a cooling or seating problem, not a workload difference. Note the node name and check the [Nodes tab](/docs/tir/SlurmCluster/manage/nodes) for XID errors before opening a support case. ::: --- ## Slurm Partition Info > Partitions you define are written to `slurm.conf`. The platform partition covering every node is > managed for you. | Column | Shows | |--------|-------| | **PARTITION** | Name. `*` marks the default. `not live` means it is in your configuration but the controller is not reporting it | | **NODES** | Node count | | **GPU TYPE** | Not currently populated — see the note below | | **AVAIL / TOTAL** | Not currently populated — see the note below | | **DEFAULT TIME** | Time limit applied when a job omits `--time` | | **MAX TIME** | Longest a job may request. `∞` means `UNLIMITED` | | **STATE** | `UP`, `DOWN`, `DRAIN`, `INACTIVE` — or `DISABLED` for one of yours that is switched off | | **ACTIONS** | ⋮ with **Edit** and **Delete** for your partitions. Platform partitions read `Managed` | An **Add Partition** button sits at the top right. Full documentation: [Manage partitions](/docs/tir/SlurmCluster/slurm-configuration/partitions). The footer records the source — `Sourced from slurmrestd · GET /slurm//partitions` — so the table reflects the live controller. :::warning GPU TYPE and AVAIL / TOTAL are not populated These two columns are not fed by the partition data the console receives: **GPU TYPE** renders blank and **AVAIL / TOTAL** shows a placeholder rather than real availability. Do not rely on them. For authoritative per-partition availability, use the login node: ```bash sinfo -o "%20P %5a %10l %6D %6t %N" ``` The [Nodes tab](/docs/tir/SlurmCluster/manage/nodes) shows real per-node GPU models and health. ::: --- ## The GPU Health Banner A red banner appears at the top when any GPU reports an **XID error** — an NVIDIA hardware or driver fault: > **GPU / Node Health Issues Detected** > ` node(s) DOWN · node(s) with XID errors · total GPU fault events` Below it, one row per fault with the XID code, the node, the GPU index, and the error text. :::warning The banner only appears for XID errors A cluster with DOWN or UNKNOWN nodes and no XID errors shows **no banner**. Node state is on the node cards above, not here — do not treat a missing banner as "all nodes healthy". ::: XID errors mean the GPU or its driver hit a fault. Some are transient; some indicate failing hardware. Investigate on the [Nodes tab](/docs/tir/SlurmCluster/manage/nodes#xid-errors), where a single affected node can be rebooted. --- ## Reading the Tab Quickly A 20-second health check before submitting a large job: 1. **Red banner?** → XID errors. Go to the Nodes tab before you submit anything expensive. 2. **Node cards** → any grey (UNKNOWN) or red (DOWN) dots? Those nodes are not usable. 3. **IDLE NODES** → is there room for what you are about to submit? 4. **Partition table** → is the partition you target `UP`, and does its **MAX TIME** allow your `--time`? 5. **GPU UTIL** → if it is high, expect to queue. :::tip This tab is a snapshot, not a stream Nothing on it auto-refreshes. Use the refresh icon in the header, and for trends over time use the [Monitoring tab](/docs/tir/SlurmCluster/manage/monitoring), which has selectable time ranges. ::: --- ## Related Resources - [Nodes and GPU health](/docs/tir/SlurmCluster/manage/nodes) - [Jobs tab](/docs/tir/SlurmCluster/manage/jobs) - [Monitoring](/docs/tir/SlurmCluster/manage/monitoring) - [Manage partitions](/docs/tir/SlurmCluster/slurm-configuration/partitions)