Manage a Slurm Cluster
Everything after creation happens on the cluster details page: ten tabs and an Actions menu. This is the map.
Open a Cluster
- Log in to the TIR console and select your project.
- Click Training Cluster in the sidebar. The list page is headed Manage Training Clusters.
- Click the cluster name.
The list shows Cluster Name, Created By, Plan, No of Nodes, Created At, a Connect
shortcut, Status, and an Actions (⋮) menu per row. It refreshes itself every few seconds while
any cluster is Creating or Deleting.
The Page Header
| Control | What it does |
|---|---|
| Select Training Cluster | Switch to another cluster without going back to the list |
| Status pill | Green for Running, red for Failed / Terminated, amber for transitional states |
| Refresh icon | Reload the current tab. The details page does not auto-refresh |
| Connect | Opens the SSH panel with copyable commands |
| Restart Workers | Shortcut for Restart All Workers. Disabled unless Running |
| Actions (⋮) | Every cluster action |
The Tabs
| Tab | What it is for | Page |
|---|---|---|
| Details | Name, image version, status, node count, plan, and how to connect | below |
| Cluster Overview | GPU and job summary, node health, and the Slurm partition table | Cluster Overview |
| Nodes | Per-node and per-GPU DCGM metrics, health filters, XID errors, node reboot | Nodes |
| Jobs | Live squeue with state counters and filters | Jobs |
| Monitoring | GPU and scheduler charts over selectable time ranges | Monitoring |
| Alerts | Attach alerts to the cluster | Monitoring |
| Logs | Controller, worker and login-node logs | Logs |
| Volumes | Mount and unmount PFS, SFS and datasets | Storage |
| Network & Security | Public IP, VPC IP, and security groups | Network and security |
| Login User Management | Per-user SSH identities and groups | Login User Management |
On a cluster that is not Running, tabs such as Cluster Overview, Nodes and Login User Management show a message like "Cluster overview is available when the cluster is running." The Details tab always works, which is where you check status and plan on a Failed or Terminated cluster. Tab availability also varies slightly by region.
Details Tab
Three collapsible sections.
Cluster Details
| Field | Meaning |
|---|---|
| Cluster Name | The name you gave it |
| Image Version | The cluster's Slurm image, e.g. v2-slurm25.11-ubuntu24.04. Change it with Update Image |
| Status | Current state |
| Number of Nodes | Worker nodes. Change it with Scale Cluster |
| Created By | Who created it |
| Created At | When |
Plan Details
| Field | Meaning |
|---|---|
| Plan Name | The plan or Private Cluster |
| CPU / Memory | Per-node resources |
| Plan Type | On-demand, committed, or private cluster |
| Private Cluster Name | The reserved cluster it draws from, where applicable |
Connection Details
| Field | Meaning |
|---|---|
| SSH Keys | The keys with root access, as chips. Change with Update SSH Keys |
| SSH Command | ssh root@<ip>, with a copy button |
v2-slurm25.11-ubuntu24.04 means Slurm 25.11 on Ubuntu 24.04. Worth knowing when you read the
SchedMD documentation — directives and flags do change between Slurm
releases.
Common Tasks
| I want to… | Go to |
|---|---|
| See why my job is not running | Jobs tab |
| Check whether the GPUs are actually busy | Monitoring |
| Find a broken GPU | Nodes |
| Add more nodes | Scale Cluster |
| Add a partition | Manage partitions |
| Change a scheduler setting | Extra slurm.conf |
| Run setup before every job | Prolog and epilog scripts |
| Attach more storage | Storage and volumes |
| Give a colleague access | Login User Management |
| Pin the cluster's IP | Network and security |
| Stop paying for it | Terminate Cluster |
| Work out why a control is missing | Action availability |