Cluster Actions
Every lifecycle operation on a Slurm Cluster runs from the Actions menu. This page documents each one — what it does, what it asks you, what it restarts, and when it is available.
Where the Actions Live
| Entry point | How to reach it |
|---|---|
| Cluster list | Training Cluster in the sidebar → the ⋮ icon on a cluster row |
| Cluster details | Open the cluster → Actions button in the top-right area |
| Quick actions | The details header also carries Connect, Restart Workers and a refresh icon as buttons |
Both menus offer the same actions in the destructive-last order below. Actions you do not have update permission for are not shown.
| Action | Available when |
|---|---|
| Update SSH Keys | Any status |
| Update Image | Running |
| Slurm Configuration | Running |
| Scale Cluster | Running, and the cluster is on an hourly plan or a private cluster |
| Convert to Committed | Hourly, non-private-cluster plans, and not Failed / Terminating / Terminated |
| Restart All Workers | Running |
| Restart Cluster | Running |
| Restart Login Service | Running |
| Reconfigure Cluster | Running or Creating |
| Clone Cluster | Any status |
| Terminate Cluster | Any status |
| Delete Training Cluster | Any status |
For the full picture of what hides or blocks an action, see Action availability and blockers.
Update SSH Keys
Replaces the set of SSH keys that can log in to the cluster.
- Actions → Update SSH Keys.
- The dialog reads "Manage SSH keys for cluster <name>".
- Use the Select SSH Keys multi-select to choose the keys. The count shows as
e.g.
2 selected. - To add a brand-new key, use the Click here link — it opens Add SSH Key ("Attach a public key to authenticate with your server") without losing your place.
- Click Update SSH Keys.
| Detail | Value |
|---|---|
| Restart required | No |
| Available at | Any cluster status |
| Confirm blocked when | Nothing selected, or the selection is unchanged |
Keys you leave out of the selection lose access. The dialog pre-selects the cluster's current keys — deselecting one removes it. If nothing is selected you see "No SSH keys selected. You need at least one SSH key to access the cluster." and cannot continue.
Rotating a shared root key affects everyone at once. For team access, use
Login User Management — each person gets
their own key, their own home directory, and can be revoked individually.
Update Image
Moves the cluster to a different version of its Slurm image — the software stack the controller, login node and workers run.
- Actions → Update Image. The dialog is titled Update Image: <name>.
- Current Image shows what is running, e.g.
ubuntu-slurm:v2-slurm25.11-ubuntu24.04. - Under SELECT IMAGE, pick the family — Ubuntu Slurm or NeMo Framework.
- Under SELECT VERSION, pick a version. Rows carry badges: Latest for the newest, Current for the one running now.
- Click Update Image, then read the confirmation.
The confirmation dialog is Confirm Image Update and is explicit about the cost:
Rolling Restart Required
- Nodes will restart one at a time to apply the new image
- Running jobs will be drained before each node restarts
- Pending jobs will queue until nodes come back online
- Estimated downtime per node: 3-8 minutes
Confirm with Apply & Rolling Restart. Success toast: "Image update applied successfully. Rolling restart initiated."
Update Image changes the cluster's own software — Slurm, the drivers, the base OS. It has
nothing to do with the container your jobs run in. To change that, pass
--container-image to srun/sbatch; see
Containers with Enroot and Pyxis. You do not need to update
the cluster image to run a different framework version.
Running jobs are drained node by node. On a cluster with long jobs, an image update can take a long time to work through and your jobs are interrupted. Schedule it, and make sure your jobs checkpoint to shared storage.
Slurm Configuration
Edits slurm.conf extras, partitions, and prolog/epilog scripts. Available only while the cluster is
Running.
Covered in full in Slurm Configuration:
Running jobs are never interrupted; queued jobs pick up the new settings; prolog and epilog changes apply from the next job.
Scale Cluster
Adds or removes worker nodes on a live cluster. The dialog is titled Scale pool: <name> and shows the GPU type and Current nodes before anything else.
Two modes, selected by the Scale up / Scale down buttons.
Scale up
- Actions → Scale Cluster → Scale up.
- Set Total Node Count with the stepper. The value is the new total, not the number to add, and the side text shows the resulting GPU count.
- A cost card shows New hourly cost and Change from current.
- Click Scale Pool.
If capacity is short you get, instead of the stepper:
Inventory Unavailable — No additional inventory is currently available.
What you can do:
- Check back later — inventory is refreshed frequently.
- Contact support for NVIDIA <GPU type> capacity.
When only some of what you asked for is free, the message reads "Only <n> nodes available — you requested <m>." with a link that sets the stepper to the maximum available.
| Detail | Value |
|---|---|
| Maximum nodes | 512 |
| Restart required | No — new nodes join; existing jobs keep running |
| New nodes are placed | On the same rack as your existing nodes, to keep fabric performance consistent |
A partition holds the nodes you named. After scaling up, edit the partition to include the new nodes —
otherwise they are only reachable via the platform all partition. See
Manage partitions.
Scale down
- Actions → Scale Cluster → Scale down.
- Under Select nodes to shut down, click the nodes to remove. A counter shows
N selectedand a cost card shows the saving. - Click Remove N nodes.
| Detail | Value |
|---|---|
| Minimum | At least one node must remain. The last unselected node cannot be chosen, and selecting everything shows "At least 1 node must remain active." |
| Restart | Scale-down also restarts the remaining worker pods. The dialog does not say so |
Nothing checks the Slurm queue before removing a node. Jobs running on a node you remove are lost, and
because the remaining workers restart, jobs on those nodes are affected too. Drain the cluster
first: scontrol update nodename=<node> state=drain, wait for squeue to clear, then scale down.
If every node a partition named is removed, that partition stops being written to slurm.conf and
shows as not live in the partition table. Repoint it at nodes that still exist. See
the scale-down note.
Scale Cluster is hidden on committed plans. The API refuses it with "Scale cluster is not supported for committed cluster".
Restart Actions
Three restart actions with very different blast radius. Each asks a single confirmation.
| Action | Dialog body | What restarts | Use it when |
|---|---|---|---|
| Restart All Workers | "Are you sure you want to restart all workers in cluster <name>?" | Every worker (slurmd) pod | Nodes are unhealthy, reporting XID errors, or wedged after a hardware fault |
| Restart Login Service | "Are you sure you want to restart the login service for cluster <name>?" Confirm button: Restart | The login node pods only | You cannot SSH in, or the login node is out of resources |
| Restart Cluster | "Are you sure you want to restart the entire cluster <name>?" | Controller, login node, all workers and the accounting database | Nothing else has worked. This is the heaviest routine action |
Restart All Workers is also a button in the details header, labelled Restart Workers. It is disabled unless the cluster is Running.
There is no drain step and no queue check. Restart All Workers and Restart Cluster end every running job. Check the Jobs tab first, and prefer draining and letting jobs finish where you can.
Anyone logged in is dropped, and anything running in a plain shell there dies. Work you care about
should be a Slurm job (sbatch), not a foreground process on the login node — a job survives a login
restart, a shell does not.
Try the narrowest action first: Restart Login Service for access problems → Restart All Workers for node or GPU problems → Restart Cluster only if the controller itself is unhealthy. For a single bad node, use the per-node reboot on the Nodes tab instead of restarting everything.
Reconfigure Cluster
Re-applies the cluster's stored configuration without changing anything. It is a repair action.
- Actions → Reconfigure Cluster.
- Confirm: "Are you sure you want to Reconfigure Cluster <name>?"
| Detail | Value |
|---|---|
| Available when | Running or Creating — one of the few actions offered on a Creating cluster |
| Changes | Nothing in your configuration. It re-pushes what is already stored |
A cluster stuck in Creating, or one whose live state has drifted from what the console shows — a volume that reads as mounted but is not there, a setting that did not take. Reconfigure re-applies the intended state. If it does not help, it has not made anything worse.
This is different from Slurm Configuration, which changes the configuration. Reconfigure just re-applies it.
Clone Cluster
Creates a new cluster pre-filled from this one. Available at any status — including Failed and Terminated, which makes it the standard way to recover from a bad cluster.
- Actions → Clone Cluster. You are taken to the create form, pre-filled.
- Review every section, adjust what you want, and submit Create Training Cluster.
| Carried over | Not carried over |
|---|---|
| Image and version | Cluster name — a new one is generated |
| SSH keys, security group | Partitions — they name the original cluster's nodes |
| Node count, plan / SKU, private cluster | Live state: jobs, node health, attached IPs |
| Lifecycle script | |
| Storage mounts (PFS, SFS, datasets, Weka) and their paths | |
Extra slurm.conf and all four prolog/epilog scripts |
It only pre-fills the form. Nothing is provisioned, and nothing is billed, until you submit. The clone reflects the source cluster's current shape, not how it was originally created.
Because partitions cannot be set at create time, a cloned cluster starts with only the platform partitions. Re-create yours from the partition table once it is Running.
Convert to Committed
Switches billing from hourly to a discounted committed plan. Titled Convert to Committed Plan.
You can reserve this training cluster by converting it into a committed service at a discounted price.
-
Actions → Convert to Committed.
-
Committed Plan — pick a plan; options read like
30 Days Committed, ₹<price>. -
Renewal Policy — what happens when the commitment ends:
Option Effect Auto Renewal Renews the same committed plan Convert to Hourly Plan Starts hourly billing at the listed rate Auto Delete Deletes the cluster at the end of the term -
Read the acknowledgement, which is ticked by default, and click Continue.
| Detail | Value |
|---|---|
| Restart / downtime | None. This is purely a billing change — nothing on the cluster changes |
| Not available for | CPU clusters, private-cluster plans, clusters already committed |
| Reversible | No, not within the commitment window |
The acknowledgement text is precise: the whole cost for the commitment period is deducted immediately, you cannot terminate before the term ends, and terminating early refunds nothing for the remaining days. Be sure of the node count and duration before confirming.
Once committed, Scale Cluster and a further Convert to Committed are both hidden. Size the cluster before you commit.
Terminate Cluster
Stops the cluster, releases its nodes, and stops compute billing.
- Actions → Terminate Cluster (in the red group at the bottom).
- Confirm: "Are you sure you want to terminate the cluster <name>?"
- Click Terminate.
The status goes to Terminating, then Terminated.
| What happens | Detail |
|---|---|
| Compute | Nodes are released back to the pool. The Slurm release, its pods, its internal volumes and secrets are destroyed |
| Billing | Compute billing stops |
| Attached storage | Unmounted, not deleted. Your PFS, SFS, dataset and Weka volumes survive — and keep billing |
| IPs | Reserved and VPC IPs are detached |
| Login users | Persistent Login is disabled as part of termination |
| The record | Stays in the list at status Terminated, so you can still Clone or Delete it |
There is no acknowledgement checkbox on this dialog — a single confirmation ends the cluster. Running jobs are lost. Anything not written to a PFS, SFS or Weka mount is gone with the nodes.
Terminating the cluster does not delete your volumes. Review them under Storage in the TIR sidebar and delete what you no longer need. See Billing.
If a VPC IP is mid-attach, terminate is refused until it settles. Wait a moment and retry, or check the Network & Security tab.
Delete Cluster
Removes the cluster record permanently. Labelled Delete Training Cluster on the list page and Delete Cluster on the details page.
- Actions → Delete Training Cluster.
- The dialog reads "This will permanently delete the training cluster and all associated resources."
- Tick "I understand and want to delete the training cluster <name>."
- Click Delete.
| Detail | Value |
|---|---|
| Available at | Any status. On an already-Terminated cluster it is immediate, since teardown is done |
| After deletion | The cluster leaves the list and no further action on it is possible |
| Attached storage | Unmounted, not deleted — the volumes survive |
Once issued, the cluster is gone from the list and every operation on it fails. Clone it first if you might want the same configuration again — a Terminated cluster can still be cloned, a deleted one cannot.
Terminate or delete?
| Terminate | Delete | |
|---|---|---|
| Final status | Terminated | Deleted |
| Still in the list | Yes | No |
| Can still be cloned | Yes | No |
| Acknowledgement checkbox | No | Yes |
| Stops billing | Yes | Yes |
The usual sequence is terminate, keep the record while you might still want to clone its configuration, and delete later during cleanup.
Per-Node Actions
Rebooting a single node is not in the Actions menu — it lives on the Nodes tab, where you can also see which node is reporting XID errors. Prefer it over Restart All Workers when only one node is unhealthy.