--- title: Cluster Actions sidebar_label: Actions --- import { Key, Box, Sliders, Maximize2, RotateCcw, RefreshCw, LogIn, Monitor, Copy, GitCommit, StopCircle, Trash2 } from 'react-feather'; # Cluster Actions Every lifecycle operation on a Slurm Cluster runs from the **Actions** menu. This page documents each one — what it does, what it asks you, what it restarts, and when it is available. }, { href: '#update-ssh-keys', label: 'Update SSH Keys', icon: }, { href: '#update-image', label: 'Update Image', icon: }, { href: '#slurm-configuration', label: 'Slurm Configuration', icon: }, { href: '#scale-cluster', label: 'Scale Cluster', icon: }, { href: '#restart-actions', label: 'Restart actions', icon: }, { href: '#reconfigure-cluster', label: 'Reconfigure', icon: }, { href: '#clone-cluster', label: 'Clone', icon: }, { href: '#convert-to-committed', label: 'Convert to Committed', icon: }, { href: '#terminate-cluster', label: 'Terminate', icon: , destructive: true }, { href: '#delete-cluster', label: 'Delete', icon: , destructive: true }, ]} /> --- ## Where the Actions Live | Entry point | How to reach it | |-------------|-----------------| | **Cluster list** | **Training Cluster** in the sidebar → the ⋮ icon on a cluster row | | **Cluster details** | Open the cluster → **Actions** button in the top-right area | | **Quick actions** | The details header also carries **Connect**, **Restart Workers** and a refresh icon as buttons | Both menus offer the same actions in the destructive-last order below. Actions you do not have update permission for are not shown. | Action | Available when | |--------|----------------| | **Update SSH Keys** | Any status | | **Update Image** | Running | | **Slurm Configuration** | Running | | **Scale Cluster** | Running, and the cluster is on an hourly plan or a private cluster | | **Convert to Committed** | Hourly, non-private-cluster plans, and not Failed / Terminating / Terminated | | **Restart All Workers** | Running | | **Restart Cluster** | Running | | **Restart Login Service** | Running | | **Reconfigure Cluster** | Running or Creating | | **Clone Cluster** | Any status | | **Terminate Cluster** | Any status | | **Delete Training Cluster** | Any status | For the full picture of what hides or blocks an action, see [Action availability and blockers](/docs/tir/SlurmCluster/manage/action-availability). --- ## Update SSH Keys Replaces the set of SSH keys that can log in to the cluster. 1. **Actions** → **Update SSH Keys**. 2. The dialog reads *"Manage SSH keys for cluster **<name>**"*. 3. Use the **Select SSH Keys** multi-select to choose the keys. The count shows as e.g. `2 selected`. 4. To add a brand-new key, use the **Click here** link — it opens **Add SSH Key** (*"Attach a public key to authenticate with your server"*) without losing your place. 5. Click **Update SSH Keys**. | Detail | Value | |--------|-------| | Restart required | No | | Available at | Any cluster status | | Confirm blocked when | Nothing selected, or the selection is unchanged | :::warning This replaces the whole set, it does not add to it Keys you leave out of the selection **lose access**. The dialog pre-selects the cluster's current keys — deselecting one removes it. If nothing is selected you see *"No SSH keys selected. You need at least one SSH key to access the cluster."* and cannot continue. ::: :::tip Prefer per-user identities for a team Rotating a shared `root` key affects everyone at once. For team access, use [Login User Management](/docs/tir/SlurmCluster/connect/login-user-management) — each person gets their own key, their own home directory, and can be revoked individually. ::: --- ## Update Image Moves the cluster to a different version of its Slurm image — the software stack the controller, login node and workers run. 1. **Actions** → **Update Image**. The dialog is titled **Update Image: <name>**. 2. **Current Image** shows what is running, e.g. `ubuntu-slurm:v2-slurm25.11-ubuntu24.04`. 3. Under **SELECT IMAGE**, pick the family — **Ubuntu Slurm** or **NeMo Framework**. 4. Under **SELECT VERSION**, pick a version. Rows carry badges: **Latest** for the newest, **Current** for the one running now. 5. Click **Update Image**, then read the confirmation. The confirmation dialog is **Confirm Image Update** and is explicit about the cost: > **Rolling Restart Required** > - Nodes will restart one at a time to apply the new image > - Running jobs will be drained before each node restarts > - Pending jobs will queue until nodes come back online > - Estimated downtime per node: 3-8 minutes Confirm with **Apply & Rolling Restart**. Success toast: *"Image update applied successfully. Rolling restart initiated."* :::danger This is not the job container image **Update Image** changes the **cluster's own** software — Slurm, the drivers, the base OS. It has nothing to do with the container your jobs run in. To change that, pass `--container-image` to `srun`/`sbatch`; see [Containers with Enroot and Pyxis](/docs/tir/SlurmCluster/containers/). You do **not** need to update the cluster image to run a different framework version. ::: :::warning Drain your queue first on a busy cluster Running jobs are drained node by node. On a cluster with long jobs, an image update can take a long time to work through and your jobs are interrupted. Schedule it, and make sure your jobs [checkpoint to shared storage](/docs/tir/SlurmCluster/containers/run-containers#mount-your-data). ::: --- ## Slurm Configuration Edits `slurm.conf` extras, partitions, and prolog/epilog scripts. Available only while the cluster is **Running**. Covered in full in [Slurm Configuration](/docs/tir/SlurmCluster/slurm-configuration/): - [Extra `slurm.conf` settings](/docs/tir/SlurmCluster/slurm-configuration/slurm-conf) - [Partitions](/docs/tir/SlurmCluster/slurm-configuration/partitions) - [Prolog and epilog scripts](/docs/tir/SlurmCluster/slurm-configuration/prolog-epilog) Running jobs are never interrupted; queued jobs pick up the new settings; prolog and epilog changes apply from the next job. --- ## Scale Cluster Adds or removes worker nodes on a live cluster. The dialog is titled **Scale pool: <name>** and shows the **GPU type** and **Current nodes** before anything else. Two modes, selected by the **Scale up** / **Scale down** buttons. ### Scale up 1. **Actions** → **Scale Cluster** → **Scale up**. 2. Set **Total Node Count** with the stepper. The value is the **new total**, not the number to add, and the side text shows the resulting GPU count. 3. A cost card shows **New hourly cost** and **Change from current**. 4. Click **Scale Pool**. If capacity is short you get, instead of the stepper: > **Inventory Unavailable** — No additional inventory is currently available. > > **What you can do:** > - Check back later — inventory is refreshed frequently. > - Contact support for NVIDIA <GPU type> capacity. When only some of what you asked for is free, the message reads *"Only <n> nodes available — you requested <m>."* with a link that sets the stepper to the maximum available. | Detail | Value | |--------|-------| | Maximum nodes | 512 | | Restart required | No — new nodes join; existing jobs keep running | | New nodes are placed | On the same rack as your existing nodes, to keep fabric performance consistent | :::info New nodes are not in your custom partitions automatically A partition holds the nodes you named. After scaling up, edit the partition to include the new nodes — otherwise they are only reachable via the platform `all` partition. See [Manage partitions](/docs/tir/SlurmCluster/slurm-configuration/partitions#edit-a-partition). ::: ### Scale down 1. **Actions** → **Scale Cluster** → **Scale down**. 2. Under **Select nodes to shut down**, click the nodes to remove. A counter shows `N selected` and a cost card shows the saving. 3. Click **Remove N nodes**. | Detail | Value | |--------|-------| | Minimum | At least one node must remain. The last unselected node cannot be chosen, and selecting everything shows *"At least 1 node must remain active."* | | Restart | **Scale-down also restarts the remaining worker pods.** The dialog does not say so | :::danger Scale-down ends jobs on the removed nodes, and restarts the rest Nothing checks the Slurm queue before removing a node. Jobs running on a node you remove are lost, and because the remaining workers restart, jobs on **those** nodes are affected too. Drain the cluster first: `scontrol update nodename= state=drain`, wait for `squeue` to clear, then scale down. ::: :::warning A partition can silently stop working If every node a partition named is removed, that partition stops being written to `slurm.conf` and shows as `not live` in the partition table. Repoint it at nodes that still exist. See [the scale-down note](/docs/tir/SlurmCluster/slurm-configuration/partitions#a-partition-that-stopped-working-after-a-scale-down). ::: **Scale Cluster is hidden on committed plans.** The API refuses it with *"Scale cluster is not supported for committed cluster"*. --- ## Restart Actions Three restart actions with very different blast radius. Each asks a single confirmation. | Action | Dialog body | What restarts | Use it when | |--------|-------------|---------------|-------------| | **Restart All Workers** | *"Are you sure you want to restart all workers in cluster **<name>**?"* | Every worker (`slurmd`) pod | Nodes are unhealthy, reporting XID errors, or wedged after a hardware fault | | **Restart Login Service** | *"Are you sure you want to restart the login service for cluster **<name>**?"* Confirm button: **Restart** | The login node pods only | You cannot SSH in, or the login node is out of resources | | **Restart Cluster** | *"Are you sure you want to restart the entire cluster **<name>**?"* | Controller, login node, **all** workers and the accounting database | Nothing else has worked. This is the heaviest routine action | **Restart All Workers** is also a button in the details header, labelled **Restart Workers**. It is disabled unless the cluster is Running. :::danger All three end running jobs on the pods they restart There is no drain step and no queue check. **Restart All Workers** and **Restart Cluster** end every running job. Check the [Jobs tab](/docs/tir/SlurmCluster/manage/jobs) first, and prefer draining and letting jobs finish where you can. ::: :::warning Restart Login Service disconnects open SSH sessions Anyone logged in is dropped, and anything running in a plain shell there dies. Work you care about should be a Slurm job (`sbatch`), not a foreground process on the login node — a job survives a login restart, a shell does not. ::: :::tip Escalate in order Try the narrowest action first: **Restart Login Service** for access problems → **Restart All Workers** for node or GPU problems → **Restart Cluster** only if the controller itself is unhealthy. For a single bad node, use the per-node reboot on the [Nodes tab](/docs/tir/SlurmCluster/manage/nodes) instead of restarting everything. ::: --- ## Reconfigure Cluster Re-applies the cluster's stored configuration without changing anything. It is a repair action. 1. **Actions** → **Reconfigure Cluster**. 2. Confirm: *"Are you sure you want to Reconfigure Cluster **<name>**?"* | Detail | Value | |--------|-------| | Available when | Running **or Creating** — one of the few actions offered on a Creating cluster | | Changes | Nothing in your configuration. It re-pushes what is already stored | :::tip When to reach for it A cluster stuck in **Creating**, or one whose live state has drifted from what the console shows — a volume that reads as mounted but is not there, a setting that did not take. Reconfigure re-applies the intended state. If it does not help, it has not made anything worse. ::: This is different from **Slurm Configuration**, which *changes* the configuration. Reconfigure just re-applies it. --- ## Clone Cluster Creates a new cluster pre-filled from this one. Available at any status — including **Failed** and **Terminated**, which makes it the standard way to recover from a bad cluster. 1. **Actions** → **Clone Cluster**. You are taken to the create form, pre-filled. 2. Review every section, adjust what you want, and submit **Create Training Cluster**. | Carried over | Not carried over | |--------------|------------------| | Image and version | **Cluster name** — a new one is generated | | SSH keys, security group | **Partitions** — they name the original cluster's nodes | | Node count, plan / SKU, private cluster | Live state: jobs, node health, attached IPs | | Lifecycle script | | | Storage mounts (PFS, SFS, datasets, Weka) and their paths | | | Extra `slurm.conf` and all four prolog/epilog scripts | | :::info Cloning does not create anything by itself It only pre-fills the form. Nothing is provisioned, and nothing is billed, until you submit. The clone reflects the source cluster's **current** shape, not how it was originally created. ::: :::warning Re-add your partitions after the clone is running Because partitions cannot be set at create time, a cloned cluster starts with only the platform partitions. Re-create yours from the [partition table](/docs/tir/SlurmCluster/slurm-configuration/partitions#add-a-partition) once it is Running. ::: --- ## Convert to Committed Switches billing from hourly to a discounted committed plan. Titled **Convert to Committed Plan**. > You can reserve this training cluster by converting it into a committed service at a discounted > price. 1. **Actions** → **Convert to Committed**. 2. **Committed Plan** — pick a plan; options read like `30 Days Committed, ₹`. 3. **Renewal Policy** — what happens when the commitment ends: | Option | Effect | |--------|--------| | **Auto Renewal** | Renews the same committed plan | | **Convert to Hourly Plan** | Starts hourly billing at the listed rate | | **Auto Delete** | Deletes the cluster at the end of the term | 4. Read the acknowledgement, which is ticked by default, and click **Continue**. | Detail | Value | |--------|-------| | Restart / downtime | **None.** This is purely a billing change — nothing on the cluster changes | | Not available for | CPU clusters, private-cluster plans, clusters already committed | | Reversible | No, not within the commitment window | :::danger The full committed amount is charged upfront and is non-refundable The acknowledgement text is precise: the whole cost for the commitment period is deducted immediately, you cannot terminate before the term ends, and terminating early refunds nothing for the remaining days. Be sure of the node count and duration before confirming. ::: :::info Committed clusters cannot be scaled Once committed, **Scale Cluster** and a further **Convert to Committed** are both hidden. Size the cluster before you commit. ::: --- ## Terminate Cluster Stops the cluster, releases its nodes, and stops compute billing. 1. **Actions** → **Terminate Cluster** (in the red group at the bottom). 2. Confirm: *"Are you sure you want to terminate the cluster **<name>**?"* 3. Click **Terminate**. The status goes to **Terminating**, then **Terminated**. | What happens | Detail | |--------------|--------| | Compute | Nodes are released back to the pool. The Slurm release, its pods, its internal volumes and secrets are destroyed | | Billing | Compute billing stops | | Attached storage | **Unmounted, not deleted.** Your PFS, SFS, dataset and Weka volumes survive — and keep billing | | IPs | Reserved and VPC IPs are detached | | Login users | Persistent Login is disabled as part of termination | | The record | Stays in the list at status **Terminated**, so you can still **Clone** or **Delete** it | :::danger Terminate is irreversible and it does not warn you twice There is no acknowledgement checkbox on this dialog — a single confirmation ends the cluster. Running jobs are lost. Anything not written to a PFS, SFS or Weka mount is gone with the nodes. ::: :::warning Storage keeps billing after termination Terminating the cluster does not delete your volumes. Review them under **Storage** in the TIR sidebar and delete what you no longer need. See [Billing](/docs/tir/SlurmCluster/billing#storage-is-billed-separately). ::: :::info Blocked by a VPC IP that is still attaching If a VPC IP is mid-attach, terminate is refused until it settles. Wait a moment and retry, or check the [Network & Security tab](/docs/tir/SlurmCluster/manage/network-security). ::: --- ## Delete Cluster Removes the cluster record permanently. Labelled **Delete Training Cluster** on the list page and **Delete Cluster** on the details page. 1. **Actions** → **Delete Training Cluster**. 2. The dialog reads *"This will permanently delete the training cluster and all associated resources."* 3. Tick **"I understand and want to delete the training cluster <name>."** 4. Click **Delete**. | Detail | Value | |--------|-------| | Available at | Any status. On an already-Terminated cluster it is immediate, since teardown is done | | After deletion | The cluster leaves the list and **no further action on it is possible** | | Attached storage | Unmounted, not deleted — the volumes survive | :::danger Delete is final Once issued, the cluster is gone from the list and every operation on it fails. Clone it first if you might want the same configuration again — a **Terminated** cluster can still be cloned, a deleted one cannot. ::: ### Terminate or delete? | | Terminate | Delete | |--|-----------|--------| | Final status | `Terminated` | `Deleted` | | Still in the list | Yes | No | | Can still be cloned | Yes | No | | Acknowledgement checkbox | No | Yes | | Stops billing | Yes | Yes | The usual sequence is **terminate**, keep the record while you might still want to clone its configuration, and **delete** later during cleanup. --- ## Per-Node Actions Rebooting a single node is not in the Actions menu — it lives on the [**Nodes** tab](/docs/tir/SlurmCluster/manage/nodes#reboot-a-node), where you can also see which node is reporting XID errors. Prefer it over **Restart All Workers** when only one node is unhealthy. --- ## Related Resources - [Action availability and blockers](/docs/tir/SlurmCluster/manage/action-availability) - [Manage a Slurm Cluster](/docs/tir/SlurmCluster/manage/) - [Slurm Configuration](/docs/tir/SlurmCluster/slurm-configuration/) - [Nodes and GPU health](/docs/tir/SlurmCluster/manage/nodes) - [Billing](/docs/tir/SlurmCluster/billing)