--- title: "Update a cluster" description: "Change a TIR Slurm Cluster's root SSH keys or software image, restart its workers, login service, or whole cluster, and reconfigure it from its stored settings." content_type: "how-to" product: "tir-slurm-cluster" platform: "TIR" url: "/tir/tir-slurm-cluster/how-to/update-a-cluster/" topic: ["manage-the-cluster"] --- # Update a cluster Change which SSH keys have root access to a running TIR Slurm Cluster, move it to another software image, or restart part of it. Every action here is on the **Actions** menu, reached from the cluster's row in the list under **Training Cluster** or from the **Actions** button on its details page. ## Before you begin - Update permission on Training Clusters in the project, which puts these actions on the **Actions** menu. - Checked the queue before any action that restarts nodes. These actions go ahead whatever is in the Slurm queue, and **Update Image** is the one that drains nodes. Open the **Jobs** tab or run `squeue`, and let jobs finish or drain the nodes first with `scontrol update nodename= state=drain reason=""`. - Checkpoints on a PFS or SFS mount, so that a lost job costs minutes of work rather than days. ## Update SSH keys **Update SSH Keys** replaces the whole set of root SSH keys on the cluster in one step. A key left out of the new selection loses access. The dialog pre-selects the current keys, so deselecting one removes it. Keep at least two keys on a cluster, so that you can still log in when one machine is lost. For team access, use [Login User Management](/tir/tir-slurm-cluster/how-to/manage-login-users/) instead, so that each person has their own key and can be removed individually. The action is offered in every status except **Creating**, **Initializing Worker Nodes**, **Health Check Running**, and **Failed**, and it restarts nothing. 1. Select **Actions**, then **Update SSH Keys**. 2. In **Select SSH Keys**, choose the keys. To add a new key, use the **Click here** link, which opens **Add SSH Key** and keeps your place in this dialog. 3. Select **Update SSH Keys**. The button becomes available once you choose at least one key and the selection differs from the current one. ## Update the cluster image **Update Image** moves the cluster to a different version of its own software image: Slurm, the drivers, and the operating system. A job's container image is set separately, so to run a different framework version, pass `--container-image` to `srun` or `sbatch`. See [Run containers in jobs](/tir/tir-slurm-cluster/how-to/run-containers-in-jobs/). The action is offered only while the cluster is **Running**. 1. Select **Actions**, then **Update Image**. 2. Choose an image family, such as **Ubuntu Slurm** or **NeMo Framework**, then a version. **Latest** marks the newest version and **Current** marks the one running now. 3. Select **Update Image**, then read the confirmation. The confirmation describes the rolling restart: nodes restart one at a time, running jobs are drained before each node restarts, and pending jobs wait until the nodes are back. It estimates 3 to 8 minutes of downtime for each node, so a long-running job is still affected; schedule the update for a quiet period. Select **Apply & Rolling Restart** to confirm. The local disks of the login node and workers are recreated, so anything stored outside a mounted volume is lost. ## Restart workers, the login service, or the cluster The three restart actions differ in how much they restart. They are offered only while the cluster is **Running**, and each goes ahead whatever is in the Slurm queue. | Action | What restarts | Use it when | | ------------------------- | ------------------------------------------------------------------------- | ------------------------------------------------------------------ | | **Restart Login Service** | The login node | SSH connections fail, or the login node is out of resources | | **Restart All Workers** | Every worker node | Nodes are unhealthy, report XID errors, or are stuck after a fault | | **Restart Cluster** | The controller, the login node, every worker, and the accounting database | The narrower restarts have failed to fix it | **Restart All Workers** is also a header button on the details page, labelled **Restart Workers**. Escalate in order: **Restart Login Service** first, then **Restart All Workers**, then **Restart Cluster**. For one bad node, reboot that node from the **Nodes** tab instead. See [Check cluster and node health](/tir/tir-slurm-cluster/how-to/check-cluster-and-node-health/). **Restart All Workers** and **Restart Cluster** end every running job. **Restart Login Service** drops every open SSH session, and anything running in a foreground shell on the login node ends with it. Submit long work with `sbatch`, so that it survives a login restart. Restarting the workers, the login service, or the whole cluster recreates those nodes and erases their local disks. Download a log first if you need it, because the earlier log is gone once the node is replaced. ## Reconfigure the cluster **Reconfigure Cluster** applies the cluster's stored settings again, exactly as they are. It is a repair action. Use it for a cluster stuck in **Creating**. Use it also when the live state seems to differ from the console, such as a volume that reads as mounted but is missing, or a setting that failed to take effect. It recreates the login node, which erases the login node's own disk, including `/root`, so keep your work on a mounted volume first. It is offered while the cluster is **Running** or still being set up, and it asks for one confirmation. **Slurm Configuration** is the action that changes the stored settings. See [Change the Slurm configuration](/tir/tir-slurm-cluster/how-to/change-the-slurm-configuration/). Your billing stays the same through all of these actions. See [Billing and plans](/tir/tir-slurm-cluster/concepts/billing-and-plans/). To change the cluster's size, see [Scale a cluster](/tir/tir-slurm-cluster/how-to/scale-a-cluster/). If an action is missing from the menu or a restart leaves a node **Unknown**, see [Troubleshooting TIR Slurm Cluster](/tir/tir-slurm-cluster/troubleshooting/tir-slurm-cluster/).