Update a cluster

Change which SSH keys have root access to a running TIR Slurm Cluster, move it to another software image, or restart part of it. Every action here is on the Actions menu, reached from the cluster’s row in the list under Training Cluster or from the Actions button on its details page.

Before you begin

• Update permission on Training Clusters in the project, which puts these actions on the Actions menu.
• Checked the queue before any action that restarts nodes. These actions go ahead whatever is in the Slurm queue, and Update Image is the one that drains nodes. Open the Jobs tab or run squeue, and let jobs finish or drain the nodes first with scontrol update nodename=<node> state=drain reason="<why>".
• Checkpoints on a PFS or SFS mount, so that a lost job costs minutes of work rather than days.

Update SSH keys

Update SSH Keys replaces the whole set of root SSH keys on the cluster in one step. A key left out of the new selection loses access. The dialog pre-selects the current keys, so deselecting one removes it. Keep at least two keys on a cluster, so that you can still log in when one machine is lost. For team access, use Login User Management instead, so that each person has their own key and can be removed individually.

The action is offered in every status except Creating, Initializing Worker Nodes, Health Check Running, and Failed, and it restarts nothing.

1. Select Actions, then Update SSH Keys.
2. In Select SSH Keys, choose the keys. To add a new key, use the Click here link, which opens Add SSH Key and keeps your place in this dialog.
3. Select Update SSH Keys. The button becomes available once you choose at least one key and the selection differs from the current one.

Update the cluster image

Update Image moves the cluster to a different version of its own software image: Slurm, the drivers, and the operating system. A job’s container image is set separately, so to run a different framework version, pass --container-image to srun or sbatch. See Run containers in jobs.

The action is offered only while the cluster is Running.

1. Select Actions, then Update Image.
2. Choose an image family, such as Ubuntu Slurm or NeMo Framework, then a version. Latest marks the newest version and Current marks the one running now.
3. Select Update Image, then read the confirmation.

The confirmation describes the rolling restart: nodes restart one at a time, running jobs are drained before each node restarts, and pending jobs wait until the nodes are back. It estimates 3 to 8 minutes of downtime for each node, so a long-running job is still affected; schedule the update for a quiet period. Select Apply & Rolling Restart to confirm.

The local disks of the login node and workers are recreated, so anything stored outside a mounted volume is lost.

Restart workers, the login service, or the cluster

The three restart actions differ in how much they restart. They are offered only while the cluster is Running, and each goes ahead whatever is in the Slurm queue.

Action	What restarts	Use it when
Restart Login Service	The login node	SSH connections fail, or the login node is out of resources
Restart All Workers	Every worker node	Nodes are unhealthy, report XID errors, or are stuck after a fault
Restart Cluster	The controller, the login node, every worker, and the accounting database	The narrower restarts have failed to fix it

Restart All Workers is also a header button on the details page, labelled Restart Workers.

Escalate in order: Restart Login Service first, then Restart All Workers, then Restart Cluster. For one bad node, reboot that node from the Nodes tab instead. See Check cluster and node health.

Restart All Workers and Restart Cluster end every running job. Restart Login Service drops every open SSH session, and anything running in a foreground shell on the login node ends with it. Submit long work with sbatch, so that it survives a login restart. Restarting the workers, the login service, or the whole cluster recreates those nodes and erases their local disks. Download a log first if you need it, because the earlier log is gone once the node is replaced.

Reconfigure the cluster

Reconfigure Cluster applies the cluster’s stored settings again, exactly as they are. It is a repair action. Use it for a cluster stuck in Creating. Use it also when the live state seems to differ from the console, such as a volume that reads as mounted but is missing, or a setting that failed to take effect.

It recreates the login node, which erases the login node’s own disk, including /root, so keep your work on a mounted volume first. It is offered while the cluster is Running or still being set up, and it asks for one confirmation.

Slurm Configuration is the action that changes the stored settings. See Change the Slurm configuration.

Your billing stays the same through all of these actions. See Billing and plans.

To change the cluster’s size, see Scale a cluster. If an action is missing from the menu or a restart leaves a node Unknown, see Troubleshooting TIR Slurm Cluster.