Clone a cluster

Open a new create form already filled in from an existing TIR Slurm Cluster. Clone a cluster to repeat a setup, or to replace a cluster that failed. Clone Cluster is available at any status, including Failed and Terminated.

Before you begin

You need a cluster to copy. You also need Create permission on Training Clusters in the project, because the action leads to the create form. Cloning fills in the form and stops there, so the cluster is created, and billed, only once you submit it. The form follows the source cluster’s current settings, including any changes made since it was created.

Clone the cluster

1. Select Actions, then Clone Cluster. The create form opens, filled in from the cluster.
2. Review every section, change what you want, and select Create Training Cluster.

The new cluster gets the original’s image and version, SSH keys, security group, node count, and plan, and its TIR Private Cluster if it used one. It also gets the lifecycle script, the storage mounts with their mount paths, and the extra slurm.conf text with all four prolog and epilog scripts. A new cluster name is generated for you.

Partitions stay behind, because a partition names the original cluster’s own nodes. Live state such as jobs, node health, and attached IPs stays behind too. Add your partitions again once the new cluster is Running. See Manage partitions.

Replace a failed cluster

When a cluster ends up Failed, its reserved nodes are released and its security group and storage selections are cleared. The failed record stays in the list, free of compute charges, until you delete it. A Failed cluster stays Failed, so to get a Running cluster, replace it:

1. Clone the failed cluster. Check the security group and storage sections, which a failed cluster has cleared.
2. Change anything that might have caused the failure. The most common cause is a plan whose nodes are all taken, so try a different GPU type, fewer nodes, or a TIR Private Cluster.
3. Create the new cluster and wait until it reaches Running.
4. Delete the failed record. See Terminate or delete a cluster.

Keep a setup for later

A cluster’s configuration survives termination. You can clone a terminated cluster until you delete it, so clone it first if you might want the same setup again. Terminating stops compute billing, which makes terminating and cloning later cheaper than keeping a cluster idle. See Billing and plans.

To create a cluster from scratch instead, see Create a Slurm Cluster. If a create keeps failing, see Troubleshooting TIR Slurm Cluster.