Scale a cluster

Add worker nodes to a TIR Slurm Cluster, or remove them. Scaling up adds the next node names, so a four-node cluster scaled to six gains slinky-4 and slinky-5. Scaling down removes the nodes you choose.

Before you begin

• A cluster in the Running state on an hourly plan or on a TIR Private Cluster. A committed cluster keeps its size, so Scale Cluster is hidden there. See Cluster statuses and actions.
• Update permission on Training Clusters in the project.
• For a scale-up, enough project credit for the extra nodes, and free nodes of the same type.
• For a scale-down, checked the queue first, because the scale-down goes ahead whatever is running.

Scale up

1. Select Actions, then Scale Cluster, then Scale up.
2. Set Total Node Count to the new total, counting the nodes you already have.
3. Check the cost estimate, which shows New hourly cost and Change from current.
4. Select Scale Pool.

How far you can scale up depends on free nodes on the same rack as the cluster’s existing nodes. When every node of that type on the rack is taken, the dialog tells you to check back later or contact support for that GPU type. If only some of the nodes you asked for are free, it names how many are available. TIR checks credit and eligibility before it reserves any node.

The billed node count rises immediately, and the scale action itself is free. New nodes join while the existing nodes keep running, and so do their jobs. Keep your work on a mounted volume, because a node’s local disk is erased whenever the node is recreated.

Your own partitions keep the nodes you named, so edit them to add the new nodes. See Manage partitions.

Scale down

Scaling down removes the nodes you select, and the change restarts the remaining workers too. Jobs on the removed nodes are lost, and jobs on the remaining nodes end as well, whatever is in the Slurm queue. To protect your jobs, drain the nodes with scontrol update nodename=<node> state=drain reason="<why>", wait for squeue to clear, and then scale down.

1. Select Actions, then Scale Cluster, then Scale down.
2. Under Select nodes to shut down, choose the nodes to remove. A counter shows how many you selected, and a cost estimate shows the saving.
3. Select Remove N nodes.

At least one node must remain, so leave at least one node unselected. The billed node count drops immediately.

If every node a partition named is removed, the partition shows as not live, and jobs sent to it stay pending. Edit it and name nodes that still exist.

Choose the right size

You pay for nodes, busy or idle, and jobs themselves are free. A pending backlog that persists means scale up, and a queue that stays empty means scale down. The Monitoring tab shows both; see Monitor a cluster. Size the cluster before you convert it to a committed plan, because a committed cluster keeps its size for the whole term. See Convert a cluster to committed.

For help when a scale-up is refused, for example on a committed cluster or when every node of the right type is taken, see Troubleshooting TIR Slurm Cluster. For how nodes are billed, see Billing and plans.