--- title: "Scale a cluster" description: "Add worker nodes to a running hourly TIR Slurm Cluster or remove nodes you pick, and know what scaling does to jobs, partitions, and billing." content_type: "how-to" product: "tir-slurm-cluster" platform: "TIR" url: "/tir/tir-slurm-cluster/how-to/scale-a-cluster/" topic: ["manage-the-cluster"] --- # Scale a cluster Add worker nodes to a TIR Slurm Cluster, or remove them. Scaling up adds the next node names, so a four-node cluster scaled to six gains `slinky-4` and `slinky-5`. Scaling down removes the nodes you choose. ## Before you begin - A cluster in the **Running** state on an hourly plan or on a TIR Private Cluster. A committed cluster keeps its size, so **Scale Cluster** is hidden there. See [Cluster statuses and actions](/tir/tir-slurm-cluster/concepts/cluster-statuses-and-actions/). - Update permission on Training Clusters in the project. - For a scale-up, enough project credit for the extra nodes, and free nodes of the same type. - For a scale-down, checked the queue first, because the scale-down goes ahead whatever is running. ## Scale up 1. Select **Actions**, then **Scale Cluster**, then **Scale up**. 2. Set **Total Node Count** to the new total, counting the nodes you already have. 3. Check the cost estimate, which shows **New hourly cost** and **Change from current**. 4. Select **Scale Pool**. How far you can scale up depends on free nodes on the same rack as the cluster's existing nodes. When every node of that type on the rack is taken, the dialog tells you to check back later or contact support for that GPU type. If only some of the nodes you asked for are free, it names how many are available. TIR checks credit and eligibility before it reserves any node. The billed node count rises immediately, and the scale action itself is free. New nodes join while the existing nodes keep running, and so do their jobs. Keep your work on a mounted volume, because a node's local disk is erased whenever the node is recreated. Your own partitions keep the nodes you named, so edit them to add the new nodes. See [Manage partitions](/tir/tir-slurm-cluster/how-to/manage-partitions/). ## Scale down Scaling down removes the nodes you select, and the change restarts the remaining workers too. Jobs on the removed nodes are lost, and jobs on the remaining nodes end as well, whatever is in the Slurm queue. To protect your jobs, drain the nodes with `scontrol update nodename= state=drain reason=""`, wait for `squeue` to clear, and then scale down. 1. Select **Actions**, then **Scale Cluster**, then **Scale down**. 2. Under **Select nodes to shut down**, choose the nodes to remove. A counter shows how many you selected, and a cost estimate shows the saving. 3. Select **Remove N nodes**. At least one node must remain, so leave at least one node unselected. The billed node count drops immediately. If every node a partition named is removed, the partition shows as **not live**, and jobs sent to it stay pending. Edit it and name nodes that still exist. ## Choose the right size You pay for nodes, busy or idle, and jobs themselves are free. A pending backlog that persists means scale up, and a queue that stays empty means scale down. The **Monitoring** tab shows both; see [Monitor a cluster](/tir/tir-slurm-cluster/how-to/monitor-a-cluster/). Size the cluster before you convert it to a committed plan, because a committed cluster keeps its size for the whole term. See [Convert a cluster to committed](/tir/tir-slurm-cluster/how-to/convert-a-cluster-to-committed/). For help when a scale-up is refused, for example on a committed cluster or when every node of the right type is taken, see [Troubleshooting TIR Slurm Cluster](/tir/tir-slurm-cluster/troubleshooting/tir-slurm-cluster/). For how nodes are billed, see [Billing and plans](/tir/tir-slurm-cluster/concepts/billing-and-plans/).