--- title: "Manage partitions" description: "Add, edit, and delete your own partitions on a TIR Slurm Cluster to split the cluster into named queues over chosen nodes, from the console." content_type: "how-to" product: "tir-slurm-cluster" platform: "TIR" url: "/tir/tir-slurm-cluster/how-to/manage-partitions/" topic: ["configure-slurm"] --- # Manage partitions Split your cluster into partitions, your own named queues over chosen nodes, each with its own state, time limits, members, and extra options. A common first one is a short-jobs queue, so quick tests get a queue of their own rather than waiting behind a long training run. ## Before you begin - A cluster in the **Running** state. A partition names specific nodes, and those nodes exist once the cluster is up, so you add partitions after creating the cluster. See [Cluster statuses and actions](/tir/tir-slurm-cluster/concepts/cluster-statuses-and-actions/). - Update permission on Training Clusters in the project. - The Slurm names of the nodes you want in the partition: `slinky-0`, `slinky-1`, and so on up to `slinky-(N-1)` for a cluster of N nodes. The **Nodes** tab shows each node's Slurm name beside its machine identifier. ## Know the platform partitions Every cluster comes with two partitions that the platform manages for you: - `all` covers every node. Jobs that leave out `--partition` land on it, unless you make another partition the default. - `slinky` covers the GPU worker nodes. Your first `sinfo` lists both of them. In the partition table they read **Managed**, and you cannot edit or delete them. The names `llm`, `asr`, `vision`, `tts`, and `default` are reserved too, so choose another name for your own partitions. TIR rejects a reserved name with "is reserved by the platform and cannot be redefined". ## Find the partition table 1. Under **Training Cluster** in the TIR sidebar, open the cluster. 2. Open the **Cluster Overview** tab and scroll to **Slurm Partition Info**. The table lists your partitions and the platform partition that covers every node, as the controller reports them. The columns are: | Column | What it shows | | ----------------- | ---------------------------------------------------------------------------- | | **PARTITION** | The name. An asterisk marks the default partition. | | **NODES** | The number of nodes in the partition. | | **GPU TYPE** | Left blank. The **Nodes** tab shows each node's GPU model. | | **AVAIL / TOTAL** | Not live availability. Run `sinfo` on the login node for free nodes. | | **DEFAULT TIME** | The time a job gets when it leaves out `--time`. | | **MAX TIME** | The longest a job may request. The infinity symbol means unlimited. | | **STATE** | The partition's state. **DISABLED** marks one of yours with **Enabled** off. | | **ACTIONS** | **Managed** for a platform partition. **Edit** and **Delete** for your own. | A **not live** marker beside a name means the partition is in your configuration but missing from what the controller reports. An empty table reads **No partition data available.** To see the nodes that are free in each partition, run this on the login node: ```bash sinfo -o "%20P %5a %10l %6D %6t %N" ``` The **Nodes** tab shows each node's real GPU model and health. See [Check cluster and node health](/tir/tir-slurm-cluster/how-to/check-cluster-and-node-health/). ## Add a partition Add one partition from the table, or several at once from the **Slurm Configuration** dialog. A cluster can have up to 20 partitions of your own, on top of the platform's partitions. To add one partition: 1. In **Slurm Partition Info**, select **Add Partition**. 2. Fill in the fields described in the next section. 3. Select **Save Partition**. The table refreshes, and the console confirms with "Partition added. The cluster is reloading slurm.conf." To add several partitions, or to change partitions and `slurm.conf` together: 1. Select **Actions**, then **Slurm Configuration**, then the **Partitions** tab. 2. Select **Add Partition** for each new partition. Each opens its own set of fields. 3. Select **Apply Configuration**. This tab lists the partitions you already have as read-only. To edit one, use the partition table. ### Fill in the partition fields | Field | What to enter | | ----------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Enabled** | On by default. Turn it off to keep the definition saved but left out of `slurm.conf`. The table then shows **DISABLED**. | | **Name** | Starts with a letter or digit, then letters, digits, hyphens, and underscores, up to 64 characters. It must be unique among your partitions regardless of letter case, and differ from the reserved names. | | **State** | `UP` accepts jobs and runs them. `DOWN` accepts submissions and keeps them queued, and running jobs continue. `DRAIN` refuses new submissions and lets queued jobs run. `INACTIVE` refuses new submissions and keeps queued jobs waiting. The default is `UP`. | | **Max time** | The longest a job may request, in Slurm time format: minutes, `MM:SS`, `HH:MM:SS`, `days-HH:MM:SS`, or `UNLIMITED`. The default is `UNLIMITED`. Set a limit so a hung job gives its GPUs back when the time runs out. | | **Default time** | The time a job gets when it leaves out `--time`. Leave it empty to use **Max time**. | | **Nodes in this partition** | At least one node. See the node list rules below. | | **Extra partition options** | One `Key=Value` per line, added to the partition line. See the next section. | | **Make this the default partition** | Off by default. Only one partition can be the default. | The **Nodes** list takes the real Slurm node names as a comma, space, or newline separated list, with bracket ranges, or the word `ALL` for every node. When a partition covers every node, the field shows the nodes spelled out, and editing that text turns it into an explicit list: ```text slinky-0 slinky-0, slinky-1, slinky-2 slinky-[0-3] slinky-[0-1,5,8-10] ALL ``` Name at least one node, and only nodes the cluster has. Otherwise TIR refuses the partition with a message that names it. A node that is yet to register with Slurm shows a warning that the partition will be short of nodes until it does. You can still save, and the node joins the partition when it registers. ### Set extra partition options Extra options accept any directive that is valid on a PartitionName= line, beyond what the structured fields cover. For example: ```ini DefCpuPerGPU=8 DefMemPerGPU=32768 MaxNodes=4 MaxCPUsPerNode=180 OverSubscribe=NO PriorityTier=10 AllowGroups=research ``` PartitionName, `Nodes`, and `Default` come from the **Name** field, the **Nodes** field, and the default switch. If you set them as extra options, TIR refuses them with "`` is derived by the platform and cannot be set on a partition". Set **State**, **Max time**, and **Default time** in their own fields too, and keep extra options for directives that lack a field of their own. Extra options are added after the fields. Repeating a field there with the same capitalization, such as MaxTime=, overrides it. With a different capitalization, such as `maxtime=`, both end up on the partition line, and Slurm may then reject the line or resolve it unpredictably. The values follow the same rules as extra `slurm.conf` lines; see [Change the Slurm configuration](/tir/tir-slurm-cluster/how-to/change-the-slurm-configuration/). To restrict a partition to one group, add AllowGroups= followed by the group name to its extra options, naming a group that already exists in Login User Management. The restriction comes from AllowGroups=; membership in a group on its own leaves the partition open. A QoS named in the options must already exist in the accounting database, because TIR does not create QoS entries. See [Manage login users](/tir/tir-slurm-cluster/how-to/manage-login-users/). ### Example: a short-jobs queue To give interactive work and quick tests a queue of their own, apart from 24-hour training runs, add a partition with these values: | Field | Value | | ------------------------------- | -------------- | | Name | `debug` | | State | `UP` | | Max time | `01:00:00` | | Default time | `00:30:00` | | Nodes in this partition | `slinky-0` | | Extra partition options | DefCpuPerGPU=8 | | Make this the default partition | off | Then submit to it from the login node: ```bash srun --partition=debug --gres=gpu:1 --pty bash ``` ### Choose the default partition The default partition is where jobs land when they leave out `--partition`. To make one of yours the default, turn on **Make this the default partition**. If that partition is disabled, has an empty name, or has lost all its nodes, the default goes back to `all`. ## Edit a partition 1. In the partition table, open the row menu of the partition and select **Edit**. 2. Change the fields and select **Save Partition**. The console confirms with "Partition updated. The cluster is reloading slurm.conf." The **Name** field is read-only while you edit. To rename a partition, delete it and add a new one with the name you want. TIR checks only the partition you open, so you can edit one partition even while another still names nodes that were removed. A partition holds the nodes you named. After you scale a cluster up, edit the partition to add the new nodes; until then they are reachable through the `all` and `slinky` partitions. If every node a partition named is later removed, for example by a scale-down, the table shows **not live** and jobs aimed at it stay pending. Edit it and name nodes that still exist. ## Delete a partition Queued jobs that target a deleted partition stay pending, because the partition they wait for is gone. Before you delete, check the **Jobs** tab or run `squeue -p `, and either let those jobs finish or move them with `scontrol update job= partition=`. 1. In the partition table, open the row menu of the partition and select **Delete**. 2. Read the dialog. Jobs already running keep running, and queued jobs that target the partition stay pending. 3. Tick the acknowledgement and select **Delete**. The console confirms with "Partition `` removed. The cluster is reloading slurm.conf." Here `` is the partition you deleted. To pause new work and keep the partition, edit it and set **State** to `DRAIN`, which lets queued jobs finish, or to `DOWN`, which keeps them queued. You can also turn **Enabled** off. All three are reversible. ## Check the partition from the login node ```bash ssh root@ ``` Replace `` with the cluster's address from **Connection Details**. Then run: ```bash sinfo sinfo -o "%20P %5a %10l %10L %6D %N" scontrol show partition debug ``` The first command lists partitions, states, and node counts. The second adds availability, maximum and default time, and node names. The third shows everything Slurm knows about one partition; use your partition's name in place of `debug`. Running jobs keep running through partition changes. **Clone Cluster** copies the Slurm configuration apart from partitions, because they name the original cluster's nodes; add them again once the new cluster is **Running**. See [Clone a cluster](/tir/tir-slurm-cluster/how-to/clone-a-cluster/). To run your own scripts around every job, see [Add prolog and epilog scripts](/tir/tir-slurm-cluster/how-to/add-prolog-and-epilog-scripts/). For a job that stays pending or a partition that stopped scheduling, see [Troubleshooting jobs on TIR Slurm Cluster](/tir/tir-slurm-cluster/troubleshooting/jobs/).