--- title: Manage Slurm Partitions sidebar_label: Partitions --- import { Layers, Plus, Edit3, Trash2, Shield, List } from 'react-feather'; # Manage Slurm Partitions A partition is a named queue over a chosen set of nodes, with its own time limits and defaults. Every TIR Slurm Cluster comes with platform partitions covering all nodes; you can add up to **20** of your own from the console, without SSH. }, { href: '#platform-partitions', label: 'Platform partitions', icon: }, { href: '#add-a-partition', label: 'Add a partition', icon: }, { href: '#partition-fields', label: 'Partition fields', icon: }, { href: '#edit-a-partition', label: 'Edit a partition', icon: }, { href: '#delete-a-partition', label: 'Delete a partition', icon: , destructive: true }, ]} /> --- ## The Partition Table 1. Open the cluster from **Training Cluster** in the TIR sidebar. 2. Go to the **Cluster Overview** tab and scroll to **Slurm Partition Info**. > Partitions you define are written to `slurm.conf`. The platform partition covering every node is > managed for you. | Column | What it shows | |--------|---------------| | **PARTITION** | The partition name. A `*` marks the cluster's default partition. A `not live` pill means the partition is in your configuration but slurmctld is not reporting it | | **NODES** | How many nodes the partition holds, as reported by Slurm | | **GPU TYPE** | Not currently populated — see the note below | | **AVAIL / TOTAL** | Not currently populated — see the note below | | **DEFAULT TIME** | Time limit applied when a job does not ask for one | | **MAX TIME** | Longest time a job may request. `∞` means `UNLIMITED` | | **STATE** | `UP`, `DOWN`, `DRAIN` or `INACTIVE`. `DISABLED` appears for one of your partitions that is switched off | | **ACTIONS** | ⋮ menu with **Edit** and **Delete** for your own partitions. Platform partitions show the text `Managed` and cannot be changed | Under the table, a note records where the data comes from — `Sourced from slurmrestd` — so the table reflects what the live controller reports, not just what you configured. Empty table? It reads **No partition data available.** :::warning GPU TYPE and AVAIL / TOTAL are not populated These two columns are not fed by the partition data the console receives: **GPU TYPE** renders blank and **AVAIL / TOTAL** shows a placeholder rather than real availability. Do not rely on them. For authoritative per-partition availability, use the login node: ```bash sinfo -o "%20P %5a %10l %6D %6t %N" ``` The [Nodes tab](/docs/tir/SlurmCluster/manage/nodes) shows real per-node GPU models and health. ::: :::info The tab depends on your region On most regions the partition table lives on the **Cluster Overview** tab. In some regions the cluster has a dedicated **Partitions** tab holding the same section. Either way the cluster must be **Running** — otherwise the panel reads *"Partitions are available when the cluster is running."* ::: --- ## Platform Partitions Every cluster gets partitions it does not manage. They appear in the table with `Managed` in the Actions column. | Partition | What it is | |-----------|------------| | `all` | Covers every node in the cluster, `State=UP`, `MaxTime=UNLIMITED`, and is the **default partition** unless you nominate one of your own | | `slinky` | The GPU node group's own partition, created automatically alongside the node group | | `cpu-workers` | The CPU node group's partition, on clusters that have one | | `llm`, `asr`, `vision`, `tts` | Region-specific platform partitions on some clusters | This is why your first `sinfo` lists partitions you did not create — `all` and `slinky` at minimum, plus `cpu-workers` and the region-specific ones where they apply. All of them show `Managed` in the partition table. These names are **reserved**. You cannot create or redefine a partition called `all`, `slinky`, `cpu-workers`, `llm`, `asr`, `vision`, `tts` or `default` — the save fails with `` `` is reserved by the platform and cannot be redefined ``. :::note `default` is reserved for a reason `PartitionName=DEFAULT` is a *template* directive in `slurm.conf`, not a partition. Allowing it would silently change the meaning of every partition line after it, so TIR blocks the name. ::: You **may** point the cluster default back at `all` — that is what happens automatically whenever your nominated default is not usable. --- ## Add a Partition Partitions can only be created on a **Running** cluster, because a partition names specific nodes and those do not exist until the cluster is up. This is why the create-cluster form has no Partitions tab. ### From the partition table 1. Open the cluster → **Cluster Overview** → **Slurm Partition Info**. 2. Click **Add Partition**. The **Add Partition** dialog opens with a single partition form. 3. Fill in the fields ([reference below](#partition-fields)). 4. Click **Save Partition**. You get the toast **"Partition added. The cluster is reloading slurm.conf."** and the table refreshes. ### From the Slurm Configuration dialog Use this when you want to add several partitions, or change partitions and `slurm.conf` together. 1. **Actions** → **Slurm Configuration** → **Partitions** tab. 2. Click **Add Partition** for each new partition — each appears as a card headed **NEW PARTITION**. 3. Click **Apply Configuration**. Partitions you already have are listed here under **Already configured** as read-only chips; click a chip to jump to the partition table where it can be edited. With nothing configured the tab shows *"No new partition yet. Use Add Partition to define one."* ### Example: a short-jobs queue To give interactive work and smoke tests their own queue with a tight time limit, so they are never stuck behind a 24-hour training run: | Field | Value | |-------|-------| | Name | `debug` | | State | `UP` | | Max time | `01:00:00` | | Default time | `00:30:00` | | Nodes in this partition | `slinky-0` | | Extra partition options | `DefCpuPerGPU=8` | | Make this the default partition | off | Then: ```bash srun --partition=debug --gres=gpu:1 --pty bash ``` --- ## Partition Fields | Field | Required | Default | Notes | |-------|----------|---------|-------| | **Enabled** (toggle) | — | On | Off keeps the definition but writes no line to `slurm.conf`. The table shows `DISABLED` | | **Name** | Yes | — | Letters, digits, `-` and `_`; must start with a letter or digit; up to 64 characters. Case-insensitively unique. Cannot be a reserved name | | **State** | Yes | `UP` | `UP` accepts and runs jobs · `DOWN` accepts nothing · `DRAIN` finishes running jobs but accepts no new ones · `INACTIVE` accepts submissions but never schedules them | | **Max time** | No | `UNLIMITED` | Longest a job may request. Slurm time format: `minutes`, `MM:SS`, `HH:MM:SS`, `days-HH:MM:SS`, or `UNLIMITED` | | **Default time** | No | empty | Applied when a job omits `--time`. Leave empty to inherit `Max time` | | **Nodes in this partition** | Yes | all nodes | See [node syntax](#node-syntax) | | **Extra partition options** | No | empty | One `Key=Value` per line, appended to the partition line | | **Make this the default partition** | No | Off | Only one partition can hold it | ### Node syntax Nodes are named `slinky-0`, `slinky-1`, … for GPU workers, and `cpu-workers-0`, `cpu-workers-1`, … on clusters with a CPU node group. The field accepts comma-, space- or newline-separated names and bracket ranges: ``` slinky-0 slinky-0, slinky-1, slinky-2 slinky-[0-3] slinky-[0-1,5,8-10] slinky-[0-3], cpu-workers-[0-4] ALL ``` `ALL` means every node in the cluster. The field displays the cluster's nodes spelled out when a partition covers all of them; editing that text converts it into an explicit list. A count pill next to the label shows how many nodes you have selected. Two warnings can appear: | Warning | Meaning | |---------|---------| | *"The cluster did not report a node list for this partition. Type the nodes it should hold."* | The box is empty — you must name at least one node | | *"`` has not registered with Slurm yet, so the partition will be short of nodes until it does."* | A valid cluster node that slurmctld has not yet seen. You can still save; the node joins when it registers | ### Extra partition options Anything valid on a `PartitionName=` line that TIR does not already control. Common ones: ```ini DefCpuPerGPU=8 DefMemPerGPU=32768 MaxNodes=4 MaxCPUsPerNode=180 OverSubscribe=NO PriorityTier=10 AllowGroups=research ``` You **cannot** set `PartitionName`, `Nodes` or `Default` here — those come from the Name field, the Nodes field and the default toggle. Attempting one fails with `` `` is derived by the platform and cannot be set on a partition ``. Values follow the same rules as [extra `slurm.conf`](/docs/tir/SlurmCluster/slurm-configuration/slurm-conf#syntax-rules) — no spaces, no `;`, no `#`. :::warning Do not restate a field here under a different capitalisation Extra options are appended after the structured fields. Repeating a field with **exactly** the same capitalisation (`MaxTime=…`) overrides it; repeating it with different capitalisation (`maxtime=…`) puts **both** on the same partition line, which Slurm may reject or resolve unpredictably. Set **State**, **Max time** and **Default time** in their own fields, and keep extra options to directives that have no field of their own. ::: :::warning `AllowGroups` needs the group to exist Restricting a partition to a POSIX group only works if that group exists on the nodes. Create it with [Login User Management](/docs/tir/SlurmCluster/connect/login-user-management) first. Similarly, a `QOS=` or `AllowQos=` option must name a QoS that already exists in the accounting database — TIR does not create QoS entries for you. ::: ### The default partition The partition marked `*` in the table is where jobs land when they omit `--partition`. Nominate one of yours with **Make this the default partition**. The nomination is dropped automatically — and `all` keeps the default — if the partition would not write a line to `slurm.conf` at all. The dialog warns you: > A partition that is unnamed, disabled or without nodes writes no line to `slurm.conf`, so it cannot > hold the default. The platform partition keeps it. --- ## Edit a Partition 1. In the partition table, open the ⋮ menu on the row → **Edit**. 2. The **Edit Partition** dialog opens, headed `Partition · `. 3. Change the fields and click **Save Partition**. Toast: **"Partition updated. The cluster is reloading slurm.conf."** :::info The name is locked when editing Slurm identifies a partition by name, so the **Name** field is read-only while editing. To rename, delete the partition and add a new one with the name you want. ::: Only the partition you opened is validated, so a stale sibling — for example one whose nodes were removed by a scale-down — can never block you from editing a different partition. ### A partition that stopped working after a scale-down If every node a partition named has been removed, the partition **silently stops being written to `slurm.conf`** and the table shows it as `not live`. That is deliberate: emitting a partition line with nodes that no longer exist would make slurmctld reject the whole configuration. Fix it by editing the partition and pointing it at nodes that exist. --- ## Delete a Partition 1. In the partition table, open the ⋮ menu on the row → **Delete**. 2. The **Delete Partition** dialog explains what happens: > Partition **<name>** is being removed from `slurm.conf` and slurmctld reloads its > configuration. > > Jobs already running keep running. Queued jobs targeting it will not be scheduled. 3. Tick the acknowledgement — *"Deleting a partition may take a few moments to complete while the cluster reloads its configuration."* 4. Click **Delete**. Toast: **"Partition <name> removed. The cluster is reloading slurm.conf."** :::danger Queued jobs targeting a deleted partition will not run They are not cancelled — they simply stay pending against a partition that no longer exists. Before deleting, check the [Jobs tab](/docs/tir/SlurmCluster/manage/jobs) or run `squeue -p `, and either let them finish or move them with `scontrol update job= partition=`. ::: If you only want to stop scheduling temporarily, **Edit** the partition and set **State** to `DRAIN` (finish what is running, accept nothing new) or switch **Enabled** off — both are reversible, unlike delete. --- ## Verify from the Command Line ```bash ssh root@ ``` ```bash sinfo # partitions, states, node counts sinfo -o "%20P %5a %10l %10L %6D %N" # partition, avail, max/default time, nodes scontrol show partition debug # everything Slurm knows about one partition ``` --- ## Rules Summary | Rule | Detail | |------|--------| | When | Only on a **Running** cluster; never at create time | | How many | At most **20** of your own partitions | | Names | `^[A-Za-z0-9][A-Za-z0-9_-]{0,63}$`, unique case-insensitively, not reserved | | Nodes | At least one, and every name must be a real cluster node (or `ALL`) | | Renaming | Not possible in place — delete and re-add | | Disabled | Definition kept, no `slurm.conf` line, shown as `DISABLED` | | Nodes gone | Partition silently stops rendering, shown as `not live`, until repointed | | Default | One at a time; falls back to `all` if the nominee cannot render | | Reserved options | `PartitionName`, `Nodes`, `Default` cannot be set as extra options | | Cloning | **Clone Cluster** does not carry partitions — they name the original cluster's nodes | | Running jobs | Never interrupted by a partition change | --- ## Related Resources - [Slurm Configuration overview](/docs/tir/SlurmCluster/slurm-configuration/) - [Extra slurm.conf settings](/docs/tir/SlurmCluster/slurm-configuration/slurm-conf) - [Prolog and epilog scripts](/docs/tir/SlurmCluster/slurm-configuration/prolog-epilog) - [Cluster Overview tab](/docs/tir/SlurmCluster/manage/overview-tab) - [Jobs tab](/docs/tir/SlurmCluster/manage/jobs) - [Slurm partition configuration (SchedMD)](https://slurm.schedmd.com/slurm.conf.html#SECTION_PARTITION-CONFIGURATION)