Skip to main content

Manage Slurm Partitions

A partition is a named queue over a chosen set of nodes, with its own time limits and defaults. Every TIR Slurm Cluster comes with platform partitions covering all nodes; you can add up to 20 of your own from the console, without SSH.

The partition tablePlatform partitionsAdd a partitionPartition fieldsEdit a partitionDelete a partition

The Partition Table​

  1. Open the cluster from Training Cluster in the TIR sidebar.
  2. Go to the Cluster Overview tab and scroll to Slurm Partition Info.

Partitions you define are written to slurm.conf. The platform partition covering every node is managed for you.

ColumnWhat it shows
PARTITIONThe partition name. A * marks the cluster's default partition. A not live pill means the partition is in your configuration but slurmctld is not reporting it
NODESHow many nodes the partition holds, as reported by Slurm
GPU TYPENot currently populated — see the note below
AVAIL / TOTALNot currently populated — see the note below
DEFAULT TIMETime limit applied when a job does not ask for one
MAX TIMELongest time a job may request. ∞ means UNLIMITED
STATEUP, DOWN, DRAIN or INACTIVE. DISABLED appears for one of your partitions that is switched off
ACTIONS⋮ menu with Edit and Delete for your own partitions. Platform partitions show the text Managed and cannot be changed

Under the table, a note records where the data comes from — Sourced from slurmrestd — so the table reflects what the live controller reports, not just what you configured.

Empty table? It reads No partition data available.

GPU TYPE and AVAIL / TOTAL are not populated

These two columns are not fed by the partition data the console receives: GPU TYPE renders blank and AVAIL / TOTAL shows a placeholder rather than real availability. Do not rely on them.

For authoritative per-partition availability, use the login node:

sinfo -o "%20P %5a %10l %6D %6t %N"

The Nodes tab shows real per-node GPU models and health.

The tab depends on your region

On most regions the partition table lives on the Cluster Overview tab. In some regions the cluster has a dedicated Partitions tab holding the same section. Either way the cluster must be Running — otherwise the panel reads "Partitions are available when the cluster is running."

Platform Partitions​

Every cluster gets partitions it does not manage. They appear in the table with Managed in the Actions column.

PartitionWhat it is
allCovers every node in the cluster, State=UP, MaxTime=UNLIMITED, and is the default partition unless you nominate one of your own
slinkyThe GPU node group's own partition, created automatically alongside the node group
cpu-workersThe CPU node group's partition, on clusters that have one
llm, asr, vision, ttsRegion-specific platform partitions on some clusters

This is why your first sinfo lists partitions you did not create — all and slinky at minimum, plus cpu-workers and the region-specific ones where they apply. All of them show Managed in the partition table.

These names are reserved. You cannot create or redefine a partition called all, slinky, cpu-workers, llm, asr, vision, tts or default — the save fails with `<name>` is reserved by the platform and cannot be redefined.

default is reserved for a reason

PartitionName=DEFAULT is a template directive in slurm.conf, not a partition. Allowing it would silently change the meaning of every partition line after it, so TIR blocks the name.

You may point the cluster default back at all — that is what happens automatically whenever your nominated default is not usable.

Add a Partition​

Partitions can only be created on a Running cluster, because a partition names specific nodes and those do not exist until the cluster is up. This is why the create-cluster form has no Partitions tab.

From the partition table​

  1. Open the cluster → Cluster Overview → Slurm Partition Info.
  2. Click Add Partition. The Add Partition dialog opens with a single partition form.
  3. Fill in the fields (reference below).
  4. Click Save Partition.

You get the toast "Partition added. The cluster is reloading slurm.conf." and the table refreshes.

From the Slurm Configuration dialog​

Use this when you want to add several partitions, or change partitions and slurm.conf together.

  1. Actions → Slurm Configuration → Partitions tab.
  2. Click Add Partition for each new partition — each appears as a card headed NEW PARTITION.
  3. Click Apply Configuration.

Partitions you already have are listed here under Already configured as read-only chips; click a chip to jump to the partition table where it can be edited. With nothing configured the tab shows "No new partition yet. Use Add Partition to define one."

Example: a short-jobs queue​

To give interactive work and smoke tests their own queue with a tight time limit, so they are never stuck behind a 24-hour training run:

FieldValue
Namedebug
StateUP
Max time01:00:00
Default time00:30:00
Nodes in this partitionslinky-0
Extra partition optionsDefCpuPerGPU=8
Make this the default partitionoff

Then:

srun --partition=debug --gres=gpu:1 --pty bash

Partition Fields​

FieldRequiredDefaultNotes
Enabled (toggle)—OnOff keeps the definition but writes no line to slurm.conf. The table shows DISABLED
NameYes—Letters, digits, - and _; must start with a letter or digit; up to 64 characters. Case-insensitively unique. Cannot be a reserved name
StateYesUPUP accepts and runs jobs · DOWN accepts nothing · DRAIN finishes running jobs but accepts no new ones · INACTIVE accepts submissions but never schedules them
Max timeNoUNLIMITEDLongest a job may request. Slurm time format: minutes, MM:SS, HH:MM:SS, days-HH:MM:SS, or UNLIMITED
Default timeNoemptyApplied when a job omits --time. Leave empty to inherit Max time
Nodes in this partitionYesall nodesSee node syntax
Extra partition optionsNoemptyOne Key=Value per line, appended to the partition line
Make this the default partitionNoOffOnly one partition can hold it

Node syntax​

Nodes are named slinky-0, slinky-1, … for GPU workers, and cpu-workers-0, cpu-workers-1, … on clusters with a CPU node group. The field accepts comma-, space- or newline-separated names and bracket ranges:

slinky-0
slinky-0, slinky-1, slinky-2
slinky-[0-3]
slinky-[0-1,5,8-10]
slinky-[0-3], cpu-workers-[0-4]
ALL

ALL means every node in the cluster. The field displays the cluster's nodes spelled out when a partition covers all of them; editing that text converts it into an explicit list.

A count pill next to the label shows how many nodes you have selected. Two warnings can appear:

WarningMeaning
"The cluster did not report a node list for this partition. Type the nodes it should hold."The box is empty — you must name at least one node
"<node> has not registered with Slurm yet, so the partition will be short of nodes until it does."A valid cluster node that slurmctld has not yet seen. You can still save; the node joins when it registers

Extra partition options​

Anything valid on a PartitionName= line that TIR does not already control. Common ones:

DefCpuPerGPU=8
DefMemPerGPU=32768
MaxNodes=4
MaxCPUsPerNode=180
OverSubscribe=NO
PriorityTier=10
AllowGroups=research

You cannot set PartitionName, Nodes or Default here — those come from the Name field, the Nodes field and the default toggle. Attempting one fails with `<Key>` is derived by the platform and cannot be set on a partition.

Values follow the same rules as extra slurm.conf — no spaces, no ;, no #.

Do not restate a field here under a different capitalisation

Extra options are appended after the structured fields. Repeating a field with exactly the same capitalisation (MaxTime=…) overrides it; repeating it with different capitalisation (maxtime=…) puts both on the same partition line, which Slurm may reject or resolve unpredictably.

Set State, Max time and Default time in their own fields, and keep extra options to directives that have no field of their own.

AllowGroups needs the group to exist

Restricting a partition to a POSIX group only works if that group exists on the nodes. Create it with Login User Management first. Similarly, a QOS= or AllowQos= option must name a QoS that already exists in the accounting database — TIR does not create QoS entries for you.

The default partition​

The partition marked * in the table is where jobs land when they omit --partition. Nominate one of yours with Make this the default partition.

The nomination is dropped automatically — and all keeps the default — if the partition would not write a line to slurm.conf at all. The dialog warns you:

A partition that is unnamed, disabled or without nodes writes no line to slurm.conf, so it cannot hold the default. The platform partition keeps it.

Edit a Partition​

  1. In the partition table, open the ⋮ menu on the row → Edit.
  2. The Edit Partition dialog opens, headed Partition · <name>.
  3. Change the fields and click Save Partition.

Toast: "Partition updated. The cluster is reloading slurm.conf."

The name is locked when editing

Slurm identifies a partition by name, so the Name field is read-only while editing. To rename, delete the partition and add a new one with the name you want.

Only the partition you opened is validated, so a stale sibling — for example one whose nodes were removed by a scale-down — can never block you from editing a different partition.

A partition that stopped working after a scale-down​

If every node a partition named has been removed, the partition silently stops being written to slurm.conf and the table shows it as not live. That is deliberate: emitting a partition line with nodes that no longer exist would make slurmctld reject the whole configuration.

Fix it by editing the partition and pointing it at nodes that exist.

Delete a Partition​

  1. In the partition table, open the ⋮ menu on the row → Delete.

  2. The Delete Partition dialog explains what happens:

    Partition <name> is being removed from slurm.conf and slurmctld reloads its configuration.

    Jobs already running keep running. Queued jobs targeting it will not be scheduled.

  3. Tick the acknowledgement — "Deleting a partition may take a few moments to complete while the cluster reloads its configuration."

  4. Click Delete.

Toast: "Partition <name> removed. The cluster is reloading slurm.conf."

Queued jobs targeting a deleted partition will not run

They are not cancelled — they simply stay pending against a partition that no longer exists. Before deleting, check the Jobs tab or run squeue -p <name>, and either let them finish or move them with scontrol update job=<id> partition=<other>.

If you only want to stop scheduling temporarily, Edit the partition and set State to DRAIN (finish what is running, accept nothing new) or switch Enabled off — both are reversible, unlike delete.

Verify from the Command Line​

ssh root@<cluster-ip>
sinfo                                     # partitions, states, node counts
sinfo -o "%20P %5a %10l %10L %6D %N"      # partition, avail, max/default time, nodes
scontrol show partition debug             # everything Slurm knows about one partition

Rules Summary​

RuleDetail
WhenOnly on a Running cluster; never at create time
How manyAt most 20 of your own partitions
Names^[A-Za-z0-9][A-Za-z0-9_-]{0,63}$, unique case-insensitively, not reserved
NodesAt least one, and every name must be a real cluster node (or ALL)
RenamingNot possible in place — delete and re-add
DisabledDefinition kept, no slurm.conf line, shown as DISABLED
Nodes gonePartition silently stops rendering, shown as not live, until repointed
DefaultOne at a time; falls back to all if the nominee cannot render
Reserved optionsPartitionName, Nodes, Default cannot be set as extra options
CloningClone Cluster does not carry partitions — they name the original cluster's nodes
Running jobsNever interrupted by a partition change
Last updated on September 10, 2026.