Manage partitions

Split your cluster into partitions, your own named queues over chosen nodes, each with its own state, time limits, members, and extra options. A common first one is a short-jobs queue, so quick tests get a queue of their own rather than waiting behind a long training run.

Before you begin

• A cluster in the Running state. A partition names specific nodes, and those nodes exist once the cluster is up, so you add partitions after creating the cluster. See Cluster statuses and actions.
• Update permission on Training Clusters in the project.
• The Slurm names of the nodes you want in the partition: slinky-0, slinky-1, and so on up to slinky-(N-1) for a cluster of N nodes. The Nodes tab shows each node’s Slurm name beside its machine identifier.

Know the platform partitions

Every cluster comes with two partitions that the platform manages for you:

• all covers every node. Jobs that leave out --partition land on it, unless you make another partition the default.
• slinky covers the GPU worker nodes.

Your first sinfo lists both of them. In the partition table they read Managed, and you cannot edit or delete them.

The names llm, asr, vision, tts, and default are reserved too, so choose another name for your own partitions. TIR rejects a reserved name with “is reserved by the platform and cannot be redefined”.

Find the partition table

1. Under Training Cluster in the TIR sidebar, open the cluster.
2. Open the Cluster Overview tab and scroll to Slurm Partition Info.

The table lists your partitions and the platform partition that covers every node, as the controller reports them. The columns are:

Column	What it shows
PARTITION	The name. An asterisk marks the default partition.
NODES	The number of nodes in the partition.
GPU TYPE	Left blank. The Nodes tab shows each node’s GPU model.
AVAIL / TOTAL	Not live availability. Run sinfo on the login node for free nodes.
DEFAULT TIME	The time a job gets when it leaves out --time.
MAX TIME	The longest a job may request. The infinity symbol means unlimited.
STATE	The partition’s state. DISABLED marks one of yours with Enabled off.
ACTIONS	Managed for a platform partition. Edit and Delete for your own.

A not live marker beside a name means the partition is in your configuration but missing from what the controller reports. An empty table reads No partition data available.

To see the nodes that are free in each partition, run this on the login node:

sinfo -o "%20P %5a %10l %6D %6t %N"

The Nodes tab shows each node’s real GPU model and health. See Check cluster and node health.

Add a partition

Add one partition from the table, or several at once from the Slurm Configuration dialog. A cluster can have up to 20 partitions of your own, on top of the platform’s partitions.

To add one partition:

1. In Slurm Partition Info, select Add Partition.
2. Fill in the fields described in the next section.
3. Select Save Partition. The table refreshes, and the console confirms with “Partition added. The cluster is reloading slurm.conf.”

To add several partitions, or to change partitions and slurm.conf together:

1. Select Actions, then Slurm Configuration, then the Partitions tab.
2. Select Add Partition for each new partition. Each opens its own set of fields.
3. Select Apply Configuration.

This tab lists the partitions you already have as read-only. To edit one, use the partition table.

Fill in the partition fields

Field	What to enter
Enabled	On by default. Turn it off to keep the definition saved but left out of slurm.conf. The table then shows DISABLED.
Name	Starts with a letter or digit, then letters, digits, hyphens, and underscores, up to 64 characters. It must be unique among your partitions regardless of letter case, and differ from the reserved names.
State	UP accepts jobs and runs them. DOWN accepts submissions and keeps them queued, and running jobs continue. DRAIN refuses new submissions and lets queued jobs run. INACTIVE refuses new submissions and keeps queued jobs waiting. The default is UP.
Max time	The longest a job may request, in Slurm time format: minutes, MM:SS, HH:MM:SS, days-HH:MM:SS, or UNLIMITED. The default is UNLIMITED. Set a limit so a hung job gives its GPUs back when the time runs out.
Default time	The time a job gets when it leaves out --time. Leave it empty to use Max time.
Nodes in this partition	At least one node. See the node list rules below.
Extra partition options	One Key=Value per line, added to the partition line. See the next section.
Make this the default partition	Off by default. Only one partition can be the default.

The Nodes list takes the real Slurm node names as a comma, space, or newline separated list, with bracket ranges, or the word ALL for every node. When a partition covers every node, the field shows the nodes spelled out, and editing that text turns it into an explicit list:

slinky-0
slinky-0, slinky-1, slinky-2
slinky-[0-3]
slinky-[0-1,5,8-10]
ALL

Name at least one node, and only nodes the cluster has. Otherwise TIR refuses the partition with a message that names it. A node that is yet to register with Slurm shows a warning that the partition will be short of nodes until it does. You can still save, and the node joins the partition when it registers.

Set extra partition options

Extra options accept any directive that is valid on a PartitionName= line, beyond what the structured fields cover. For example:

DefCpuPerGPU=8
DefMemPerGPU=32768
MaxNodes=4
MaxCPUsPerNode=180
OverSubscribe=NO
PriorityTier=10
AllowGroups=research

PartitionName, Nodes, and Default come from the Name field, the Nodes field, and the default switch. If you set them as extra options, TIR refuses them with “<Key> is derived by the platform and cannot be set on a partition”. Set State, Max time, and Default time in their own fields too, and keep extra options for directives that lack a field of their own. Extra options are added after the fields. Repeating a field there with the same capitalization, such as MaxTime=, overrides it. With a different capitalization, such as maxtime=, both end up on the partition line, and Slurm may then reject the line or resolve it unpredictably.

The values follow the same rules as extra slurm.conf lines; see Change the Slurm configuration.

To restrict a partition to one group, add AllowGroups= followed by the group name to its extra options, naming a group that already exists in Login User Management. The restriction comes from AllowGroups=; membership in a group on its own leaves the partition open. A QoS named in the options must already exist in the accounting database, because TIR does not create QoS entries. See Manage login users.

Example: a short-jobs queue

To give interactive work and quick tests a queue of their own, apart from 24-hour training runs, add a partition with these values:

Field	Value
Name	debug
State	UP
Max time	01:00:00
Default time	00:30:00
Nodes in this partition	slinky-0
Extra partition options	DefCpuPerGPU=8
Make this the default partition	off

Then submit to it from the login node:

srun --partition=debug --gres=gpu:1 --pty bash

Choose the default partition

The default partition is where jobs land when they leave out --partition. To make one of yours the default, turn on Make this the default partition. If that partition is disabled, has an empty name, or has lost all its nodes, the default goes back to all.

Edit a partition

1. In the partition table, open the row menu of the partition and select Edit.
2. Change the fields and select Save Partition. The console confirms with “Partition updated. The cluster is reloading slurm.conf.”

The Name field is read-only while you edit. To rename a partition, delete it and add a new one with the name you want.

TIR checks only the partition you open, so you can edit one partition even while another still names nodes that were removed.

A partition holds the nodes you named. After you scale a cluster up, edit the partition to add the new nodes; until then they are reachable through the all and slinky partitions. If every node a partition named is later removed, for example by a scale-down, the table shows not live and jobs aimed at it stay pending. Edit it and name nodes that still exist.

Delete a partition

Queued jobs that target a deleted partition stay pending, because the partition they wait for is gone. Before you delete, check the Jobs tab or run squeue -p <name>, and either let those jobs finish or move them with scontrol update job=<id> partition=<other>.

1. In the partition table, open the row menu of the partition and select Delete.
2. Read the dialog. Jobs already running keep running, and queued jobs that target the partition stay pending.
3. Tick the acknowledgement and select Delete. The console confirms with “Partition <name> removed. The cluster is reloading slurm.conf.” Here <name> is the partition you deleted.

To pause new work and keep the partition, edit it and set State to DRAIN, which lets queued jobs finish, or to DOWN, which keeps them queued. You can also turn Enabled off. All three are reversible.

Check the partition from the login node

ssh root@<cluster-ip>

Replace <cluster-ip> with the cluster’s address from Connection Details. Then run:

sinfo
sinfo -o "%20P %5a %10l %10L %6D %N"
scontrol show partition debug

The first command lists partitions, states, and node counts. The second adds availability, maximum and default time, and node names. The third shows everything Slurm knows about one partition; use your partition’s name in place of debug.

Running jobs keep running through partition changes. Clone Cluster copies the Slurm configuration apart from partitions, because they name the original cluster’s nodes; add them again once the new cluster is Running. See Clone a cluster.

To run your own scripts around every job, see Add prolog and epilog scripts. For a job that stays pending or a partition that stopped scheduling, see Troubleshooting jobs on TIR Slurm Cluster.