Change the Slurm configuration

Add your own Key=Value settings to slurm.conf on every node of a TIR Slurm Cluster, from the console, on the cluster you already have. E2E manages the Slurm installation and keeps the settings that wire the cluster together. Everything else in slurm.conf is yours to set.

The same dialog also holds your partitions and your prolog and epilog scripts; see Manage partitions and Add prolog and epilog scripts.

Before you begin

• A cluster in the Running state. Slurm Configuration is available only while the cluster is Running. See Cluster statuses and actions.
• Update permission on Training Clusters in the project, which puts the action on the Actions menu.

Open the Slurm Configuration dialog

You can open the dialog from three places:

• The row menu of a cluster in the cluster list, under Training Cluster in the TIR sidebar.
• The Actions menu on the cluster’s details page.
• The create form, under Advanced Settings, then Slurm Configuration (Optional), then Configure Slurm.

What you set here applies to slurm.conf on every node of the cluster. The platform manages the cluster name, accounting, and the node and topology wiring, so those settings cannot be overridden.

On a running cluster you can also set partitions. At create time you cannot, because a partition names specific nodes, and those nodes exist only once the cluster is running. You can still pre-load an extra slurm.conf block and all four prolog and epilog scripts at create time, and save them with Save Configuration.

The dialog loads the cluster’s current configuration every time it opens. Closing it with Cancel discards what you typed, and the cluster’s configuration stays as it was.

Add settings to slurm.conf

1. On the slurm.conf tab, find the Extra slurm.conf box. It holds one Key=Value per line.

2. Type your settings. For example:

   # Keep finished jobs visible to squeue for two minutes
   MinJobAge=120
   # Run the prolog at allocation instead of at the first job step
   PrologFlags=Alloc
   # Extra scheduler logging while tuning
   DebugFlags=Backfill

3. Select Apply Configuration. The cluster reloads slurm.conf.

The platform manages the scheduler itself, so it rejects a SchedulerType line, even though the box’s example text includes one.

The box shows the configuration stored on the cluster, so whatever you leave in it is what the cluster keeps. Removing a line removes that setting.

Applying replaces the cluster’s whole stored Slurm configuration in one request: what the dialog holds at that moment, including every partition and every script, becomes the complete configuration. Running jobs keep running, queued jobs are scheduled under the new settings, and prolog and epilog changes take effect on the next job.

On some clusters, applying restarts the controller; newer clusters reconfigure in place while the controller keeps running. The dialog says which before you apply.

If applying the change fails, the stored settings go back to what they were, and the console shows “Failed to update cluster on infrastructure”. If restoring them also fails, support is alerted, and the console shows “Applying this configuration failed and restoring the previous one also failed. Support has been alerted.”

Follow the syntax rules

Each line in the box must pass these checks before TIR applies the change:

• A line is Key=Value, with a key before the = and a value after it.
• A key starts with a letter, then uses only letters, digits, and underscores, up to 64 characters.
• A value is 1 to 512 characters. It can use letters, digits, and the punctuation marks listed after this list. A space, a hash sign, a semicolon, or a backslash is rejected.
• The same key twice is rejected as a duplicate, even when the capital letters differ.
• Lines that start with #, and blank lines, are accepted and ignored. They stay in the box, so you can record why a setting is there, and only the settings themselves reach slurm.conf.
• The whole block stays under 8 KB.

The console reports one problem per block, the first it finds. Fix that line and apply again; the console then names the next problem, if there is one. The messages are:

Message	Cause
“<line> is not a valid Key=Value line”	The line has no =, an empty key, or an empty value.
“<key> is not a valid slurm.conf key”	The key starts with something other than a letter, or uses a character other than letters, digits, and underscore.
“Value for <key> contains characters that are not allowed”	Usually a space, a hash sign, a semicolon, or a backslash in the value.
“<key> is managed by the platform and cannot be overridden”	The key is one the platform reserves.
“<key> is set more than once”	The same key twice, even when the capital letters differ.
A message that the block exceeds the maximum size of 8 KB	The block is too large.

The punctuation marks a value can use are underscore, period, comma, colon, slash, equals, plus, at sign, percent, asterisk, square brackets, braces, parentheses, angle brackets, tilde, caret, exclamation mark, question mark, vertical bar, dollar sign, ampersand, single and double quotes, and hyphen.

For a value that lists several items, use the comma-separated form Slurm accepts, such as DebugFlags=Backfill,Steps. For what each setting means, see the slurm.conf reference from SchedMD for the Slurm version your image ships.

Choose useful settings

These settings are yours to set, and they are often useful on a training cluster:

Setting	Example	What it does
PrologFlags	PrologFlags=Alloc	Runs the worker prolog at allocation instead of at the first job step. Use it when the prolog prepares something a job step depends on
MinJobAge	MinJobAge=120	Keeps finished jobs in squeue and on the Jobs tab longer
DefMemPerGPU	DefMemPerGPU=32768	Sets the memory per GPU that a job gets by default
DefCpuPerGPU	DefCpuPerGPU=8	Sets the CPU cores per GPU that a job gets by default
MaxArraySize	MaxArraySize=10001	Raises the ceiling for job arrays, for example in sweeps
MaxJobCount	MaxJobCount=50000	Raises the queue ceiling for very large sweeps
KillWait	KillWait=60	Gives jobs longer to checkpoint after SIGTERM
UnkillableStepTimeout	UnkillableStepTimeout=180	Allows slow teardown on large multi-GPU steps
SlurmctldDebug, SlurmdDebug	SlurmdDebug=debug2	Raises log detail while you investigate a scheduling problem
DebugFlags	DebugFlags=Backfill	Logs why the backfill scheduler made its choices
MessageTimeout	MessageTimeout=30	Allows slower messages on very large clusters
TmpFS	TmpFS=/tmp	Points Slurm’s --tmp accounting at the right path

Turn SlurmdDebug=debug2 and broad DebugFlags back down once you have what you need, because they fill the Logs tab quickly.

Settings the platform keeps

The platform reserves 46 slurm.conf keys. If you set one, the dialog rejects it with “<key> is managed by the platform and cannot be overridden”.

Group	Keys
Cluster identity and node wiring	ClusterName, NodeName, PartitionName, SlurmctldHost, SlurmctldPort, SlurmctldParameters, SlurmdPort, SlurmctldPidFile, SlurmdPidFile, SlurmdSpoolDir, StateSaveLocation, SlurmUser, TopologyPlugin, Include
Authentication	AuthType, AuthInfo, AuthAltTypes, AuthAltParameters, CredType, JobCredentialPrivateKey, JobCredentialPublicCertificate
Accounting	AccountingStorageType, AccountingStorageHost, AccountingStoragePort, AccountingStorageEnforce
Prolog and epilog hooks	Prolog, Epilog, PrologSlurmctld, EpilogSlurmctld, TaskProlog, TaskEpilog
Scheduling, priority, preemption	SchedulerType, SchedulerParameters, SelectType, MpiDefault, PreemptType, PreemptMode, PreemptExemptTime, PriorityType, PriorityDecayHalfLife, PriorityMaxAge, PriorityWeightQOS, PriorityWeightAge, PriorityWeightFairshare, PriorityWeightJobSize, PriorityWeightPartition

For the settings people most often reach for, this is what to do instead:

You want to set	Do this instead
Prolog, Epilog, PrologSlurmctld, EpilogSlurmctld	Use the Prolog & Epilog tab. Your script is added alongside the platform’s hook. See Add prolog and epilog scripts.
PartitionName	Use the Partitions tab. See Manage partitions.
NodeName	The node list follows the cluster’s node count. Change it with Scale Cluster. See Scale a cluster.
TaskProlog, TaskEpilog	Leave these to the platform, which uses them itself.
SchedulerType, SelectType, Priority and Preempt keys	These are fixed for every cluster. Raise a support request if your jobs need different scheduling.
AccountingStorage keys	Accounting is fully managed, and sacct and sstat already work.

The platform manages gres.conf, cgroup.conf, plugstack.conf, mpi.conf, oci.conf, and job_container.conf, which have no console screen. It manages accounting, including its database, too. On a GPU cluster, the platform also sets GresTypes, SelectTypeParameters, and AccountingStorageTRES; see the TIR Slurm Cluster FAQ. TIR does not create QoS entries, so a partition can name a QoS only if it already exists in the accounting database.

Check that the setting is live

After you apply, connect to the login node and read the setting back:

ssh root@<cluster-ip>

Replace <cluster-ip> with the cluster’s address from Connection Details on the details page. Then read one setting, or the whole live configuration:

scontrol show config | grep -i MinJobAge
scontrol show config

If the old value still shows, wait a few seconds and run the command again. The console confirms as soon as the change is accepted, before the scheduler has finished reloading.

Change settings only through the dialog

Edit slurm.conf in this dialog. Any later change to the cluster, including a scale, an image update, a volume mount, or an SSH key update, overwrites a hand edit on the controller.

Actions, then Reconfigure Cluster, is a different action. It pushes the cluster’s stored configuration again, exactly as it is. Slurm Configuration is the action that changes what the stored configuration says. See Update a cluster.

Clone Cluster carries the extra slurm.conf text and all four prolog and epilog scripts to the new cluster; partitions stay with the source cluster. See Clone a cluster.

To split the cluster into queues, see Manage partitions. If a line is rejected or a setting seems to be ignored, see Troubleshooting TIR Slurm Cluster.