Extra slurm.conf Settings

The slurm.conf tab of the Slurm Configuration dialog appends your own directives to slurm.conf on every node of the cluster. Use it to tune scheduling behaviour, job defaults, logging and limits that the platform leaves open to you.

Add settingsSyntax rulesUseful settingsKeys you cannot setValidation messagesVerify the change

---

Add or Change Settings

1. Open the cluster in the TIR console and confirm the status is Running.

2. Open the Actions menu and select Slurm Configuration.

3. Stay on the slurm.conf tab. The section is headed Extra slurm.conf, with the helper line "Custom Slurm configuration appended to slurm.conf on all nodes. One Key=Value per line."

4. Type your directives, one Key=Value per line. For example:

   # Keep completed jobs visible to squeue for two minutes
   MinJobAge=120
   # Let a job step inherit the submitting shell's limits
   PrologFlags=Alloc,Contain
   # Extra scheduler logging while tuning
   DebugFlags=Backfill,Steps

5. Press Apply Configuration.

The box always shows the configuration currently stored on the cluster, so editing is additive: whatever you leave in the box is what the cluster keeps. Removing a line removes the directive.

Use the reference link

The slurm.conf reference link at the top right of the section opens SchedMD's full slurm.conf manual in a new tab — the authoritative list of every directive and accepted value.

---

Syntax Rules

Rule	Detail
One directive per line	Key=Value. A line without =, with an empty key, or with an empty value is rejected
Key format	Must start with a letter, then letters, digits and _; up to 64 characters
Value format	1–512 characters from A-Za-z0-9_.,:/=+@%*[]{}()<>~^!?|$&'"-
No spaces in values	A space inside a value is rejected — use the comma-separated form Slurm accepts, e.g. DebugFlags=Backfill,Steps
No ;, # or \ in values	; in particular is reserved by the rendering pipeline
Comments	Lines beginning with # and blank lines are accepted and ignored
No duplicates	The same key twice — in any capitalisation — is rejected
Size	The whole block must stay under 8 KB

Comments are kept in the form, not in the file

Your comments and blank lines are stored with the text and shown again next time you open the dialog, which makes the box a decent place to record why a setting is there. They are stripped before slurm.conf is written, so they never reach the controller.

How your lines are merged

The rendered file puts your directives first, then the platform's managed directives, then partition lines. Because the managed values are applied last, a platform-owned setting can never be displaced — which is why those keys are rejected at validation time rather than silently ignored.

---

Useful Settings

These are commonly useful on a training cluster and are not platform-managed. Always check the slurm.conf reference for the exact semantics of the Slurm version your image ships.

Setting	Example	Why you might set it
PrologFlags	PrologFlags=Alloc	Run the prolog at allocation instead of first step launch — needed if your worker prolog prepares state a job step depends on
MinJobAge	MinJobAge=120	Keep finished jobs in squeue output longer while debugging
DefMemPerGPU	DefMemPerGPU=32768	Default memory a job gets per GPU when it does not ask
DefCpuPerGPU	DefCpuPerGPU=8	Default CPU cores per GPU
MaxArraySize	MaxArraySize=10001	Raise the ceiling for job arrays in hyperparameter sweeps
MaxJobCount	MaxJobCount=50000	Raise the queue ceiling for very large sweeps
KillWait	KillWait=60	Give jobs longer to checkpoint after SIGTERM before SIGKILL
UnkillableStepTimeout	UnkillableStepTimeout=180	Tolerate slow teardown on large multi-GPU steps before the node is drained
SlurmctldDebug / SlurmdDebug	SlurmdDebug=debug2	Raise log verbosity while investigating a scheduling problem
DebugFlags	DebugFlags=Backfill	Log why the backfill scheduler made the decisions it made
MessageTimeout	MessageTimeout=30	Tolerate slower RPCs on very large clusters
TmpFS	TmpFS=/tmp	Point Slurm's --tmp accounting at the right path

Raise verbosity temporarily

SlurmdDebug=debug2 and broad DebugFlags produce a lot of output. Turn them back down once you have what you need — the Logs tab becomes hard to read otherwise.

---

Keys You Cannot Set

46 directives are owned by the platform because they wire the cluster together, authenticate it, or connect it to accounting. Attempting any of them fails with `<Key>` is managed by the platform and cannot be overridden.

The full list is grouped in What you can and cannot change. The ones people reach for most often, and what to do instead:

You wanted	Do this instead
Prolog, Epilog, PrologSlurmctld, EpilogSlurmctld	Use the Prolog & Epilog tab — your script is added alongside the platform's, not instead of it
PartitionName=	Use the Partitions tab or the partition table
NodeName=	Node definitions follow the cluster's node pool. Change the pool with Scale Cluster
SchedulerType, SelectType, Priority*, Preempt*	Fixed per region so scheduling stays predictable across tenants. Raise a support request if your workload genuinely needs different scheduling
AccountingStorage*	Accounting is fully managed; job accounting already works (sacct, sstat)
TaskProlog	Not available. A TaskProlog runs inside the container, and the platform uses that slot itself

---

Three GPU keys are accepted and then ignored

GresTypes, SelectTypeParameters and AccountingStorageTRES are not on the rejected list, so the form and the API both accept them and the save succeeds — but on GPU clusters the platform sets its own values afterwards, so yours never reach slurm.conf.

Do not use them. If you set one and nothing changes, this is why: check with scontrol show config | grep -i <key> and you will see the platform's value.

---

Validation Messages

Only the first bad line is reported

The console reports one problem per block, not a list of every bad line. Fix the line it names, save again, and it will name the next one if there is one.

Message	Cause
Extra slurm.conf: `<line>` is not a valid Key=Value line	Missing =, empty key, empty value, or the line contains ;
Line <n> of `extra_slurm_conf` is not a valid `Key=Value` slurm.conf entry: `<line>`	Server-side form of the same check
`<key>` is not a valid slurm.conf key	Key does not start with a letter, or uses characters other than letters, digits and _
Value for `<key>` contains characters that are not allowed in slurm.conf	Usually a space, #, ; or backslash in the value
`<key>` is managed by the platform and cannot be overridden	The key is on the reserved list
`<key>` is set more than once	Duplicate key, compared case-insensitively
`extra_slurm_conf` exceeds the maximum size of 8 KB	Block too large

---

Verify the Change

After the toast "Slurm configuration applied. The cluster is reloading slurm.conf.", confirm the directive is live from the login node:

ssh root@<cluster-ip>

# Show one setting as slurmctld sees it
scontrol show config | grep -i MinJobAge

# Dump the whole live configuration
scontrol show config

If the value has not changed yet, give the controller a few seconds — the API returns as soon as the re-render is accepted, not after slurmctld has finished reloading. The Logs tab with Select Replica → slurm-login and the controller replica shows the reload as it happens.

Do not edit slurm.conf over SSH

slurm.conf is rendered by the platform. Anything you write directly on the controller is overwritten the next time the cluster is re-rendered — which also happens on scale, image update and storage mount, not just on a configuration change. The dialog is the only durable path.

---

Related Resources

• Slurm Configuration overview
• Manage partitions
• Prolog and epilog scripts
• Troubleshoot jobs