Slurm Configuration
A TIR Slurm Cluster ships with a working slurm.conf that the platform owns and maintains. On top of
that baseline you can layer your own configuration from the TIR console — without SSH, without
editing files on the controller, and without recreating the cluster.
There are three things you can change:
| What | What it does |
|---|---|
Extra slurm.conf | Appends your own Key=Value directives to slurm.conf on every node |
| Partitions | Adds PartitionName= lines that carve the cluster into named queues over chosen nodes |
| Prolog & Epilog scripts | Runs your own shell scripts around every job, on the workers and on the controller |
Open the Slurm Configuration Dialog
The same Slurm Configuration dialog is reachable from several places.
| From | How | Tabs you get | Confirm button |
|---|---|---|---|
| Cluster list | Open the ⋮ menu on the cluster row → Slurm Configuration | slurm.conf, Partitions, Prolog & Epilog | Apply Configuration |
| Cluster details | Actions menu (top right) → Slurm Configuration | slurm.conf, Partitions, Prolog & Epilog | Apply Configuration |
| Create Training Cluster | Advanced Settings → Slurm Configuration (Optional) → Configure Slurm | slurm.conf, Prolog & Epilog only | Save Configuration |
| Partition table | Cluster Overview tab → Slurm Partition Info → Add Partition, or a row's ⋮ → Edit | Single-partition form | Save Partition |
The dialog always opens on the slurm.conf tab, and its intro line reads:
Applied to
slurm.confon every node of the cluster. Platform managed settings (cluster name, accounting, node and topology wiring) cannot be overridden.
Slurm Configuration appears only while the cluster status is Running, and only for users with update permission on Training Clusters. The API refuses the change for any other status with "Slurm configuration can only be updated on a running cluster".
The form reloads from the cluster every time the dialog opens, so closing it with Cancel discards everything you typed. Nothing is sent until you press Apply Configuration.
What You Can and Cannot Change
The platform reserves the directives it needs to keep the cluster wired together, authenticated, and
accounted for. Everything else in slurm.conf is yours.
Reserved for the platform
If you set one of these keys, the save is rejected with
`<Key>` is managed by the platform and cannot be overridden.
| Group | Keys |
|---|---|
| Identity & wiring | ClusterName, NodeName, PartitionName, SlurmctldHost, SlurmctldPort, SlurmctldParameters, SlurmdPort, SlurmctldPidFile, SlurmdPidFile, SlurmdSpoolDir, StateSaveLocation, SlurmUser, TopologyPlugin, Include |
| Authentication | AuthType, AuthInfo, AuthAltTypes, AuthAltParameters, CredType, JobCredentialPrivateKey, JobCredentialPublicCertificate |
| Accounting | AccountingStorageType, AccountingStorageHost, AccountingStoragePort, AccountingStorageEnforce |
| Prolog / epilog wiring | Prolog, Epilog, PrologSlurmctld, EpilogSlurmctld, TaskProlog, TaskEpilog |
| Scheduling, priority, preemption | SchedulerType, SchedulerParameters, SelectType, MpiDefault, PreemptType, PreemptMode, PreemptExemptTime, PriorityType, PriorityDecayHalfLife, PriorityMaxAge, PriorityWeightQOS, PriorityWeightAge, PriorityWeightFairshare, PriorityWeightJobSize, PriorityWeightPartition |
The Extra slurm.conf box shows PrologFlags=Alloc and SchedulerType=sched/backfill as grey
placeholder text. SchedulerType is platform-managed and will be rejected if you actually type
it. Use it as a shape reference only.
The prolog and epilog hooks are managed because the platform points them at its own script directories — but you still get your own scripts through the Prolog & Epilog tab, which is the supported way in.
Managed by the platform, not editable at all
These files are rendered for you and have no console surface: gres.conf (GPU topology),
cgroup.conf, plugstack.conf (which is what enables
Pyxis), mpi.conf, oci.conf and job_container.conf.
Accounting (slurmdbd plus its database) is likewise fully managed.
TIR does not create QoS entries. You may reference a QoS from a partition's extra options, but the
QoS itself has to already exist in the accounting database — the platform does not run sacctmgr on
your behalf.
Applying a Configuration
Pressing Apply Configuration replaces the cluster's whole stored Slurm configuration and re-renders the cluster. The dialog states the consequences up front, under the heading Cluster Reconfigure Required:
slurm.confis rewritten and slurmctld reloads its configuration- Running jobs keep running; queued jobs are scheduled with the new settings
- Prolog and epilog changes take effect on the next job
- The controller will be restarted — this fourth bullet appears only on clusters whose chart version recreates the controller pod. On newer clusters the reconfigure happens in place and the bullet is absent.
On success you get the toast "Slurm configuration applied. The cluster is reloading slurm.conf." and the page refreshes both the cluster details and the live partition list.
The console always sends the entire configuration, even when you changed one partition. There is no partial update — whatever the dialog holds when you press Apply becomes the complete configuration.
If applying fails
The change is applied synchronously. If the cluster cannot be re-rendered, TIR restores the previous configuration automatically and returns "Failed to update cluster on infrastructure". In the rare case where the rollback also fails you get "Applying this configuration failed and restoring the previous one also failed. Support has been alerted." — and a ticket is raised for you.
Reconfigure Cluster is a different action
Actions → Reconfigure Cluster re-pushes the current configuration without changing it. Use it when you want the cluster re-rendered from its stored spec; use Slurm Configuration when you want to change what that spec says.
Limits and Validation
The console validates before sending and lists every problem in a panel above the buttons; the server validates again and is the final authority.
| Item | Limit |
|---|---|
Extra slurm.conf block | 8 KB |
| Each prolog / epilog script | 16 KB |
| Partitions per cluster | 20 |
slurm.conf key | starts with a letter, then letters, digits, _; up to 64 characters |
slurm.conf value | 1–512 characters; no spaces, no ;, no #, no backslash |
| Partition name | ^[A-Za-z0-9][A-Za-z0-9_-]{0,63}$ — letters, digits, - and _ |
Comment lines starting with # and blank lines are allowed. They are kept in the box so you can read
them later, but they are not written into slurm.conf.
Configuration at Create Time
You can pre-load a configuration while creating the cluster, from
Advanced Settings → Slurm Configuration (Optional) → Configure Slurm. The console shows a summary
next to the button — Not configured, or a list such as extra slurm.conf, 2 partitions, 1 script
— and a Clear Configuration button once something is set.
The create-time dialog has no Partitions tab, because a partition has to name specific nodes and those do not exist yet. The block text says so directly:
Extra
slurm.confsettings and prolog/epilog scripts for the cluster. Partitions are added from the cluster's Overview tab once it is running, since a partition selects specific nodes.
Clone Cluster carries over the extra slurm.conf and all four prolog/epilog scripts. It does
not carry partitions — those name the original cluster's nodes.