--- title: Slurm Configuration sidebar_label: Overview --- import { FileText, Layers, Terminal, AlertTriangle, Shield } from 'react-feather'; # Slurm Configuration A TIR Slurm Cluster ships with a working `slurm.conf` that the platform owns and maintains. On top of that baseline you can layer your own configuration from the TIR console — without SSH, without editing files on the controller, and without recreating the cluster. There are three things you can change: | What | What it does | |------|--------------| | [**Extra `slurm.conf`**](/docs/tir/SlurmCluster/slurm-configuration/slurm-conf) | Appends your own `Key=Value` directives to `slurm.conf` on every node | | [**Partitions**](/docs/tir/SlurmCluster/slurm-configuration/partitions) | Adds `PartitionName=` lines that carve the cluster into named queues over chosen nodes | | [**Prolog & Epilog scripts**](/docs/tir/SlurmCluster/slurm-configuration/prolog-epilog) | Runs your own shell scripts around every job, on the workers and on the controller | }, { href: '#what-you-can-and-cannot-change', label: 'What you can change', icon: }, { href: '#applying-a-configuration', label: 'Applying a configuration', icon: }, { href: '#limits-and-validation', label: 'Limits & validation', icon: }, { href: '#configuration-at-create-time', label: 'At create time', icon: }, ]} /> --- ## Open the Slurm Configuration Dialog The same **Slurm Configuration** dialog is reachable from several places. | From | How | Tabs you get | Confirm button | |------|-----|--------------|----------------| | **Cluster list** | Open the ⋮ menu on the cluster row → **Slurm Configuration** | `slurm.conf`, `Partitions`, `Prolog & Epilog` | **Apply Configuration** | | **Cluster details** | **Actions** menu (top right) → **Slurm Configuration** | `slurm.conf`, `Partitions`, `Prolog & Epilog` | **Apply Configuration** | | **Create Training Cluster** | **Advanced Settings** → **Slurm Configuration (Optional)** → **Configure Slurm** | `slurm.conf`, `Prolog & Epilog` only | **Save Configuration** | | **Partition table** | **Cluster Overview** tab → **Slurm Partition Info** → **Add Partition**, or a row's ⋮ → **Edit** | Single-partition form | **Save Partition** | The dialog always opens on the **slurm.conf** tab, and its intro line reads: > Applied to `slurm.conf` on every node of the cluster. Platform managed settings (cluster name, > accounting, node and topology wiring) cannot be overridden. :::info When the menu item is available **Slurm Configuration** appears only while the cluster status is **Running**, and only for users with update permission on Training Clusters. The API refuses the change for any other status with *"Slurm configuration can only be updated on a running cluster"*. ::: :::note Cancel is safe The form reloads from the cluster every time the dialog opens, so closing it with **Cancel** discards everything you typed. Nothing is sent until you press **Apply Configuration**. ::: --- ## What You Can and Cannot Change The platform reserves the directives it needs to keep the cluster wired together, authenticated, and accounted for. Everything else in `slurm.conf` is yours. ### Reserved for the platform If you set one of these keys, the save is rejected with `` `` is managed by the platform and cannot be overridden ``. | Group | Keys | |-------|------| | **Identity & wiring** | `ClusterName`, `NodeName`, `PartitionName`, `SlurmctldHost`, `SlurmctldPort`, `SlurmctldParameters`, `SlurmdPort`, `SlurmctldPidFile`, `SlurmdPidFile`, `SlurmdSpoolDir`, `StateSaveLocation`, `SlurmUser`, `TopologyPlugin`, `Include` | | **Authentication** | `AuthType`, `AuthInfo`, `AuthAltTypes`, `AuthAltParameters`, `CredType`, `JobCredentialPrivateKey`, `JobCredentialPublicCertificate` | | **Accounting** | `AccountingStorageType`, `AccountingStorageHost`, `AccountingStoragePort`, `AccountingStorageEnforce` | | **Prolog / epilog wiring** | `Prolog`, `Epilog`, `PrologSlurmctld`, `EpilogSlurmctld`, `TaskProlog`, `TaskEpilog` | | **Scheduling, priority, preemption** | `SchedulerType`, `SchedulerParameters`, `SelectType`, `MpiDefault`, `PreemptType`, `PreemptMode`, `PreemptExemptTime`, `PriorityType`, `PriorityDecayHalfLife`, `PriorityMaxAge`, `PriorityWeightQOS`, `PriorityWeightAge`, `PriorityWeightFairshare`, `PriorityWeightJobSize`, `PriorityWeightPartition` | :::warning The placeholder is not a working example The **Extra slurm.conf** box shows `PrologFlags=Alloc` and `SchedulerType=sched/backfill` as grey placeholder text. `SchedulerType` is platform-managed and **will be rejected** if you actually type it. Use it as a shape reference only. ::: The prolog and epilog **hooks** are managed because the platform points them at its own script directories — but you still get your own scripts through the [**Prolog & Epilog**](/docs/tir/SlurmCluster/slurm-configuration/prolog-epilog) tab, which is the supported way in. ### Managed by the platform, not editable at all These files are rendered for you and have no console surface: `gres.conf` (GPU topology), `cgroup.conf`, `plugstack.conf` (which is what enables [Pyxis](/docs/tir/SlurmCluster/containers/)), `mpi.conf`, `oci.conf` and `job_container.conf`. Accounting (`slurmdbd` plus its database) is likewise fully managed. :::info QoS is a reference, not a definition TIR does not create QoS entries. You may reference a QoS from a partition's extra options, but the QoS itself has to already exist in the accounting database — the platform does not run `sacctmgr` on your behalf. ::: --- ## Applying a Configuration Pressing **Apply Configuration** replaces the cluster's whole stored Slurm configuration and re-renders the cluster. The dialog states the consequences up front, under the heading **Cluster Reconfigure Required**: - `slurm.conf` is rewritten and slurmctld reloads its configuration - Running jobs keep running; queued jobs are scheduled with the new settings - Prolog and epilog changes take effect on the next job - **The controller will be restarted** — this fourth bullet appears only on clusters whose chart version recreates the controller pod. On newer clusters the reconfigure happens in place and the bullet is absent. On success you get the toast **"Slurm configuration applied. The cluster is reloading slurm.conf."** and the page refreshes both the cluster details and the live partition list. :::note Every save is a full replacement The console always sends the entire configuration, even when you changed one partition. There is no partial update — whatever the dialog holds when you press Apply becomes the complete configuration. ::: ### If applying fails The change is applied synchronously. If the cluster cannot be re-rendered, TIR restores the previous configuration automatically and returns *"Failed to update cluster on infrastructure"*. In the rare case where the rollback also fails you get *"Applying this configuration failed and restoring the previous one also failed. Support has been alerted."* — and a ticket is raised for you. ### Reconfigure Cluster is a different action **Actions → Reconfigure Cluster** re-pushes the current configuration without changing it. Use it when you want the cluster re-rendered from its stored spec; use **Slurm Configuration** when you want to change what that spec says. --- ## Limits and Validation The console validates before sending and lists every problem in a panel above the buttons; the server validates again and is the final authority. | Item | Limit | |------|-------| | Extra `slurm.conf` block | 8 KB | | Each prolog / epilog script | 16 KB | | Partitions per cluster | 20 | | `slurm.conf` key | starts with a letter, then letters, digits, `_`; up to 64 characters | | `slurm.conf` value | 1–512 characters; **no spaces, no `;`, no `#`, no backslash** | | Partition name | `^[A-Za-z0-9][A-Za-z0-9_-]{0,63}$` — letters, digits, `-` and `_` | Comment lines starting with `#` and blank lines are allowed. They are kept in the box so you can read them later, but they are not written into `slurm.conf`. --- ## Configuration at Create Time You can pre-load a configuration while creating the cluster, from **Advanced Settings → Slurm Configuration (Optional) → Configure Slurm**. The console shows a summary next to the button — **Not configured**, or a list such as `extra slurm.conf, 2 partitions, 1 script` — and a **Clear Configuration** button once something is set. The create-time dialog has no **Partitions** tab, because a partition has to name specific nodes and those do not exist yet. The block text says so directly: > Extra `slurm.conf` settings and prolog/epilog scripts for the cluster. Partitions are added from the > cluster's Overview tab once it is running, since a partition selects specific nodes. :::info Cloning **Clone Cluster** carries over the extra `slurm.conf` and all four prolog/epilog scripts. It does **not** carry partitions — those name the original cluster's nodes. ::: --- ## Related Resources - [Extra slurm.conf settings](/docs/tir/SlurmCluster/slurm-configuration/slurm-conf) - [Manage partitions](/docs/tir/SlurmCluster/slurm-configuration/partitions) - [Prolog and epilog scripts](/docs/tir/SlurmCluster/slurm-configuration/prolog-epilog) - [Run containers with Enroot and Pyxis](/docs/tir/SlurmCluster/containers/) - [Cluster actions](/docs/tir/SlurmCluster/manage/actions) - [`slurm.conf` reference (SchedMD)](https://slurm.schedmd.com/slurm.conf.html)