--- title: "Change the Slurm configuration" description: "Add your own settings to a TIR Slurm Cluster's slurm.conf from the console, apply them to every node, and check that the scheduler picked them up." content_type: "how-to" product: "tir-slurm-cluster" platform: "TIR" url: "/tir/tir-slurm-cluster/how-to/change-the-slurm-configuration/" topic: ["configure-slurm"] --- # Change the Slurm configuration Add your own `Key=Value` settings to `slurm.conf` on every node of a TIR Slurm Cluster, from the console, on the cluster you already have. E2E manages the Slurm installation and keeps the settings that wire the cluster together. Everything else in `slurm.conf` is yours to set. The same dialog also holds your partitions and your prolog and epilog scripts; see [Manage partitions](/tir/tir-slurm-cluster/how-to/manage-partitions/) and [Add prolog and epilog scripts](/tir/tir-slurm-cluster/how-to/add-prolog-and-epilog-scripts/). ## Before you begin - A cluster in the **Running** state. **Slurm Configuration** is available only while the cluster is **Running**. See [Cluster statuses and actions](/tir/tir-slurm-cluster/concepts/cluster-statuses-and-actions/). - Update permission on Training Clusters in the project, which puts the action on the **Actions** menu. ## Open the Slurm Configuration dialog You can open the dialog from three places: - The row menu of a cluster in the cluster list, under **Training Cluster** in the TIR sidebar. - The **Actions** menu on the cluster's details page. - The create form, under **Advanced Settings**, then **Slurm Configuration (Optional)**, then **Configure Slurm**. What you set here applies to `slurm.conf` on every node of the cluster. The platform manages the cluster name, accounting, and the node and topology wiring, so those settings cannot be overridden. On a running cluster you can also set partitions. At create time you cannot, because a partition names specific nodes, and those nodes exist only once the cluster is running. You can still pre-load an extra `slurm.conf` block and all four prolog and epilog scripts at create time, and save them with **Save Configuration**. The dialog loads the cluster's current configuration every time it opens. Closing it with **Cancel** discards what you typed, and the cluster's configuration stays as it was. ## Add settings to slurm.conf 1. On the **slurm.conf** tab, find the **Extra slurm.conf** box. It holds one `Key=Value` per line. 2. Type your settings. For example: ```ini # Keep finished jobs visible to squeue for two minutes MinJobAge=120 # Run the prolog at allocation instead of at the first job step PrologFlags=Alloc # Extra scheduler logging while tuning DebugFlags=Backfill ``` 3. Select **Apply Configuration**. The cluster reloads `slurm.conf`. The platform manages the scheduler itself, so it rejects a SchedulerType line, even though the box's example text includes one. The box shows the configuration stored on the cluster, so whatever you leave in it is what the cluster keeps. Removing a line removes that setting. Applying replaces the cluster's whole stored Slurm configuration in one request: what the dialog holds at that moment, including every partition and every script, becomes the complete configuration. Running jobs keep running, queued jobs are scheduled under the new settings, and prolog and epilog changes take effect on the next job. On some clusters, applying restarts the controller; newer clusters reconfigure in place while the controller keeps running. The dialog says which before you apply. If applying the change fails, the stored settings go back to what they were, and the console shows "Failed to update cluster on infrastructure". If restoring them also fails, support is alerted, and the console shows "Applying this configuration failed and restoring the previous one also failed. Support has been alerted." ## Follow the syntax rules Each line in the box must pass these checks before TIR applies the change: - A line is `Key=Value`, with a key before the `=` and a value after it. - A key starts with a letter, then uses only letters, digits, and underscores, up to 64 characters. - A value is 1 to 512 characters. It can use letters, digits, and the punctuation marks listed after this list. A space, a hash sign, a semicolon, or a backslash is rejected. - The same key twice is rejected as a duplicate, even when the capital letters differ. - Lines that start with `#`, and blank lines, are accepted and ignored. They stay in the box, so you can record why a setting is there, and only the settings themselves reach `slurm.conf`. - The whole block stays under 8 KB. The console reports one problem per block, the first it finds. Fix that line and apply again; the console then names the next problem, if there is one. The messages are: | Message | Cause | | ------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | | "`` is not a valid Key=Value line" | The line has no `=`, an empty key, or an empty value. | | "`` is not a valid slurm.conf key" | The key starts with something other than a letter, or uses a character other than letters, digits, and underscore. | | "Value for `` contains characters that are not allowed" | Usually a space, a hash sign, a semicolon, or a backslash in the value. | | "`` is managed by the platform and cannot be overridden" | The key is one the platform reserves. | | "`` is set more than once" | The same key twice, even when the capital letters differ. | | A message that the block exceeds the maximum size of 8 KB | The block is too large. | The punctuation marks a value can use are underscore, period, comma, colon, slash, equals, plus, at sign, percent, asterisk, square brackets, braces, parentheses, angle brackets, tilde, caret, exclamation mark, question mark, vertical bar, dollar sign, ampersand, single and double quotes, and hyphen. For a value that lists several items, use the comma-separated form Slurm accepts, such as DebugFlags=Backfill,Steps. For what each setting means, see the [`slurm.conf` reference from SchedMD](https://slurm.schedmd.com/slurm.conf.html) for the Slurm version your image ships. ## Choose useful settings These settings are yours to set, and they are often useful on a training cluster: | Setting | Example | What it does | | --------------------------- | ------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | | PrologFlags | PrologFlags=Alloc | Runs the worker prolog at allocation instead of at the first job step. Use it when the prolog prepares something a job step depends on | | MinJobAge | MinJobAge=120 | Keeps finished jobs in `squeue` and on the **Jobs** tab longer | | DefMemPerGPU | DefMemPerGPU=32768 | Sets the memory per GPU that a job gets by default | | DefCpuPerGPU | DefCpuPerGPU=8 | Sets the CPU cores per GPU that a job gets by default | | MaxArraySize | MaxArraySize=10001 | Raises the ceiling for job arrays, for example in sweeps | | MaxJobCount | MaxJobCount=50000 | Raises the queue ceiling for very large sweeps | | KillWait | KillWait=60 | Gives jobs longer to checkpoint after `SIGTERM` | | UnkillableStepTimeout | UnkillableStepTimeout=180 | Allows slow teardown on large multi-GPU steps | | SlurmctldDebug, SlurmdDebug | SlurmdDebug=debug2 | Raises log detail while you investigate a scheduling problem | | DebugFlags | DebugFlags=Backfill | Logs why the backfill scheduler made its choices | | MessageTimeout | MessageTimeout=30 | Allows slower messages on very large clusters | | TmpFS | TmpFS=/tmp | Points Slurm's `--tmp` accounting at the right path | Turn SlurmdDebug=debug2 and broad DebugFlags back down once you have what you need, because they fill the **Logs** tab quickly. ## Settings the platform keeps The platform reserves 46 `slurm.conf` keys. If you set one, the dialog rejects it with "`` is managed by the platform and cannot be overridden". | Group | Keys | | -------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Cluster identity and node wiring | ClusterName, NodeName, PartitionName, SlurmctldHost, SlurmctldPort, SlurmctldParameters, SlurmdPort, SlurmctldPidFile, SlurmdPidFile, SlurmdSpoolDir, StateSaveLocation, SlurmUser, TopologyPlugin, `Include` | | Authentication | AuthType, AuthInfo, AuthAltTypes, AuthAltParameters, CredType, JobCredentialPrivateKey, JobCredentialPublicCertificate | | Accounting | AccountingStorageType, AccountingStorageHost, AccountingStoragePort, AccountingStorageEnforce | | Prolog and epilog hooks | `Prolog`, `Epilog`, PrologSlurmctld, EpilogSlurmctld, TaskProlog, TaskEpilog | | Scheduling, priority, preemption | SchedulerType, SchedulerParameters, SelectType, MpiDefault, PreemptType, PreemptMode, PreemptExemptTime, PriorityType, PriorityDecayHalfLife, PriorityMaxAge, PriorityWeightQOS, PriorityWeightAge, PriorityWeightFairshare, PriorityWeightJobSize, PriorityWeightPartition | For the settings people most often reach for, this is what to do instead: | You want to set | Do this instead | | -------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `Prolog`, `Epilog`, PrologSlurmctld, EpilogSlurmctld | Use the **Prolog & Epilog** tab. Your script is added alongside the platform's hook. See [Add prolog and epilog scripts](/tir/tir-slurm-cluster/how-to/add-prolog-and-epilog-scripts/). | | PartitionName | Use the **Partitions** tab. See [Manage partitions](/tir/tir-slurm-cluster/how-to/manage-partitions/). | | NodeName | The node list follows the cluster's node count. Change it with **Scale Cluster**. See [Scale a cluster](/tir/tir-slurm-cluster/how-to/scale-a-cluster/). | | TaskProlog, TaskEpilog | Leave these to the platform, which uses them itself. | | SchedulerType, SelectType, `Priority` and `Preempt` keys | These are fixed for every cluster. Raise a support request if your jobs need different scheduling. | | AccountingStorage keys | Accounting is fully managed, and `sacct` and `sstat` already work. | The platform manages `gres.conf`, `cgroup.conf`, `plugstack.conf`, `mpi.conf`, `oci.conf`, and job_container.conf, which have no console screen. It manages accounting, including its database, too. On a GPU cluster, the platform also sets GresTypes, SelectTypeParameters, and AccountingStorageTRES; see the [TIR Slurm Cluster FAQ](/tir/tir-slurm-cluster/faq/#why-does-my-grestypes-setting-not-take-effect). TIR does not create QoS entries, so a partition can name a QoS only if it already exists in the accounting database. ## Check that the setting is live After you apply, connect to the login node and read the setting back: ```bash ssh root@ ``` Replace `` with the cluster's address from **Connection Details** on the details page. Then read one setting, or the whole live configuration: ```bash scontrol show config | grep -i MinJobAge scontrol show config ``` If the old value still shows, wait a few seconds and run the command again. The console confirms as soon as the change is accepted, before the scheduler has finished reloading. ## Change settings only through the dialog Edit `slurm.conf` in this dialog. Any later change to the cluster, including a scale, an image update, a volume mount, or an SSH key update, overwrites a hand edit on the controller. **Actions**, then **Reconfigure Cluster**, is a different action. It pushes the cluster's stored configuration again, exactly as it is. **Slurm Configuration** is the action that changes what the stored configuration says. See [Update a cluster](/tir/tir-slurm-cluster/how-to/update-a-cluster/). **Clone Cluster** carries the extra `slurm.conf` text and all four prolog and epilog scripts to the new cluster; partitions stay with the source cluster. See [Clone a cluster](/tir/tir-slurm-cluster/how-to/clone-a-cluster/). To split the cluster into queues, see [Manage partitions](/tir/tir-slurm-cluster/how-to/manage-partitions/). If a line is rejected or a setting seems to be ignored, see [Troubleshooting TIR Slurm Cluster](/tir/tir-slurm-cluster/troubleshooting/tir-slurm-cluster/).