--- title: Extra slurm.conf Settings sidebar_label: Extra slurm.conf --- import { Edit3, CheckCircle, XCircle, Shield, List } from 'react-feather'; # Extra `slurm.conf` Settings The **slurm.conf** tab of the **Slurm Configuration** dialog appends your own directives to `slurm.conf` on every node of the cluster. Use it to tune scheduling behaviour, job defaults, logging and limits that the platform leaves open to you. }, { href: '#syntax-rules', label: 'Syntax rules', icon: }, { href: '#useful-settings', label: 'Useful settings', icon: }, { href: '#keys-you-cannot-set', label: 'Keys you cannot set', icon: }, { href: '#validation-messages', label: 'Validation messages', icon: }, { href: '#verify-the-change', label: 'Verify the change', icon: }, ]} /> --- ## Add or Change Settings 1. Open the cluster in the TIR console and confirm the status is **Running**. 2. Open the **Actions** menu and select **Slurm Configuration**. 3. Stay on the **slurm.conf** tab. The section is headed **Extra slurm.conf**, with the helper line *"Custom Slurm configuration appended to slurm.conf on all nodes. One Key=Value per line."* 4. Type your directives, one `Key=Value` per line. For example: ```ini # Keep completed jobs visible to squeue for two minutes MinJobAge=120 # Let a job step inherit the submitting shell's limits PrologFlags=Alloc,Contain # Extra scheduler logging while tuning DebugFlags=Backfill,Steps ``` 5. Press **Apply Configuration**. The box always shows the configuration currently stored on the cluster, so editing is additive: whatever you leave in the box is what the cluster keeps. **Removing a line removes the directive.** :::tip Use the reference link The **slurm.conf reference** link at the top right of the section opens SchedMD's full [`slurm.conf` manual](https://slurm.schedmd.com/slurm.conf.html) in a new tab — the authoritative list of every directive and accepted value. ::: --- ## Syntax Rules | Rule | Detail | |------|--------| | One directive per line | `Key=Value`. A line without `=`, with an empty key, or with an empty value is rejected | | Key format | Must start with a letter, then letters, digits and `_`; up to 64 characters | | Value format | 1–512 characters from `A-Za-z0-9_.,:/=+@%*[]{}()<>~^!?\|$&'"-` | | No spaces in values | A space inside a value is rejected — use the comma-separated form Slurm accepts, e.g. `DebugFlags=Backfill,Steps` | | No `;`, `#` or `\` in values | `;` in particular is reserved by the rendering pipeline | | Comments | Lines beginning with `#` and blank lines are accepted and ignored | | No duplicates | The same key twice — in any capitalisation — is rejected | | Size | The whole block must stay under **8 KB** | :::info Comments are kept in the form, not in the file Your comments and blank lines are stored with the text and shown again next time you open the dialog, which makes the box a decent place to record *why* a setting is there. They are stripped before `slurm.conf` is written, so they never reach the controller. ::: ### How your lines are merged The rendered file puts your directives first, then the platform's managed directives, then partition lines. Because the managed values are applied last, a platform-owned setting can never be displaced — which is why those keys are rejected at validation time rather than silently ignored. --- ## Useful Settings These are commonly useful on a training cluster and are **not** platform-managed. Always check the [`slurm.conf` reference](https://slurm.schedmd.com/slurm.conf.html) for the exact semantics of the Slurm version your image ships. | Setting | Example | Why you might set it | |---------|---------|----------------------| | `PrologFlags` | `PrologFlags=Alloc` | Run the prolog at allocation instead of first step launch — needed if your worker prolog prepares state a job step depends on | | `MinJobAge` | `MinJobAge=120` | Keep finished jobs in `squeue` output longer while debugging | | `DefMemPerGPU` | `DefMemPerGPU=32768` | Default memory a job gets per GPU when it does not ask | | `DefCpuPerGPU` | `DefCpuPerGPU=8` | Default CPU cores per GPU | | `MaxArraySize` | `MaxArraySize=10001` | Raise the ceiling for job arrays in hyperparameter sweeps | | `MaxJobCount` | `MaxJobCount=50000` | Raise the queue ceiling for very large sweeps | | `KillWait` | `KillWait=60` | Give jobs longer to checkpoint after `SIGTERM` before `SIGKILL` | | `UnkillableStepTimeout` | `UnkillableStepTimeout=180` | Tolerate slow teardown on large multi-GPU steps before the node is drained | | `SlurmctldDebug` / `SlurmdDebug` | `SlurmdDebug=debug2` | Raise log verbosity while investigating a scheduling problem | | `DebugFlags` | `DebugFlags=Backfill` | Log why the backfill scheduler made the decisions it made | | `MessageTimeout` | `MessageTimeout=30` | Tolerate slower RPCs on very large clusters | | `TmpFS` | `TmpFS=/tmp` | Point Slurm's `--tmp` accounting at the right path | :::warning Raise verbosity temporarily `SlurmdDebug=debug2` and broad `DebugFlags` produce a lot of output. Turn them back down once you have what you need — the [Logs tab](/docs/tir/SlurmCluster/manage/logs) becomes hard to read otherwise. ::: --- ## Keys You Cannot Set 46 directives are owned by the platform because they wire the cluster together, authenticate it, or connect it to accounting. Attempting any of them fails with `` `` is managed by the platform and cannot be overridden ``. The full list is grouped in [What you can and cannot change](/docs/tir/SlurmCluster/slurm-configuration/#what-you-can-and-cannot-change). The ones people reach for most often, and what to do instead: | You wanted | Do this instead | |------------|-----------------| | `Prolog`, `Epilog`, `PrologSlurmctld`, `EpilogSlurmctld` | Use the [**Prolog & Epilog** tab](/docs/tir/SlurmCluster/slurm-configuration/prolog-epilog) — your script is added alongside the platform's, not instead of it | | `PartitionName=` | Use the [**Partitions** tab or the partition table](/docs/tir/SlurmCluster/slurm-configuration/partitions) | | `NodeName=` | Node definitions follow the cluster's node pool. Change the pool with **Scale Cluster** | | `SchedulerType`, `SelectType`, `Priority*`, `Preempt*` | Fixed per region so scheduling stays predictable across tenants. Raise a support request if your workload genuinely needs different scheduling | | `AccountingStorage*` | Accounting is fully managed; job accounting already works (`sacct`, `sstat`) | | `TaskProlog` | Not available. A `TaskProlog` runs *inside* the container, and the platform uses that slot itself | --- :::warning Three GPU keys are accepted and then ignored `GresTypes`, `SelectTypeParameters` and `AccountingStorageTRES` are **not** on the rejected list, so the form and the API both accept them and the save succeeds — but on GPU clusters the platform sets its own values afterwards, so yours never reach `slurm.conf`. Do not use them. If you set one and nothing changes, this is why: check with `scontrol show config | grep -i ` and you will see the platform's value. ::: --- ## Validation Messages :::note Only the first bad line is reported The console reports **one** problem per block, not a list of every bad line. Fix the line it names, save again, and it will name the next one if there is one. ::: | Message | Cause | |---------|-------| | ``Extra slurm.conf: `` is not a valid Key=Value line`` | Missing `=`, empty key, empty value, or the line contains `;` | | ``Line of `extra_slurm_conf` is not a valid `Key=Value` slurm.conf entry: `` `` | Server-side form of the same check | | `` `` is not a valid slurm.conf key `` | Key does not start with a letter, or uses characters other than letters, digits and `_` | | ``Value for `` contains characters that are not allowed in slurm.conf`` | Usually a space, `#`, `;` or backslash in the value | | `` `` is managed by the platform and cannot be overridden `` | The key is on the reserved list | | `` `` is set more than once `` | Duplicate key, compared case-insensitively | | `` `extra_slurm_conf` exceeds the maximum size of 8 KB `` | Block too large | --- ## Verify the Change After the toast **"Slurm configuration applied. The cluster is reloading slurm.conf."**, confirm the directive is live from the login node: ```bash ssh root@ ``` ```bash # Show one setting as slurmctld sees it scontrol show config | grep -i MinJobAge ``` ```bash # Dump the whole live configuration scontrol show config ``` If the value has not changed yet, give the controller a few seconds — the API returns as soon as the re-render is accepted, not after slurmctld has finished reloading. The [Logs tab](/docs/tir/SlurmCluster/manage/logs) with **Select Replica → slurm-login** and the controller replica shows the reload as it happens. :::danger Do not edit slurm.conf over SSH `slurm.conf` is rendered by the platform. Anything you write directly on the controller is overwritten the next time the cluster is re-rendered — which also happens on scale, image update and storage mount, not just on a configuration change. The dialog is the only durable path. ::: --- ## Related Resources - [Slurm Configuration overview](/docs/tir/SlurmCluster/slurm-configuration/) - [Manage partitions](/docs/tir/SlurmCluster/slurm-configuration/partitions) - [Prolog and epilog scripts](/docs/tir/SlurmCluster/slurm-configuration/prolog-epilog) - [Troubleshoot jobs](/docs/tir/SlurmCluster/troubleshoot/jobs)