--- title: Prolog and Epilog Scripts sidebar_label: Prolog & Epilog --- import { Terminal, Server, AlertTriangle, Clock, Code, Shield } from 'react-feather'; # Prolog and Epilog Scripts Prolog and epilog scripts are shell scripts Slurm runs around every job. Use them to prepare the environment, stage data, record accounting information, or clean up after a job — automatically, for every job on the cluster, without touching anyone's job script. }, { href: '#add-a-script', label: 'Add a script', icon: }, { href: '#writing-a-safe-script', label: 'Writing a safe script', icon: }, { href: '#available-environment', label: 'Available environment', icon: }, { href: '#worked-examples', label: 'Worked examples', icon: }, { href: '#platform-managed-scripts', label: 'Platform scripts', icon: }, ]} /> --- ## The Four Script Slots The **Prolog & Epilog** tab of the **Slurm Configuration** dialog gives you four independent scripts. | Slot | Runs on | Runs when | Slurm directive | |------|---------|-----------|-----------------| | **Worker Prolog** | Every worker node in the allocation | Before each job starts on that node | `Prolog` | | **Worker Epilog** | Every worker node in the allocation | After each job ends on that node | `Epilog` | | **Controller Prolog** | The Slurm controller | Once per job, at allocation | `PrologSlurmctld` | | **Controller Epilog** | The Slurm controller | Once per job, at completion | `EpilogSlurmctld` | The console repeats this next to each box: - *Runs on Slurm workers before each job starts.* - *Runs on Slurm workers after each job ends. Use for cleanup.* - *Runs on slurmctld at job allocation. Central job setup.* - *Runs on slurmctld at job completion. Central job teardown.* ### Which slot do I want? | Goal | Slot | |------|------| | Set up per-node scratch space, warm a cache, check hardware health before the job | Worker Prolog | | Delete per-node temporary files, flush node-local logs to shared storage | Worker Epilog | | Record a job in an external system, notify a webhook, enforce a policy centrally | Controller Prolog | | Post job results to a tracker, send a completion notification, aggregate accounting | Controller Epilog | | Do something *inside* the job's container | Not available — see [the container note](#worker-scripts-run-on-the-host-not-in-your-container) | :::info Worker scripts run once per node, controller scripts once per job On a 4-node job, the Worker Prolog runs 4 times (once per node) and the Controller Prolog runs once. Anything that must happen exactly once — an API call, a database row — belongs in the controller slot. ::: --- ## Add a Script 1. Open the cluster and confirm the status is **Running**. 2. **Actions** → **Slurm Configuration** → **Prolog & Epilog** tab. 3. Paste your script into the relevant box. It **must start with a shebang**. 4. Click **Apply Configuration**. The dialog reminds you what applying means, including: > Prolog and epilog changes take effect on the next job Jobs already running are unaffected. The next job submitted picks up the new scripts. ### Requirements | Requirement | Detail | |-------------|--------| | Shebang | The first characters must be `#!`. Missing it fails with *"`` must start with a shebang, for example `#!/bin/bash`"* | | Size | 16 KB per script | | Line endings | Normalised to LF for you — pasting from Windows is safe | | Removing a script | Clear the box and apply. An empty script is removed entirely | :::tip Keep long logic in a file on shared storage The 16 KB limit is generous for a hook but not for a program. For anything substantial, put the real script on your PFS or SFS mount and make the prolog a thin, defensive wrapper: ```bash #!/bin/bash [ -x /pfs/ops/job-prolog.sh ] && /pfs/ops/job-prolog.sh exit 0 ``` You can then iterate on the logic without re-applying the cluster configuration — but note the wrapper must tolerate the file being missing or the storage being unmounted. ::: --- ## Writing a Safe Script ### A non-zero exit from a Worker Prolog drains the node This is the most important rule on this page. If your Worker Prolog exits non-zero, Slurm marks that node **DRAIN** and it stops accepting work. A prolog with a bug can take your whole cluster out of service, one node per job, until you notice it in the [Nodes tab](/docs/tir/SlurmCluster/manage/nodes). Always end with an explicit success, and never let a failing command decide the exit code: ```bash #!/bin/bash # Do not use `set -e` here — a single failing command would drain the node. set -uo pipefail # ... your work, each step tolerant of failure ... mkdir -p /tmp/job_${SLURM_JOB_ID} 2>/dev/null || true exit 0 ``` If you genuinely *want* a health check to remove a bad node from service, exiting non-zero is the mechanism — just make it deliberate, and log why: ```bash #!/bin/bash if ! nvidia-smi -L >/dev/null 2>&1; then logger -t slurm-prolog "GPU enumeration failed on $(hostname); draining" exit 1 fi exit 0 ``` Recover a drained node with `scontrol update nodename= state=resume`, or [**Restart All Workers**](/docs/tir/SlurmCluster/manage/actions#restart-actions) from the Actions menu. ### Worker scripts run on the host, not in your container Worker and controller scripts run **on the host as root**, outside any [Enroot/Pyxis container](/docs/tir/SlurmCluster/containers/) your job starts. So: - `bash` is available — `#!/bin/bash` is fine. - You **cannot** see or modify the inside of the job's container from a prolog. - Environment variables you export in a prolog do **not** reach the job's processes. Slurm's `TaskProlog` is the only hook that crosses into the container, and it is platform-managed. To set environment variables for a job, do it in the job script, or bake them into the image. ### Keep them fast A prolog runs before **every** job on every node, and the job waits for it. A ten-second prolog adds ten seconds to every job's start-up, and forty seconds of wall clock across a four-node allocation. Push anything slow into the job itself or into a scheduled task elsewhere. ### Other hazards | Hazard | Guidance | |--------|----------| | Running as root | The script has full node privileges. Quote variables, avoid `rm -rf` with an unquoted or possibly-empty path | | Secrets | Do not embed credentials — the configuration is stored with the cluster and echoed back into the dialog. Read secrets from a file on your mounted storage instead | | Storage assumptions | A prolog may run before you expect a mount to be ready. Test for the path rather than assuming it | | Idempotency | Scripts run repeatedly, sometimes concurrently on the same node for different jobs. Use `mkdir -p`, guard with `|| true`, and namespace by `$SLURM_JOB_ID` | | Interactive commands | There is no TTY. Nothing may prompt or block on input | --- ## Available Environment Slurm exports job context into prolog and epilog scripts. The commonly useful ones: | Variable | Available in | Meaning | |----------|--------------|---------| | `SLURM_JOB_ID` | all | The job's ID | | `SLURM_JOB_USER` | all | User who submitted the job | | `SLURM_JOB_NAME` | controller slots | Job name | | `SLURM_JOB_PARTITION` | all | Partition the job is in | | `SLURM_JOB_NODELIST` | controller slots | Compact list of allocated nodes | | `SLURM_JOB_NUM_NODES` | all | Node count | | `SLURM_NODEID` | worker slots | This node's index in the allocation | | `SLURM_JOB_GPUS` | worker slots | GPU indices allocated on this node | | `SLURM_CLUSTER_NAME` | all | The Slurm cluster name | | `SLURM_JOB_ACCOUNT` | all | Accounting account, when set | | `SLURM_JOB_EXIT_CODE` | epilog slots | The job's exit code | Check what your Slurm version provides with [SchedMD's Prolog and Epilog guide](https://slurm.schedmd.com/prolog_epilog.html). :::note Prolog output does not appear in the job's log `stdout` from a prolog is not written to the job's `--output` file. Use `logger` to send it to the node's syslog, or append to a file on shared storage — then read it from the [Logs tab](/docs/tir/SlurmCluster/manage/logs) or over SSH. ::: ### `PrologFlags` changes when the prolog runs By default Slurm runs the `Prolog` when the first job step launches. If your prolog prepares something the allocation itself depends on, set the flag in [extra `slurm.conf`](/docs/tir/SlurmCluster/slurm-configuration/slurm-conf): ```ini PrologFlags=Alloc ``` This runs the prolog at resource allocation instead. `PrologFlags` is yours to set — the `Prolog` directive itself is platform-managed. --- ## Worked Examples ### Per-job scratch directory Worker Prolog: ```bash #!/bin/bash set -uo pipefail SCRATCH="/tmp/slurm-${SLURM_JOB_ID}" mkdir -p "$SCRATCH" || true chown "${SLURM_JOB_USER}" "$SCRATCH" 2>/dev/null || true chmod 700 "$SCRATCH" 2>/dev/null || true exit 0 ``` Worker Epilog: ```bash #!/bin/bash set -uo pipefail SCRATCH="/tmp/slurm-${SLURM_JOB_ID}" # Guard the path: never rm -rf an empty variable if [ -n "${SLURM_JOB_ID:-}" ] && [ -d "$SCRATCH" ]; then rm -rf "$SCRATCH" || true fi exit 0 ``` ### GPU health gate before every job Worker Prolog — deliberately drains a node whose GPUs are not enumerable, so jobs are not scheduled onto broken hardware: ```bash #!/bin/bash set -uo pipefail EXPECTED=8 FOUND=$(nvidia-smi -L 2>/dev/null | wc -l) if [ "$FOUND" -lt "$EXPECTED" ]; then logger -t slurm-prolog "job ${SLURM_JOB_ID}: only ${FOUND}/${EXPECTED} GPUs visible on $(hostname); draining" exit 1 fi exit 0 ``` :::warning Test a draining prolog on one node first Point a partition at a single node, run a job there, confirm the behaviour, and only then apply it cluster-wide. A health gate with the threshold set wrong drains every node it touches. ::: ### Clear the GPU state between jobs Worker Epilog: ```bash #!/bin/bash set -uo pipefail # Kill anything the job left holding a GPU nvidia-smi --gpu-reset >/dev/null 2>&1 || true exit 0 ``` ### Record jobs to a shared audit log Controller Prolog — runs once per job, so no duplicates: ```bash #!/bin/bash set -uo pipefail LOG=/pfs/ops/job-audit.log if [ -w "$(dirname "$LOG")" ]; then printf '%s START job=%s user=%s partition=%s nodes=%s\n' \ "$(date -Is)" "$SLURM_JOB_ID" "$SLURM_JOB_USER" \ "$SLURM_JOB_PARTITION" "$SLURM_JOB_NODELIST" >> "$LOG" 2>/dev/null || true fi exit 0 ``` Controller Epilog: ```bash #!/bin/bash set -uo pipefail LOG=/pfs/ops/job-audit.log if [ -w "$(dirname "$LOG")" ]; then printf '%s END job=%s exit=%s\n' \ "$(date -Is)" "$SLURM_JOB_ID" "${SLURM_JOB_EXIT_CODE:-unknown}" >> "$LOG" 2>/dev/null || true fi exit 0 ``` ### Flush node-local logs to shared storage Worker Epilog — saves output that would otherwise be lost when the node is reused: ```bash #!/bin/bash set -uo pipefail SRC="/tmp/slurm-${SLURM_JOB_ID}" DEST="/pfs/logs/node-logs/${SLURM_JOB_ID}/$(hostname)" if [ -d "$SRC" ] && [ -d /pfs ]; then mkdir -p "$DEST" 2>/dev/null || true cp -r "$SRC"/*.log "$DEST"/ 2>/dev/null || true fi exit 0 ``` --- ## Platform-Managed Scripts Your scripts are **added alongside** the platform's, never instead of them. Each one lands in its own file, so nothing you write can disable or overwrite platform behaviour — and nothing the platform does overwrites yours. The platform already runs, in the Worker Prolog hook: | Script | What it does | |--------|--------------| | GPU health hook | Writes the GPU-to-job mapping that per-GPU DCGM metrics in the [Nodes tab](/docs/tir/SlurmCluster/manage/nodes) depend on | | Enroot mount setup | Writes the Enroot mount rule that makes Slurm's config cache readable inside [Pyxis containers](/docs/tir/SlurmCluster/containers/) | | Lifecycle script | Your create-time **Lifecycle Script**, if you supplied one. It keeps its own file, so a Worker Prolog does not replace it | | Storage monitoring hook | On some clusters, a hook that tags jobs for storage accounting | The three controller and worker **epilog** hooks contain only your script unless you set one. :::info You cannot set the Slurm directives directly `Prolog`, `Epilog`, `PrologSlurmctld`, `EpilogSlurmctld`, `TaskProlog` and `TaskEpilog` are platform-managed keys in `slurm.conf` — the platform points them at the script directories it manages. The **Prolog & Epilog** tab is the supported way to add your own, and it is a superset of what setting the directive would give you. ::: ### Lifecycle Script vs Worker Prolog Both let you run your own shell code, but they are not the same thing: | | Lifecycle Script | Worker Prolog | |--|------------------|---------------| | When | Once, on each node, after the cluster is created | Before **every job**, on every node in the allocation | | Set from | Create form → **Advanced Settings** → **Lifecycle Script** → **Add Script** | **Slurm Configuration** → **Prolog & Epilog** | | Changeable later | No — it is a create-time field | Yes, any time the cluster is Running | | Use for | One-time node setup: install a package, write a config file | Per-job setup and teardown | --- ## Cloning and Verification **Clone Cluster** carries all four prolog/epilog scripts and the extra `slurm.conf` to the new cluster. It does not carry [partitions](/docs/tir/SlurmCluster/slurm-configuration/partitions). To confirm a script is live, submit a trivial job and look for its effect: ```bash sbatch --wrap='sleep 5' --partition=all ``` ```bash # On the node the job landed on ls -la /tmp/slurm- grep -i slurm-prolog /var/log/syslog | tail ``` The [Logs tab](/docs/tir/SlurmCluster/manage/logs) shows controller and worker output; pick the replica with **Select Replica** and turn on **Auto Refresh** while you test. --- ## Related Resources - [Slurm Configuration overview](/docs/tir/SlurmCluster/slurm-configuration/) - [Extra slurm.conf settings](/docs/tir/SlurmCluster/slurm-configuration/slurm-conf) - [Manage partitions](/docs/tir/SlurmCluster/slurm-configuration/partitions) - [Containers with Enroot and Pyxis](/docs/tir/SlurmCluster/containers/) - [Nodes and GPU health](/docs/tir/SlurmCluster/manage/nodes) - [Prolog and Epilog guide (SchedMD)](https://slurm.schedmd.com/prolog_epilog.html)