--- title: "Add prolog and epilog scripts" description: "Run your own shell scripts before and after every job on a TIR Slurm Cluster, on the worker nodes and the controller, with safe examples to start from." content_type: "how-to" product: "tir-slurm-cluster" platform: "TIR" url: "/tir/tir-slurm-cluster/how-to/add-prolog-and-epilog-scripts/" topic: ["configure-slurm"] --- # Add prolog and epilog scripts Add shell scripts that Slurm runs around every job on your TIR Slurm Cluster, leaving everyone's job scripts untouched. Use them to prepare the environment, stage data, record accounting information, or clean up after a job. ## Before you begin - A cluster in the **Running** state. See [Cluster statuses and actions](/tir/tir-slurm-cluster/concepts/cluster-statuses-and-actions/). - Update permission on Training Clusters in the project. - A script that starts with a shebang, such as `#!/bin/bash`, and is under 16 KB. ## Choose a script slot The **Prolog & Epilog** tab of the **Slurm Configuration** dialog has four independent scripts: | Slot | Runs on | Runs when | | --------------------- | ----------------------------------- | --------------------------------------------- | | **Worker Prolog** | Every worker node in the allocation | Before each job starts on that node | | **Worker Epilog** | Every worker node in the allocation | After each job ends on that node, for cleanup | | **Controller Prolog** | The Slurm controller | Once per job, at allocation | | **Controller Epilog** | The Slurm controller | Once per job, at completion | A worker script runs once for each node in the job's allocation, and a controller script runs once per job however many nodes it uses. On a four-node job the Worker Prolog runs four times and the Controller Prolog runs once. Anything that must happen exactly once, such as an API call or a database row, belongs in a controller slot. | Goal | Slot | | ------------------------------------------------------------------------ | ----------------- | | Set up per-node scratch space, warm a cache, check hardware health | Worker Prolog | | Delete per-node temporary files, flush node-local logs to shared storage | Worker Epilog | | Record a job in an external system, notify a webhook, enforce a policy | Controller Prolog | | Post job results, send a completion notification, aggregate accounting | Controller Epilog | ## Add a script 1. Select **Actions**, then **Slurm Configuration**, then the **Prolog & Epilog** tab. 2. Paste your script into the box for the slot. It must start with a shebang, or the dialog rejects it with "`` must start with a shebang, for example `#!/bin/bash`". 3. Select **Apply Configuration**. Running jobs carry on as they are, and the next job picks up the new scripts. Each script must stay under 16 KB. Pasting from Windows is fine, because line endings are converted for you. To remove a script, clear its box and apply. You can also add scripts while you create the cluster, under **Advanced Settings**, then **Slurm Configuration (Optional)**. See [Change the Slurm configuration](/tir/tir-slurm-cluster/how-to/change-the-slurm-configuration/). For anything substantial, keep the real logic on shared storage and make the script in the console a thin wrapper. You then change the logic by editing the file on shared storage, and leave the configuration as you applied it. Write the wrapper so it still exits cleanly when the file is missing: ```bash #!/bin/bash [ -x /pfs/ops/job-prolog.sh ] && /pfs/ops/job-prolog.sh exit 0 ``` ## Write a safe script A prolog or epilog runs on the host as root, outside any container the job starts. Quote variables, and keep `rm -rf` away from any path that could be empty. End every Worker Prolog with `exit 0`. A non-zero exit from a Worker Prolog marks that node `DRAIN`, and the node stops accepting work. A prolog with a bug can therefore take the whole cluster out of service, one node per job. Leave out `set -e`, because one failing command would then drain the node: ```bash #!/bin/bash set -uo pipefail mkdir -p /tmp/job_${SLURM_JOB_ID} 2>/dev/null || true exit 0 ``` To have a health check take a bad node out of service, exit non-zero on purpose and log why: ```bash #!/bin/bash if ! nvidia-smi -L >/dev/null 2>&1; then logger -t slurm-prolog "GPU enumeration failed on $(hostname); draining" exit 1 fi exit 0 ``` Bring a drained node back with `scontrol update nodename= state=resume`, or select **Restart All Workers** from the **Actions** menu. See [Update a cluster](/tir/tir-slurm-cluster/how-to/update-a-cluster/). Keep these points in mind as you write: - Set the job's environment elsewhere. A script runs outside the job's container, so the container's contents and the job's environment variables are out of its reach. Slurm's TaskProlog is the hook that reaches into the container, and the platform manages it. Set environment variables in the job script, or bake them into the image. - Keep scripts fast. A prolog runs before every job on every node, and the job waits for it. A 10-second prolog adds 10 seconds to every job's start-up. Push slow work into the job itself. - Keep secrets out. The configuration is stored with the cluster and shown again in the dialog. Read secrets from a file on mounted storage instead. - Make scripts safe to repeat. A script can run again, sometimes at the same time on the same node for different jobs. Use `mkdir -p`, guard commands with `|| true`, and name temporary paths by job ID. - Test for storage. A script may run before a mount is ready, so check that the path exists before you use it. - Write scripts to run unattended, because a script has no terminal to prompt for input. - Send output to a log. The output of a prolog or epilog stays out of the job's own output file, so send it to the node's syslog with `logger`, or append it to a file on shared storage. Slurm exports job details to these scripts. The commonly useful variables are: | Variable | Available in | Meaning | | ------------------- | ---------------- | ------------------------------------- | | SLURM_JOB_ID | All slots | The job's ID | | SLURM_JOB_USER | All slots | The user who submitted the job | | SLURM_JOB_NAME | Controller slots | The job name | | SLURM_JOB_PARTITION | All slots | The partition the job is in | | SLURM_JOB_NODELIST | Controller slots | The compact list of allocated nodes | | SLURM_JOB_NUM_NODES | All slots | The node count | | SLURM_NODEID | Worker slots | The node's index in the allocation | | SLURM_JOB_GPUS | Worker slots | The GPU indices allocated on the node | | SLURM_CLUSTER_NAME | All slots | The Slurm cluster name | | SLURM_JOB_ACCOUNT | All slots | The accounting account, when set | | SLURM_JOB_EXIT_CODE | Epilog slots | The job's exit code | Check what your Slurm version provides in [SchedMD's prolog and epilog guide](https://slurm.schedmd.com/prolog_epilog.html). ### Run the worker prolog at allocation By default Slurm runs the prolog when the first job step launches. If your prolog prepares something the allocation itself depends on, add this line to the extra `slurm.conf`: ```ini PrologFlags=Alloc ``` The prolog then runs at resource allocation instead. You can set PrologFlags yourself. The platform manages the `Prolog` setting. ## Start from a worked example ### Per-job scratch directory Worker Prolog: ```bash #!/bin/bash set -uo pipefail SCRATCH="/tmp/slurm-${SLURM_JOB_ID}" mkdir -p "$SCRATCH" || true chown "${SLURM_JOB_USER}" "$SCRATCH" 2>/dev/null || true chmod 700 "$SCRATCH" 2>/dev/null || true exit 0 ``` Worker Epilog: ```bash #!/bin/bash set -uo pipefail SCRATCH="/tmp/slurm-${SLURM_JOB_ID}" # Guard the path: never rm -rf an empty variable if [ -n "${SLURM_JOB_ID:-}" ] && [ -d "$SCRATCH" ]; then rm -rf "$SCRATCH" || true fi exit 0 ``` ### GPU health check before every job This Worker Prolog drains a node that lists fewer GPUs than expected, so jobs are scheduled only onto healthy hardware. Test it on one node first: point a partition at a single node, run a job there, confirm the behaviour, and then apply it to the whole cluster. A check with the wrong threshold drains every node it touches. ```bash #!/bin/bash set -uo pipefail EXPECTED=8 FOUND=$(nvidia-smi -L 2>/dev/null | wc -l) if [ "$FOUND" -lt "$EXPECTED" ]; then logger -t slurm-prolog "job ${SLURM_JOB_ID}: only ${FOUND}/${EXPECTED} GPUs visible on $(hostname); draining" exit 1 fi exit 0 ``` ### Clear the GPU state between jobs A crashed job can leave a process holding GPU memory and cause a false out-of-memory error in the next job. This Worker Epilog prevents it: ```bash #!/bin/bash set -uo pipefail # Kill anything the job left holding a GPU nvidia-smi --gpu-reset >/dev/null 2>&1 || true exit 0 ``` ### Record jobs in a shared audit log The Controller Prolog runs once per job, so there are no duplicate lines. Controller Prolog: ```bash #!/bin/bash set -uo pipefail LOG=/pfs/ops/job-audit.log if [ -w "$(dirname "$LOG")" ]; then printf '%s START job=%s user=%s partition=%s nodes=%s\n' \ "$(date -Is)" "$SLURM_JOB_ID" "$SLURM_JOB_USER" \ "$SLURM_JOB_PARTITION" "$SLURM_JOB_NODELIST" >> "$LOG" 2>/dev/null || true fi exit 0 ``` Controller Epilog: ```bash #!/bin/bash set -uo pipefail LOG=/pfs/ops/job-audit.log if [ -w "$(dirname "$LOG")" ]; then printf '%s END job=%s exit=%s\n' \ "$(date -Is)" "$SLURM_JOB_ID" "${SLURM_JOB_EXIT_CODE:-unknown}" >> "$LOG" 2>/dev/null || true fi exit 0 ``` ### Copy node-local logs to shared storage This Worker Epilog saves output that would otherwise be lost when the node is reused: ```bash #!/bin/bash set -uo pipefail SRC="/tmp/slurm-${SLURM_JOB_ID}" DEST="/pfs/logs/node-logs/${SLURM_JOB_ID}/$(hostname)" if [ -d "$SRC" ] && [ -d /pfs ]; then mkdir -p "$DEST" 2>/dev/null || true cp -r "$SRC"/*.log "$DEST"/ 2>/dev/null || true fi exit 0 ``` ## Know what the platform already runs The platform runs its own scripts in the Worker Prolog slot, including a GPU health check, and your **Lifecycle Script** runs there too if you added one when you created the cluster. Your scripts are added alongside these, each in its own file, so your scripts and the platform's stay intact side by side. The platform manages `Prolog`, `Epilog`, PrologSlurmctld, EpilogSlurmctld, TaskProlog, and TaskEpilog, so the extra `slurm.conf` refuses them. Add your own scripts on the **Prolog & Epilog** tab instead. The create-time **Lifecycle Script** runs once on each node after the cluster is created, while the Worker Prolog runs before every job. The Lifecycle Script keeps its own file, so a later Worker Prolog leaves it in place. You set a Lifecycle Script on the create form, under **Advanced Settings**, and it stays fixed after that. You can change a Worker Prolog whenever the cluster is **Running**. Use a Lifecycle Script for one-time node setup, such as installing a package, and a Worker Prolog for per-job work. ## Check that the script runs Submit a short job and look for the script's effect: ```bash sbatch --wrap='sleep 5' --partition=all ``` On the node the job landed on, for the scratch directory example: ```bash ls -la /tmp/slurm- grep -i slurm-prolog /var/log/syslog | tail ``` Replace `` with the ID that `sbatch` printed. To watch it live, open the **Logs** tab, choose the worker the job landed on in **Select Replica**, and turn on **Auto Refresh**. See [See jobs and logs](/tir/tir-slurm-cluster/how-to/see-jobs-and-logs/). **Clone Cluster** carries all four scripts to the new cluster. See [Clone a cluster](/tir/tir-slurm-cluster/how-to/clone-a-cluster/). If nodes start draining after a prolog change, see [Troubleshooting jobs on TIR Slurm Cluster](/tir/tir-slurm-cluster/troubleshooting/jobs/).