Add prolog and epilog scripts

Add shell scripts that Slurm runs around every job on your TIR Slurm Cluster, leaving everyone’s job scripts untouched. Use them to prepare the environment, stage data, record accounting information, or clean up after a job.

Before you begin

• A cluster in the Running state. See Cluster statuses and actions.
• Update permission on Training Clusters in the project.
• A script that starts with a shebang, such as #!/bin/bash, and is under 16 KB.

Choose a script slot

The Prolog & Epilog tab of the Slurm Configuration dialog has four independent scripts:

Slot	Runs on	Runs when
Worker Prolog	Every worker node in the allocation	Before each job starts on that node
Worker Epilog	Every worker node in the allocation	After each job ends on that node, for cleanup
Controller Prolog	The Slurm controller	Once per job, at allocation
Controller Epilog	The Slurm controller	Once per job, at completion

A worker script runs once for each node in the job’s allocation, and a controller script runs once per job however many nodes it uses. On a four-node job the Worker Prolog runs four times and the Controller Prolog runs once. Anything that must happen exactly once, such as an API call or a database row, belongs in a controller slot.

Goal	Slot
Set up per-node scratch space, warm a cache, check hardware health	Worker Prolog
Delete per-node temporary files, flush node-local logs to shared storage	Worker Epilog
Record a job in an external system, notify a webhook, enforce a policy	Controller Prolog
Post job results, send a completion notification, aggregate accounting	Controller Epilog

Add a script

1. Select Actions, then Slurm Configuration, then the Prolog & Epilog tab.
2. Paste your script into the box for the slot. It must start with a shebang, or the dialog rejects it with “<Slot> must start with a shebang, for example #!/bin/bash”.
3. Select Apply Configuration.

Running jobs carry on as they are, and the next job picks up the new scripts. Each script must stay under 16 KB. Pasting from Windows is fine, because line endings are converted for you. To remove a script, clear its box and apply.

You can also add scripts while you create the cluster, under Advanced Settings, then Slurm Configuration (Optional). See Change the Slurm configuration.

For anything substantial, keep the real logic on shared storage and make the script in the console a thin wrapper. You then change the logic by editing the file on shared storage, and leave the configuration as you applied it. Write the wrapper so it still exits cleanly when the file is missing:

#!/bin/bash
[ -x /pfs/ops/job-prolog.sh ] && /pfs/ops/job-prolog.sh
exit 0

Write a safe script

A prolog or epilog runs on the host as root, outside any container the job starts. Quote variables, and keep rm -rf away from any path that could be empty.

End every Worker Prolog with exit 0. A non-zero exit from a Worker Prolog marks that node DRAIN, and the node stops accepting work. A prolog with a bug can therefore take the whole cluster out of service, one node per job. Leave out set -e, because one failing command would then drain the node:

#!/bin/bash
set -uo pipefail
mkdir -p /tmp/job_${SLURM_JOB_ID} 2>/dev/null || true
exit 0

To have a health check take a bad node out of service, exit non-zero on purpose and log why:

#!/bin/bash
if ! nvidia-smi -L >/dev/null 2>&1; then
  logger -t slurm-prolog "GPU enumeration failed on $(hostname); draining"
  exit 1
fi
exit 0

Bring a drained node back with scontrol update nodename=<node> state=resume, or select Restart All Workers from the Actions menu. See Update a cluster.

Keep these points in mind as you write:

• Set the job’s environment elsewhere. A script runs outside the job’s container, so the container’s contents and the job’s environment variables are out of its reach. Slurm’s TaskProlog is the hook that reaches into the container, and the platform manages it. Set environment variables in the job script, or bake them into the image.
• Keep scripts fast. A prolog runs before every job on every node, and the job waits for it. A 10-second prolog adds 10 seconds to every job’s start-up. Push slow work into the job itself.
• Keep secrets out. The configuration is stored with the cluster and shown again in the dialog. Read secrets from a file on mounted storage instead.
• Make scripts safe to repeat. A script can run again, sometimes at the same time on the same node for different jobs. Use mkdir -p, guard commands with || true, and name temporary paths by job ID.
• Test for storage. A script may run before a mount is ready, so check that the path exists before you use it.
• Write scripts to run unattended, because a script has no terminal to prompt for input.
• Send output to a log. The output of a prolog or epilog stays out of the job’s own output file, so send it to the node’s syslog with logger, or append it to a file on shared storage.

Slurm exports job details to these scripts. The commonly useful variables are:

Variable	Available in	Meaning
SLURM_JOB_ID	All slots	The job’s ID
SLURM_JOB_USER	All slots	The user who submitted the job
SLURM_JOB_NAME	Controller slots	The job name
SLURM_JOB_PARTITION	All slots	The partition the job is in
SLURM_JOB_NODELIST	Controller slots	The compact list of allocated nodes
SLURM_JOB_NUM_NODES	All slots	The node count
SLURM_NODEID	Worker slots	The node’s index in the allocation
SLURM_JOB_GPUS	Worker slots	The GPU indices allocated on the node
SLURM_CLUSTER_NAME	All slots	The Slurm cluster name
SLURM_JOB_ACCOUNT	All slots	The accounting account, when set
SLURM_JOB_EXIT_CODE	Epilog slots	The job’s exit code

Check what your Slurm version provides in SchedMD’s prolog and epilog guide.

Run the worker prolog at allocation

By default Slurm runs the prolog when the first job step launches. If your prolog prepares something the allocation itself depends on, add this line to the extra slurm.conf:

PrologFlags=Alloc

The prolog then runs at resource allocation instead. You can set PrologFlags yourself. The platform manages the Prolog setting.

Start from a worked example

Per-job scratch directory

Worker Prolog:

#!/bin/bash
set -uo pipefail
SCRATCH="/tmp/slurm-${SLURM_JOB_ID}"
mkdir -p "$SCRATCH" || true
chown "${SLURM_JOB_USER}" "$SCRATCH" 2>/dev/null || true
chmod 700 "$SCRATCH" 2>/dev/null || true
exit 0

Worker Epilog:

#!/bin/bash
set -uo pipefail
SCRATCH="/tmp/slurm-${SLURM_JOB_ID}"
# Guard the path: never rm -rf an empty variable
if [ -n "${SLURM_JOB_ID:-}" ] && [ -d "$SCRATCH" ]; then
  rm -rf "$SCRATCH" || true
fi
exit 0

GPU health check before every job

This Worker Prolog drains a node that lists fewer GPUs than expected, so jobs are scheduled only onto healthy hardware. Test it on one node first: point a partition at a single node, run a job there, confirm the behaviour, and then apply it to the whole cluster. A check with the wrong threshold drains every node it touches.

#!/bin/bash
set -uo pipefail
EXPECTED=8
FOUND=$(nvidia-smi -L 2>/dev/null | wc -l)
if [ "$FOUND" -lt "$EXPECTED" ]; then
  logger -t slurm-prolog "job ${SLURM_JOB_ID}: only ${FOUND}/${EXPECTED} GPUs visible on $(hostname); draining"
  exit 1
fi
exit 0

Clear the GPU state between jobs

A crashed job can leave a process holding GPU memory and cause a false out-of-memory error in the next job. This Worker Epilog prevents it:

#!/bin/bash
set -uo pipefail
# Kill anything the job left holding a GPU
nvidia-smi --gpu-reset >/dev/null 2>&1 || true
exit 0

Record jobs in a shared audit log

The Controller Prolog runs once per job, so there are no duplicate lines.

Controller Prolog:

#!/bin/bash
set -uo pipefail
LOG=/pfs/ops/job-audit.log
if [ -w "$(dirname "$LOG")" ]; then
  printf '%s START job=%s user=%s partition=%s nodes=%s\n' \
    "$(date -Is)" "$SLURM_JOB_ID" "$SLURM_JOB_USER" \
    "$SLURM_JOB_PARTITION" "$SLURM_JOB_NODELIST" >> "$LOG" 2>/dev/null || true
fi
exit 0

Controller Epilog:

#!/bin/bash
set -uo pipefail
LOG=/pfs/ops/job-audit.log
if [ -w "$(dirname "$LOG")" ]; then
  printf '%s END   job=%s exit=%s\n' \
    "$(date -Is)" "$SLURM_JOB_ID" "${SLURM_JOB_EXIT_CODE:-unknown}" >> "$LOG" 2>/dev/null || true
fi
exit 0

Copy node-local logs to shared storage

This Worker Epilog saves output that would otherwise be lost when the node is reused:

#!/bin/bash
set -uo pipefail
SRC="/tmp/slurm-${SLURM_JOB_ID}"
DEST="/pfs/logs/node-logs/${SLURM_JOB_ID}/$(hostname)"
if [ -d "$SRC" ] && [ -d /pfs ]; then
  mkdir -p "$DEST" 2>/dev/null || true
  cp -r "$SRC"/*.log "$DEST"/ 2>/dev/null || true
fi
exit 0

Know what the platform already runs

The platform runs its own scripts in the Worker Prolog slot, including a GPU health check, and your Lifecycle Script runs there too if you added one when you created the cluster. Your scripts are added alongside these, each in its own file, so your scripts and the platform’s stay intact side by side.

The platform manages Prolog, Epilog, PrologSlurmctld, EpilogSlurmctld, TaskProlog, and TaskEpilog, so the extra slurm.conf refuses them. Add your own scripts on the Prolog & Epilog tab instead.

The create-time Lifecycle Script runs once on each node after the cluster is created, while the Worker Prolog runs before every job. The Lifecycle Script keeps its own file, so a later Worker Prolog leaves it in place. You set a Lifecycle Script on the create form, under Advanced Settings, and it stays fixed after that. You can change a Worker Prolog whenever the cluster is Running. Use a Lifecycle Script for one-time node setup, such as installing a package, and a Worker Prolog for per-job work.

Check that the script runs

Submit a short job and look for the script’s effect:

sbatch --wrap='sleep 5' --partition=all

On the node the job landed on, for the scratch directory example:

ls -la /tmp/slurm-<jobid>
grep -i slurm-prolog /var/log/syslog | tail

Replace <jobid> with the ID that sbatch printed. To watch it live, open the Logs tab, choose the worker the job landed on in Select Replica, and turn on Auto Refresh. See See jobs and logs.

Clone Cluster carries all four scripts to the new cluster. See Clone a cluster.

If nodes start draining after a prolog change, see Troubleshooting jobs on TIR Slurm Cluster.