Prolog and Epilog Scripts

Prolog and epilog scripts are shell scripts Slurm runs around every job. Use them to prepare the environment, stage data, record accounting information, or clean up after a job — automatically, for every job on the cluster, without touching anyone's job script.

The four slotsAdd a scriptWriting a safe scriptAvailable environmentWorked examplesPlatform scripts

---

The Four Script Slots

The Prolog & Epilog tab of the Slurm Configuration dialog gives you four independent scripts.

Slot	Runs on	Runs when	Slurm directive
Worker Prolog	Every worker node in the allocation	Before each job starts on that node	Prolog
Worker Epilog	Every worker node in the allocation	After each job ends on that node	Epilog
Controller Prolog	The Slurm controller	Once per job, at allocation	PrologSlurmctld
Controller Epilog	The Slurm controller	Once per job, at completion	EpilogSlurmctld

The console repeats this next to each box:

• Runs on Slurm workers before each job starts.
• Runs on Slurm workers after each job ends. Use for cleanup.
• Runs on slurmctld at job allocation. Central job setup.
• Runs on slurmctld at job completion. Central job teardown.

Which slot do I want?

Goal	Slot
Set up per-node scratch space, warm a cache, check hardware health before the job	Worker Prolog
Delete per-node temporary files, flush node-local logs to shared storage	Worker Epilog
Record a job in an external system, notify a webhook, enforce a policy centrally	Controller Prolog
Post job results to a tracker, send a completion notification, aggregate accounting	Controller Epilog
Do something inside the job's container	Not available — see the container note

Worker scripts run once per node, controller scripts once per job

On a 4-node job, the Worker Prolog runs 4 times (once per node) and the Controller Prolog runs once. Anything that must happen exactly once — an API call, a database row — belongs in the controller slot.

---

Add a Script

1. Open the cluster and confirm the status is Running.
2. Actions → Slurm Configuration → Prolog & Epilog tab.
3. Paste your script into the relevant box. It must start with a shebang.
4. Click Apply Configuration.

The dialog reminds you what applying means, including:

Prolog and epilog changes take effect on the next job

Jobs already running are unaffected. The next job submitted picks up the new scripts.

Requirements

Requirement	Detail
Shebang	The first characters must be #!. Missing it fails with "<Slot> must start with a shebang, for example #!/bin/bash"
Size	16 KB per script
Line endings	Normalised to LF for you — pasting from Windows is safe
Removing a script	Clear the box and apply. An empty script is removed entirely

Keep long logic in a file on shared storage

The 16 KB limit is generous for a hook but not for a program. For anything substantial, put the real script on your PFS or SFS mount and make the prolog a thin, defensive wrapper:

#!/bin/bash
[ -x /pfs/ops/job-prolog.sh ] && /pfs/ops/job-prolog.sh
exit 0

You can then iterate on the logic without re-applying the cluster configuration — but note the wrapper must tolerate the file being missing or the storage being unmounted.

---

Writing a Safe Script

A non-zero exit from a Worker Prolog drains the node

This is the most important rule on this page. If your Worker Prolog exits non-zero, Slurm marks that node DRAIN and it stops accepting work. A prolog with a bug can take your whole cluster out of service, one node per job, until you notice it in the Nodes tab.

Always end with an explicit success, and never let a failing command decide the exit code:

#!/bin/bash
# Do not use `set -e` here — a single failing command would drain the node.
set -uo pipefail

# ... your work, each step tolerant of failure ...
mkdir -p /tmp/job_${SLURM_JOB_ID} 2>/dev/null || true

exit 0

If you genuinely want a health check to remove a bad node from service, exiting non-zero is the mechanism — just make it deliberate, and log why:

#!/bin/bash
if ! nvidia-smi -L >/dev/null 2>&1; then
  logger -t slurm-prolog "GPU enumeration failed on $(hostname); draining"
  exit 1
fi
exit 0

Recover a drained node with scontrol update nodename=<node> state=resume, or Restart All Workers from the Actions menu.

Worker scripts run on the host, not in your container

Worker and controller scripts run on the host as root, outside any Enroot/Pyxis container your job starts. So:

• bash is available — #!/bin/bash is fine.
• You cannot see or modify the inside of the job's container from a prolog.
• Environment variables you export in a prolog do not reach the job's processes. Slurm's TaskProlog is the only hook that crosses into the container, and it is platform-managed.

To set environment variables for a job, do it in the job script, or bake them into the image.

Keep them fast

A prolog runs before every job on every node, and the job waits for it. A ten-second prolog adds ten seconds to every job's start-up, and forty seconds of wall clock across a four-node allocation. Push anything slow into the job itself or into a scheduled task elsewhere.

Other hazards

Hazard	Guidance
Running as root	The script has full node privileges. Quote variables, avoid rm -rf with an unquoted or possibly-empty path
Secrets	Do not embed credentials — the configuration is stored with the cluster and echoed back into the dialog. Read secrets from a file on your mounted storage instead
Storage assumptions	A prolog may run before you expect a mount to be ready. Test for the path rather than assuming it
Idempotency	Scripts run repeatedly, sometimes concurrently on the same node for different jobs. Use mkdir -p, guard with `
Interactive commands	There is no TTY. Nothing may prompt or block on input

---

Available Environment

Slurm exports job context into prolog and epilog scripts. The commonly useful ones:

Variable	Available in	Meaning
SLURM_JOB_ID	all	The job's ID
SLURM_JOB_USER	all	User who submitted the job
SLURM_JOB_NAME	controller slots	Job name
SLURM_JOB_PARTITION	all	Partition the job is in
SLURM_JOB_NODELIST	controller slots	Compact list of allocated nodes
SLURM_JOB_NUM_NODES	all	Node count
SLURM_NODEID	worker slots	This node's index in the allocation
SLURM_JOB_GPUS	worker slots	GPU indices allocated on this node
SLURM_CLUSTER_NAME	all	The Slurm cluster name
SLURM_JOB_ACCOUNT	all	Accounting account, when set
SLURM_JOB_EXIT_CODE	epilog slots	The job's exit code

Check what your Slurm version provides with SchedMD's Prolog and Epilog guide.

Prolog output does not appear in the job's log

stdout from a prolog is not written to the job's --output file. Use logger to send it to the node's syslog, or append to a file on shared storage — then read it from the Logs tab or over SSH.

PrologFlags changes when the prolog runs

By default Slurm runs the Prolog when the first job step launches. If your prolog prepares something the allocation itself depends on, set the flag in extra slurm.conf:

PrologFlags=Alloc

This runs the prolog at resource allocation instead. PrologFlags is yours to set — the Prolog directive itself is platform-managed.

---

Worked Examples

Per-job scratch directory

Worker Prolog:

#!/bin/bash
set -uo pipefail
SCRATCH="/tmp/slurm-${SLURM_JOB_ID}"
mkdir -p "$SCRATCH" || true
chown "${SLURM_JOB_USER}" "$SCRATCH" 2>/dev/null || true
chmod 700 "$SCRATCH" 2>/dev/null || true
exit 0

Worker Epilog:

#!/bin/bash
set -uo pipefail
SCRATCH="/tmp/slurm-${SLURM_JOB_ID}"
# Guard the path: never rm -rf an empty variable
if [ -n "${SLURM_JOB_ID:-}" ] && [ -d "$SCRATCH" ]; then
  rm -rf "$SCRATCH" || true
fi
exit 0

GPU health gate before every job

Worker Prolog — deliberately drains a node whose GPUs are not enumerable, so jobs are not scheduled onto broken hardware:

#!/bin/bash
set -uo pipefail

EXPECTED=8
FOUND=$(nvidia-smi -L 2>/dev/null | wc -l)

if [ "$FOUND" -lt "$EXPECTED" ]; then
  logger -t slurm-prolog "job ${SLURM_JOB_ID}: only ${FOUND}/${EXPECTED} GPUs visible on $(hostname); draining"
  exit 1
fi
exit 0

Test a draining prolog on one node first

Point a partition at a single node, run a job there, confirm the behaviour, and only then apply it cluster-wide. A health gate with the threshold set wrong drains every node it touches.

Clear the GPU state between jobs

Worker Epilog:

#!/bin/bash
set -uo pipefail
# Kill anything the job left holding a GPU
nvidia-smi --gpu-reset >/dev/null 2>&1 || true
exit 0

Record jobs to a shared audit log

Controller Prolog — runs once per job, so no duplicates:

#!/bin/bash
set -uo pipefail
LOG=/pfs/ops/job-audit.log
if [ -w "$(dirname "$LOG")" ]; then
  printf '%s START job=%s user=%s partition=%s nodes=%s\n' \
    "$(date -Is)" "$SLURM_JOB_ID" "$SLURM_JOB_USER" \
    "$SLURM_JOB_PARTITION" "$SLURM_JOB_NODELIST" >> "$LOG" 2>/dev/null || true
fi
exit 0

Controller Epilog:

#!/bin/bash
set -uo pipefail
LOG=/pfs/ops/job-audit.log
if [ -w "$(dirname "$LOG")" ]; then
  printf '%s END   job=%s exit=%s\n' \
    "$(date -Is)" "$SLURM_JOB_ID" "${SLURM_JOB_EXIT_CODE:-unknown}" >> "$LOG" 2>/dev/null || true
fi
exit 0

Flush node-local logs to shared storage

Worker Epilog — saves output that would otherwise be lost when the node is reused:

#!/bin/bash
set -uo pipefail
SRC="/tmp/slurm-${SLURM_JOB_ID}"
DEST="/pfs/logs/node-logs/${SLURM_JOB_ID}/$(hostname)"
if [ -d "$SRC" ] && [ -d /pfs ]; then
  mkdir -p "$DEST" 2>/dev/null || true
  cp -r "$SRC"/*.log "$DEST"/ 2>/dev/null || true
fi
exit 0

---

Platform-Managed Scripts

Your scripts are added alongside the platform's, never instead of them. Each one lands in its own file, so nothing you write can disable or overwrite platform behaviour — and nothing the platform does overwrites yours.

The platform already runs, in the Worker Prolog hook:

Script	What it does
GPU health hook	Writes the GPU-to-job mapping that per-GPU DCGM metrics in the Nodes tab depend on
Enroot mount setup	Writes the Enroot mount rule that makes Slurm's config cache readable inside Pyxis containers
Lifecycle script	Your create-time Lifecycle Script, if you supplied one. It keeps its own file, so a Worker Prolog does not replace it
Storage monitoring hook	On some clusters, a hook that tags jobs for storage accounting

The three controller and worker epilog hooks contain only your script unless you set one.

You cannot set the Slurm directives directly

Prolog, Epilog, PrologSlurmctld, EpilogSlurmctld, TaskProlog and TaskEpilog are platform-managed keys in slurm.conf — the platform points them at the script directories it manages. The Prolog & Epilog tab is the supported way to add your own, and it is a superset of what setting the directive would give you.

Lifecycle Script vs Worker Prolog

Both let you run your own shell code, but they are not the same thing:

	Lifecycle Script	Worker Prolog
When	Once, on each node, after the cluster is created	Before every job, on every node in the allocation
Set from	Create form → Advanced Settings → Lifecycle Script → Add Script	Slurm Configuration → Prolog & Epilog
Changeable later	No — it is a create-time field	Yes, any time the cluster is Running
Use for	One-time node setup: install a package, write a config file	Per-job setup and teardown

---

Cloning and Verification

Clone Cluster carries all four prolog/epilog scripts and the extra slurm.conf to the new cluster. It does not carry partitions.

To confirm a script is live, submit a trivial job and look for its effect:

sbatch --wrap='sleep 5' --partition=all

# On the node the job landed on
ls -la /tmp/slurm-<jobid>
grep -i slurm-prolog /var/log/syslog | tail

The Logs tab shows controller and worker output; pick the replica with Select Replica and turn on Auto Refresh while you test.

---

Related Resources

• Slurm Configuration overview
• Extra slurm.conf settings
• Manage partitions
• Containers with Enroot and Pyxis
• Nodes and GPU health
• Prolog and Epilog guide (SchedMD)