Logs
The Logs tab streams logs from the cluster's login node and worker nodes, so you can see what the Slurm daemons are doing without SSH.
Which Logs You Can Read
Select Replica lists one entry per readable component:
| Replica | What it is |
|---|---|
slurm-login | The login node — sshd and submission activity |
worker-0, worker-1, … | One per worker node — the slurmd log for that node |
Only the login node and workers are selectable. slurmctld, the accounting database and the REST
service do not appear. For controller-side questions — why a job is pending, why a partition was
rejected — use scontrol and sacct on the login node, described
below.
The terminal panel is titled with the underlying pod name, e.g.
slinky-slurm-492-worker-slinky-0, and lines are numbered.
Controls
| Control | Options | What it does |
|---|---|---|
| Select Replica | slurm-login, worker-N | Which component's log to show. The first is selected for you |
| Refresh | — | Reload now |
| Auto Refresh | Disable (default), 5 Seconds, 10 Seconds, 30 Seconds, 1 Minute | Poll continuously. The view does not flash while polling |
| Filter By | Last N Lines (default), Last N Seconds, None | Intended to limit how much is returned |
| Value of N | Default 1000 | The N for the filter. Confirm an edit with the ✓ button, or press Refresh |
| Download | — | Save the log to a file |
On Slurm Clusters the Filter By and Value of N controls may not actually trim what the API returns — you can receive the full available log regardless of the value. If you need a precise window, read the log over SSH:
tail -n 200 /var/log/slurm/slurmd.log
journalctl -u sshd --since "10 minutes ago"
Set the replica to the worker your job lands on, Auto Refresh to 5 Seconds, then submit. You
see the node's slurmd activity as it happens — the fastest way to confirm a
prolog script is running. Set it back to
Disable afterwards.
What to Look For
On a worker (worker-N)
| Symptom in the log | Usually means |
|---|---|
error: Prolog failed, or the node draining right after a job starts | Your Worker Prolog exited non-zero |
Munge decode failed, authentication errors | The node is out of sync with the controller. Try Restart All Workers |
error: Node configuration differs from hardware | The node's real resources do not match what Slurm expects — usually after a hardware change |
slurmd started repeatedly | The pod is restarting in a loop |
Nothing at all, and the node is UNKNOWN | The worker is not up. Check the Nodes tab |
On the login node (slurm-login)
| Symptom | Usually means |
|---|---|
Connection closed by authenticating user root <ip> [preauth] | Failed SSH attempts. A public IP attracts constant background scanning — this is normal noise |
Invalid user <name> from <ip> | Same. Restrict the source range in your security group if it bothers you |
Accepted publickey for <user> | A successful login |
Failed publickey for janedoe | A member's key is not attached, or their group has no cluster access |
A cluster with a public IP will show a continuous stream of rejected login attempts from the internet. They are being rejected — password authentication is off and only your keys are accepted. It is not a sign of compromise, but it is a reason to keep the security group's source range as narrow as your team allows.
Download Logs
The download icon saves the current replica's log to a file named after its pod, for example
slinky-slurm-492-worker-slinky-0-logs.json. The file is JSON, not plain text — extract the lines
with jq if you want a flat log:
jq -r '.. | .logs? // empty | .[]' slinky-slurm-492-worker-slinky-0-logs.json > worker-0.log
The button is disabled until at least one line is on screen.
Restarting workers or the login service recreates the pods, and the previous pod's log is gone. If you are about to restart something to fix a problem, download the log first — otherwise you lose the evidence of what the problem was.
Job Output Is Not Here
This tab shows the Slurm daemons' logs. Your job's own stdout and stderr go to the files
your job script named:
#SBATCH --output=/pfs/logs/%x-%j.out
#SBATCH --error=/pfs/logs/%x-%j.err
tail -f /pfs/logs/train-12345.out
Output written to a node's local filesystem is lost when the pod is recreated — taking your only
record of the failure with it. %x is the job name and %j the job ID, which keeps runs from
overwriting each other.
The command-line alternatives
ssh root@<cluster-ip>
# Why is a job pending, and what did Slurm decide?
scontrol show job <jobid>
squeue -o "%.18i %.9P %.8T %.20R"
# What happened to a finished job?
sacct -j <jobid> --format=JobID,State,ExitCode,Elapsed,MaxRSS,NodeList
# Node-level detail, including drain reasons
sinfo -N -o "%N %t %E"
scontrol show node slinky-0
# The live configuration the controller is using
scontrol show config
sacct is the one to reach for after the fact — it reads the accounting database, so it answers "why
did last night's job fail?" long after the job has left the queue.