See jobs and logs

Check the queue and read what the Slurm daemons are doing straight from the console. The Jobs tab is squeue in the browser, and the Logs tab shows the login node and worker logs.

Before you begin

You need a cluster that is Running. The Jobs and Logs tabs open once the workers have joined. See Cluster statuses and actions for which tabs open in each status.

Read the Jobs tab

Under Training Cluster in the TIR sidebar, open the cluster, then open Jobs. The tab reads live from the Slurm controller each time it loads; select refresh to reload it.

Five totals count jobs: Running, Pending, Completed, Failed, and Unknown. Slurm has more states than these five. A job that is cancelled, timed out, suspended, preempted, configuring, or completing is counted and shown under Unknown. Run squeue or sacct for the exact state.

The job table has these columns:

Column	What it shows
Job ID	Slurm’s job ID. Use it with scontrol show job and sacct.
Name	The job name, from --job-name or the script file name.
User	Who submitted the job. With Login User Management this is a real person; otherwise everything is root.
Partition	The partition the job is in.
Nodes	The nodes allocated to the job.
GPUs	The GPUs allocated.
Run Time	The time since the job started.
Time Limit	The maximum runtime: UNLIMITED, or HH:MM:SS or D-HH:MM:SS.
State	Running, Pending, Completed, Failed, or Unknown.
Priority	The scheduling priority, as a bar and a number.

Compare the number beside the priority bar rather than the bar itself when you rank two pending jobs, because Slurm’s priorities are very large numbers.

Do the rest from the command line

The Jobs tab is read-only, and its rows stay in the order the controller reports them. To cancel a job, run scancel <jobid> over SSH. To hold or requeue one, use scontrol hold or scontrol requeue. Read the job’s output in the --output file on shared storage. The tab shows what the controller still holds, so use sacct for older jobs.

By default Slurm drops finished jobs from squeue quickly, so they leave this tab too. To keep them longer, set MinJobAge in the extra slurm.conf, for example MinJobAge=300. See Change the Slurm configuration.

Connect to the login node:

ssh root@<cluster-ip>

Replace <cluster-ip> with the address from Connection Details. Then inspect the queue and jobs:

squeue
squeue -u $USER
squeue -p all -t PENDING
squeue -o "%.18i %.30j %.8u %.8T %.10M %.6D %R"
scontrol show job <jobid>
sstat -j <jobid>

These show everything, only your jobs, pending jobs in one partition, custom columns with the reason, full detail for one job, and the live resource use of a running job. For history and accounting, which sacct reads long after a job leaves squeue:

sacct -j <jobid> --format=JobID,JobName,State,ExitCode,Elapsed,MaxRSS,AllocTRES%40
sacct -u $USER -S today
sacct -S 2026-09-01 -E 2026-09-10 --format=JobID,JobName,State,Elapsed

To control jobs:

scancel <jobid>
scancel -u $USER
scancel --state=PENDING -u $USER
scontrol hold <jobid>
scontrol release <jobid>
scontrol update job=<jobid> TimeLimit=04:00:00
scontrol update job=<jobid> Partition=debug

scancel -u cancels everything the user owns. If everyone logs in as root, scancel -u root cancels every job on the cluster, so cancel by job ID and give your team their own identities. See Manage login users.

For why a job is pending or failed, see Troubleshooting jobs on TIR Slurm Cluster.

Read the Logs tab

Open the Logs tab and choose a log under Select Replica: slurm-login, the login node, or one worker-N for each worker. The controller, the accounting database, and the REST service are left out of this list, so for controller-side questions, such as why a job is pending, use scontrol and sacct on the login node.

You can refresh the log automatically, every 5 seconds to 1 minute, with Auto Refresh. Filter By keeps the last N lines, 1,000 by default, or the last N seconds, and the download button saves the log you are reading.

Treat Filter By and Value of N as a rough guide, because the log shown can run longer than N lines or seconds. To read an exact window, read the log over SSH:

tail -n 200 /var/log/slurm/slurmd.log
journalctl -u sshd --since "10 minutes ago"

To confirm a prolog runs, choose the worker your job lands on, set Auto Refresh to 5 Seconds, then submit the job. Set it back to Disable afterwards.

Download a log

Download saves the current replica’s log as a JSON file named after the replica, such as slinky-slurm-492-worker-slinky-0-logs.json. It is available once at least one line is on screen. To get a flat text log from the file:

jq -r '.. | .logs? // empty | .[]' <file>.json > worker-0.log

Replace <file> with the name of the downloaded file. Download a log before you restart anything that recreates its node, because the earlier log is gone once the node is replaced.

Know what a worker log line means

In a worker’s log	It usually means
error: Prolog failed, or the node draining right after a job starts	Your Worker Prolog exited non-zero.
Munge decode failed, or authentication errors	The node is out of sync with the controller. Try Restart All Workers.
error: Node configuration differs from hardware	The node’s real resources differ from what Slurm expects, usually after a hardware change.
slurmd started repeating	The node is restarting in a loop.
An empty log, with the node Unknown	The worker is offline. Check the Nodes tab.

Know what a login node log line means

In the login node’s log	It usually means
Accepted publickey for <user>	A successful login.
Failed publickey for <name>	That member’s key is missing, or their group lacks cluster access.
Connection closed by authenticating user root <ip> [preauth]	A rejected attempt. Any public IP attracts background scanning.
Invalid user <name> from <ip>	The same kind of rejected scanning attempt.

Rejected attempts are normal background noise: password login is off and only your keys are accepted. Narrow the source range of the SSH security group rather than removing its rule, because Any Network on port 22 is what fills the log with scanning. See Manage network and security.

Find your job’s own output

The Logs tab shows the Slurm daemons’ logs. A job’s stdout and stderr go to the files the job script names. Point them at shared storage, because output written to a node-local path is lost when the node is recreated, and it is often the only record of the failure:

#SBATCH --output=/pfs/logs/%x-%j.out
#SBATCH --error=/pfs/logs/%x-%j.err

%x is the job name and %j is the job ID, which keeps runs from overwriting each other. Follow a running job with tail -f /pfs/logs/train-12345.out.

For controller-side detail from the login node:

scontrol show job <jobid>
sacct -j <jobid> --format=JobID,State,ExitCode,Elapsed,MaxRSS,NodeList
sinfo -N -o "%N %t %E"
scontrol show node slinky-0
scontrol show config

To see how busy the GPUs are over time, continue to Monitor a cluster.