--- title: "See jobs and logs" description: "Read a TIR Slurm Cluster's job queue and its login and worker logs from the Jobs and Logs tabs, and use the command line for everything beyond them." content_type: "how-to" product: "tir-slurm-cluster" platform: "TIR" url: "/tir/tir-slurm-cluster/how-to/see-jobs-and-logs/" topic: ["monitor"] --- # See jobs and logs Check the queue and read what the Slurm daemons are doing straight from the console. The **Jobs** tab is `squeue` in the browser, and the **Logs** tab shows the login node and worker logs. ## Before you begin You need a cluster that is **Running**. The **Jobs** and **Logs** tabs open once the workers have joined. See [Cluster statuses and actions](/tir/tir-slurm-cluster/concepts/cluster-statuses-and-actions/) for which tabs open in each status. ## Read the Jobs tab Under **Training Cluster** in the TIR sidebar, open the cluster, then open **Jobs**. The tab reads live from the Slurm controller each time it loads; select refresh to reload it. Five totals count jobs: **Running**, **Pending**, **Completed**, **Failed**, and **Unknown**. Slurm has more states than these five. A job that is cancelled, timed out, suspended, preempted, configuring, or completing is counted and shown under **Unknown**. Run `squeue` or `sacct` for the exact state. The job table has these columns: | Column | What it shows | | -------------- | -------------------------------------------------------------------------------------------------------- | | **Job ID** | Slurm's job ID. Use it with `scontrol show job` and `sacct`. | | **Name** | The job name, from `--job-name` or the script file name. | | **User** | Who submitted the job. With Login User Management this is a real person; otherwise everything is `root`. | | **Partition** | The partition the job is in. | | **Nodes** | The nodes allocated to the job. | | **GPUs** | The GPUs allocated. | | **Run Time** | The time since the job started. | | **Time Limit** | The maximum runtime: `UNLIMITED`, or `HH:MM:SS` or `D-HH:MM:SS`. | | **State** | **Running**, **Pending**, **Completed**, **Failed**, or **Unknown**. | | **Priority** | The scheduling priority, as a bar and a number. | Compare the number beside the priority bar rather than the bar itself when you rank two pending jobs, because Slurm's priorities are very large numbers. ### Do the rest from the command line The **Jobs** tab is read-only, and its rows stay in the order the controller reports them. To cancel a job, run `scancel ` over SSH. To hold or requeue one, use `scontrol hold` or `scontrol requeue`. Read the job's output in the `--output` file on shared storage. The tab shows what the controller still holds, so use `sacct` for older jobs. By default Slurm drops finished jobs from `squeue` quickly, so they leave this tab too. To keep them longer, set MinJobAge in the extra `slurm.conf`, for example MinJobAge=300. See [Change the Slurm configuration](/tir/tir-slurm-cluster/how-to/change-the-slurm-configuration/). Connect to the login node: ```bash ssh root@ ``` Replace `` with the address from **Connection Details**. Then inspect the queue and jobs: ```bash squeue squeue -u $USER squeue -p all -t PENDING squeue -o "%.18i %.30j %.8u %.8T %.10M %.6D %R" scontrol show job sstat -j ``` These show everything, only your jobs, pending jobs in one partition, custom columns with the reason, full detail for one job, and the live resource use of a running job. For history and accounting, which `sacct` reads long after a job leaves `squeue`: ```bash sacct -j --format=JobID,JobName,State,ExitCode,Elapsed,MaxRSS,AllocTRES%40 sacct -u $USER -S today sacct -S 2026-09-01 -E 2026-09-10 --format=JobID,JobName,State,Elapsed ``` To control jobs: ```bash scancel scancel -u $USER scancel --state=PENDING -u $USER scontrol hold scontrol release scontrol update job= TimeLimit=04:00:00 scontrol update job= Partition=debug ``` `scancel -u` cancels everything the user owns. If everyone logs in as `root`, `scancel -u root` cancels every job on the cluster, so cancel by job ID and give your team their own identities. See [Manage login users](/tir/tir-slurm-cluster/how-to/manage-login-users/). For why a job is pending or failed, see [Troubleshooting jobs on TIR Slurm Cluster](/tir/tir-slurm-cluster/troubleshooting/jobs/). ## Read the Logs tab Open the **Logs** tab and choose a log under **Select Replica**: `slurm-login`, the login node, or one `worker-N` for each worker. The controller, the accounting database, and the REST service are left out of this list, so for controller-side questions, such as why a job is pending, use `scontrol` and `sacct` on the login node. You can refresh the log automatically, every 5 seconds to 1 minute, with **Auto Refresh**. **Filter By** keeps the last N lines, 1,000 by default, or the last N seconds, and the download button saves the log you are reading. Treat **Filter By** and **Value of N** as a rough guide, because the log shown can run longer than N lines or seconds. To read an exact window, read the log over SSH: ```bash tail -n 200 /var/log/slurm/slurmd.log journalctl -u sshd --since "10 minutes ago" ``` To confirm a prolog runs, choose the worker your job lands on, set **Auto Refresh** to **5 Seconds**, then submit the job. Set it back to **Disable** afterwards. ### Download a log Download saves the current replica's log as a JSON file named after the replica, such as `slinky-slurm-492-worker-slinky-0-logs.json`. It is available once at least one line is on screen. To get a flat text log from the file: ```bash jq -r '.. | .logs? // empty | .[]' .json > worker-0.log ``` Replace `` with the name of the downloaded file. Download a log before you restart anything that recreates its node, because the earlier log is gone once the node is replaced. ### Know what a worker log line means | In a worker's log | It usually means | | --------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ | | `error: Prolog failed`, or the node draining right after a job starts | Your Worker Prolog exited non-zero. | | `Munge decode failed`, or authentication errors | The node is out of sync with the controller. Try **Restart All Workers**. | | `error: Node configuration differs from hardware` | The node's real resources differ from what Slurm expects, usually after a hardware change. | | `slurmd started` repeating | The node is restarting in a loop. | | An empty log, with the node **Unknown** | The worker is offline. Check the **Nodes** tab. | ### Know what a login node log line means | In the login node's log | It usually means | | -------------------------------------------------------------- | ------------------------------------------------------------------ | | `Accepted publickey for ` | A successful login. | | `Failed publickey for ` | That member's key is missing, or their group lacks cluster access. | | `Connection closed by authenticating user root [preauth]` | A rejected attempt. Any public IP attracts background scanning. | | `Invalid user from ` | The same kind of rejected scanning attempt. | Rejected attempts are normal background noise: password login is off and only your keys are accepted. Narrow the source range of the SSH security group rather than removing its rule, because `Any Network` on port 22 is what fills the log with scanning. See [Manage network and security](/tir/tir-slurm-cluster/how-to/manage-network-and-security/). ### Find your job's own output The **Logs** tab shows the Slurm daemons' logs. A job's `stdout` and `stderr` go to the files the job script names. Point them at shared storage, because output written to a node-local path is lost when the node is recreated, and it is often the only record of the failure: ```bash #SBATCH --output=/pfs/logs/%x-%j.out #SBATCH --error=/pfs/logs/%x-%j.err ``` `%x` is the job name and `%j` is the job ID, which keeps runs from overwriting each other. Follow a running job with `tail -f /pfs/logs/train-12345.out`. For controller-side detail from the login node: ```bash scontrol show job sacct -j --format=JobID,State,ExitCode,Elapsed,MaxRSS,NodeList sinfo -N -o "%N %t %E" scontrol show node slinky-0 scontrol show config ``` To see how busy the GPUs are over time, continue to [Monitor a cluster](/tir/tir-slurm-cluster/how-to/monitor-a-cluster/).