--- title: Logs sidebar_label: Logs --- import { FileText, RefreshCw, Download, Terminal, Filter } from 'react-feather'; # Logs The **Logs** tab streams logs from the cluster's login node and worker nodes, so you can see what the Slurm daemons are doing without SSH. }, { href: '#controls', label: 'Controls', icon: }, { href: '#what-to-look-for', label: 'What to look for', icon: }, { href: '#download-logs', label: 'Download', icon: }, { href: '#job-output-is-not-here', label: 'Job output is elsewhere', icon: }, ]} /> --- ## Which Logs You Can Read **Select Replica** lists one entry per readable component: | Replica | What it is | |---------|------------| | `slurm-login` | The login node — `sshd` and submission activity | | `worker-0`, `worker-1`, … | One per worker node — the `slurmd` log for that node | :::info The controller log is not exposed here Only the login node and workers are selectable. `slurmctld`, the accounting database and the REST service do not appear. For controller-side questions — why a job is pending, why a partition was rejected — use `scontrol` and `sacct` on the login node, described [below](#the-command-line-alternatives). ::: The terminal panel is titled with the underlying pod name, e.g. `slinky-slurm-492-worker-slinky-0`, and lines are numbered. --- ## Controls | Control | Options | What it does | |---------|---------|--------------| | **Select Replica** | `slurm-login`, `worker-N` | Which component's log to show. The first is selected for you | | Refresh | — | Reload now | | **Auto Refresh** | `Disable` (default), `5 Seconds`, `10 Seconds`, `30 Seconds`, `1 Minute` | Poll continuously. The view does not flash while polling | | **Filter By** | `Last N Lines` (default), `Last N Seconds`, `None` | Intended to limit how much is returned | | **Value of N** | Default `1000` | The N for the filter. Confirm an edit with the ✓ button, or press Refresh | | Download | — | Save the log to a file | :::warning Treat the line-count filter as a hint, not a guarantee On Slurm Clusters the **Filter By** and **Value of N** controls may not actually trim what the API returns — you can receive the full available log regardless of the value. If you need a precise window, read the log over SSH: ```bash tail -n 200 /var/log/slurm/slurmd.log journalctl -u sshd --since "10 minutes ago" ``` ::: :::tip Auto Refresh plus a worker replica is the right setup while testing a prolog Set the replica to the worker your job lands on, **Auto Refresh** to `5 Seconds`, then submit. You see the node's `slurmd` activity as it happens — the fastest way to confirm a [prolog script](/docs/tir/SlurmCluster/slurm-configuration/prolog-epilog) is running. Set it back to `Disable` afterwards. ::: --- ## What to Look For ### On a worker (`worker-N`) | Symptom in the log | Usually means | |--------------------|---------------| | `error: Prolog failed`, or the node draining right after a job starts | Your [Worker Prolog](/docs/tir/SlurmCluster/slurm-configuration/prolog-epilog) exited non-zero | | `Munge decode failed`, authentication errors | The node is out of sync with the controller. Try [Restart All Workers](/docs/tir/SlurmCluster/manage/actions#restart-actions) | | `error: Node configuration differs from hardware` | The node's real resources do not match what Slurm expects — usually after a hardware change | | `slurmd started` repeatedly | The pod is restarting in a loop | | Nothing at all, and the node is `UNKNOWN` | The worker is not up. Check the [Nodes tab](/docs/tir/SlurmCluster/manage/nodes) | ### On the login node (`slurm-login`) | Symptom | Usually means | |---------|---------------| | `Connection closed by authenticating user root [preauth]` | Failed SSH attempts. A public IP attracts constant background scanning — this is normal noise | | `Invalid user from ` | Same. Restrict the source range in your [security group](/docs/tir/SlurmCluster/manage/network-security#security-groups) if it bothers you | | `Accepted publickey for ` | A successful login | | `Failed publickey for janedoe` | A [member](/docs/tir/SlurmCluster/connect/login-user-management)'s key is not attached, or their group has no cluster access | :::note Background SSH noise is expected A cluster with a public IP will show a continuous stream of rejected login attempts from the internet. They are being rejected — password authentication is off and only your keys are accepted. It is not a sign of compromise, but it is a reason to keep the security group's source range as narrow as your team allows. ::: --- ## Download Logs The download icon saves the current replica's log to a file named after its pod, for example `slinky-slurm-492-worker-slinky-0-logs.json`. The file is JSON, not plain text — extract the lines with `jq` if you want a flat log: ```bash jq -r '.. | .logs? // empty | .[]' slinky-slurm-492-worker-slinky-0-logs.json > worker-0.log ``` The button is disabled until at least one line is on screen. :::tip Download before you restart Restarting workers or the login service recreates the pods, and the previous pod's log is gone. If you are about to restart something to fix a problem, download the log first — otherwise you lose the evidence of what the problem was. ::: --- ## Job Output Is Not Here This tab shows the Slurm **daemons'** logs. Your job's own `stdout` and `stderr` go to the files your job script named: ```bash #SBATCH --output=/pfs/logs/%x-%j.out #SBATCH --error=/pfs/logs/%x-%j.err ``` ```bash tail -f /pfs/logs/train-12345.out ``` :::danger Write job output to shared storage, always Output written to a node's local filesystem is lost when the pod is recreated — taking your only record of the failure with it. `%x` is the job name and `%j` the job ID, which keeps runs from overwriting each other. ::: ### The command-line alternatives ```bash ssh root@ ``` ```bash # Why is a job pending, and what did Slurm decide? scontrol show job squeue -o "%.18i %.9P %.8T %.20R" # What happened to a finished job? sacct -j --format=JobID,State,ExitCode,Elapsed,MaxRSS,NodeList # Node-level detail, including drain reasons sinfo -N -o "%N %t %E" scontrol show node slinky-0 # The live configuration the controller is using scontrol show config ``` `sacct` is the one to reach for after the fact — it reads the accounting database, so it answers "why did last night's job fail?" long after the job has left the queue. --- ## Related Resources - [Jobs tab](/docs/tir/SlurmCluster/manage/jobs) - [Nodes and GPU health](/docs/tir/SlurmCluster/manage/nodes) - [Prolog and epilog scripts](/docs/tir/SlurmCluster/slurm-configuration/prolog-epilog) - [Troubleshoot jobs](/docs/tir/SlurmCluster/troubleshoot/jobs)