--- title: "Troubleshooting jobs on TIR Slurm Cluster" description: "Fix TIR Slurm Cluster jobs that stay pending, fail at once, run out of GPU memory, run slowly, or vanish, from the reason Slurm reports." content_type: "troubleshooting" product: "tir-slurm-cluster" platform: "TIR" url: "/tir/tir-slurm-cluster/troubleshooting/jobs/" relationships: {"troubleshooting-for":["/tir/tir-slurm-cluster/how-to/see-jobs-and-logs/","/tir/tir-slurm-cluster/how-to/manage-partitions/","/tir/tir-slurm-cluster/how-to/add-prolog-and-epilog-scripts/","/tir/tir-slurm-cluster/how-to/monitor-a-cluster/"]} --- # Troubleshooting jobs on TIR Slurm Cluster Fix jobs that stay pending, fail, run out of memory, run slowly, or disappear. Start with the job's state and reason from the login node, or from the **Jobs** tab. See [See jobs and logs](/tir/tir-slurm-cluster/how-to/see-jobs-and-logs/). For other problems, see [Troubleshooting TIR Slurm Cluster](/tir/tir-slurm-cluster/troubleshooting/tir-slurm-cluster/), [Troubleshooting connecting to TIR Slurm Cluster](/tir/tir-slurm-cluster/troubleshooting/connecting/), and [Troubleshooting containers on TIR Slurm Cluster](/tir/tir-slurm-cluster/troubleshooting/containers/). ## My job output or log is missing after a failure A log written to a node-local path is lost when the node is recreated after a failure, and your only record of the failure goes with it. Point `--output` and `--error` at a PFS or SFS mount. To make jobs survive a restart or a node failure: - Checkpoint to shared storage regularly, and make the script resume from the latest checkpoint. - Always set `--time`, so a hung job ends at the limit and frees its GPUs. - Write `--output` and `--error` to PFS or SFS, and use `%x-%j` in log file names so runs stop overwriting each other. - Set a partition **Max time** as a cluster-wide safety net. A node failure, a preemption, or a restart ends the job, so checkpointing to shared storage is up to you. ## My job stays in Pending Pending is normal until Slurm finds resources that match the request. The reason is in the last column of `squeue`: ```bash squeue -j -o "%.18i %.9P %.30j %.8T %.10M %.30R" ``` | Reason | What it means | What to do | | ------------------------------------------------ | --------------------------------------------------------- | ----------------------------------------------------- | | `Resources` | Too little free capacity right now | Wait, ask for less, or scale up. | | `Priority` | A higher-priority job is ahead | Wait, or use a partition with a higher priority tier. | | PartitionNodeLimit, PartitionTimeLimit | The request exceeds the partition's nodes or maximum time | Fix `--nodes` or `--time`, or edit the partition. | | ReqNodeNotAvail | A requested node is down, drained, or unknown | Check the **Nodes** tab and `sinfo -N -o "%N %t %E"`. | | PartitionDown, PartitionInactive | The partition is down or inactive | Set its **State** back to `UP`. | | AssocMaxJobsLimit, or a reason starting `QOSMax` | An account or QoS limit is reached | Wait for your other jobs to finish. | | InvalidQOS, InvalidAccount | The job names a QoS or account that is missing | Fix the job script. TIR does not create QoS entries. | | `Dependency` | A job it depends on is yet to finish | Check that job's state. | | BeginTime | The job is scheduled to start later | Wait. | See [Scale a cluster](/tir/tir-slurm-cluster/how-to/scale-a-cluster/) and [Manage partitions](/tir/tir-slurm-cluster/how-to/manage-partitions/). ### My request asks for more than a node has `--gres` counts GPUs per node rather than per job. `--gres=gpu:16` on 8-GPU nodes stays pending indefinitely and silently, because Slurm keeps waiting for a node with 16 GPUs, which the cluster will never have. For 16 GPUs across two 8-GPU nodes, ask for: ```bash #SBATCH --nodes=2 #SBATCH --gpus-per-node=8 ``` To see what a node offers and what the job asked for: ```bash sinfo -N -o "%N %c %m %G" scontrol show node slinky-0 scontrol show job | grep -E 'NumNodes|NumCPUs|TRES|MinMemory' ``` ### Capacity looks free but nothing schedules A node in `drain` is closed to new work. `sinfo -N -o "%N %t %E"` shows the reason. A reason you recognise from your own Worker Prolog means the prolog is failing and draining nodes one job at a time. Bring a node back with: ```bash scontrol update nodename=slinky-3 state=resume ``` Jobs aimed at a partition that lost all its nodes after a scale-down stay pending until you edit the partition. If a partition is the constraint, or lost its nodes, see [Troubleshooting TIR Slurm Cluster](/tir/tir-slurm-cluster/troubleshooting/tir-slurm-cluster/#capacity-looks-free-but-nothing-schedules). See also [Add prolog and epilog scripts](/tir/tir-slurm-cluster/how-to/add-prolog-and-epilog-scripts/). ## My job fails immediately Read the exit code and your own log first: ```bash sacct -j --format=JobID,JobName,State,ExitCode,Elapsed,NodeList,Reason cat /pfs/logs/-.err ``` | What you see | Cause | What to do | | ------------------------------------------------------ | ------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | | ExitCode 1:0 with a traceback | Your code | Read the traceback in the `.err` file. | | ExitCode 127 | Command not found | The command is missing from the environment. In a container job, check the image has it. `#!/bin/bash` in a minimal image also gives 127, where `sh` works. | | ExitCode 2 from the shell | A script syntax error or a missing file | Check the paths and that the script is executable. | | No output file at all | The `--output` directory is missing or read-only | Run `mkdir -p /pfs/logs`, and confirm the path is on a mounted volume. | | `Permission denied` writing output | Output points at a read-only dataset mount | Use a PFS or SFS mount. | | slurmstepd: error: execve(): No such file or directory | The command is missing on the node | Fix the path, or run it in a container. | | `CANCELLED ... DUE TO TIME LIMIT` | The job hit `--time` | Raise `--time`, within the partition's **Max time**. | | A kill for host memory | Host RAM rather than GPU memory | Raise `--mem`, or reduce data-loading workers. | | NODE_FAIL | The node died during the job | Check the **Nodes** tab for XID errors. | To reproduce a failure interactively, which is faster than resubmitting a batch job: ```bash srun --partition=all --nodes=1 --gres=gpu:1 --time=00:30:00 --pty bash ``` You land on a compute node with the same resources. Run the command by hand and watch it fail. ## Invalid generic resource (gres) specification The full message is `srun: error: Unable to allocate resources: Invalid generic resource (gres) specification`. Confirm what the nodes offer: ```bash srun --gres=gpu:1 --time=00:05:00 nvidia-smi -L sinfo -N -o "%N %G" scontrol show node slinky-0 ``` If Slurm rejects a per-GPU request, take whole nodes instead: ```bash #SBATCH --nodes=1 #SBATCH --exclusive ``` The job then has every GPU on the node, and `nvidia-smi` inside it shows all of them. For multi-node work, `--nodes=N --exclusive` plus `--ntasks-per-node` behaves as you would expect. If your jobs need per-GPU allocation, raise a support request with the cluster name and image version. ## CUDA out of memory The GPU ran out of memory. Host RAM exhaustion looks different: Slurm kills the job and says so. Try these, in this order: 1. Reduce the batch size. 2. Use gradient accumulation, which keeps the effective batch size while lowering peak memory. 3. Use mixed precision, with `torch.autocast` or `bf16`, which roughly halves activation memory. 4. Use gradient checkpointing, which trades compute for memory on deep models. 5. Ask for a bigger GPU. An H200 has 141 GB against an H100's 80 GB. 6. Shard the model with FSDP, DeepSpeed ZeRO, or tensor parallelism. Watch usage on the **Nodes** tab, or from the login node: ```bash srun --jobid= --overlap nvidia-smi ``` A crashed job can leave a process holding GPU memory, which causes a false out-of-memory error at a batch size that worked yesterday. Check for it, and consider a Worker Epilog that clears GPU state between jobs: ```bash srun --nodelist=slinky-0 nvidia-smi ``` ## My job is slower than expected Open the **Monitoring** tab while the job runs and read **GPU Utilisation**. | Pattern | Diagnosis | What to do | | --------------------------------------- | --------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | | High and steady | The GPUs are the bottleneck, which is healthy | Optimise the model or add GPUs. | | Sawtoothing | The data pipeline is the bottleneck | Add data-loading workers, prefetch, and keep the dataset on a PFS. | | Near zero | The GPU sits idle | Check the model and tensors are on the device, and that the job is past start-up. | | One node lower than the rest | A straggler rank | Open the per-GPU view for that node. | | Multi-node much slower than single-node | Collectives fell back to TCP | Check the NCCL transport; see [Troubleshooting containers on TIR Slurm Cluster](/tir/tir-slurm-cluster/troubleshooting/containers/). | | Slow first step only | Image import or dataset cache warming | See [Cache container images](/tir/tir-slurm-cluster/how-to/cache-container-images/). | Check utilization on the first run of every new job script. A job that completes at 15% utilization costs the same as one at 95% and takes far longer. For multi-node runs, benchmark the fabric first: see [Run multi-node training](/tir/tir-slurm-cluster/how-to/run-multi-node-training/). ## My job disappeared from the queue `sacct` keeps a job long after `squeue` forgets it: ```bash sacct -j sacct -u $USER -S today --format=JobID,JobName,State,ExitCode,Elapsed,End ``` | Cause | What `sacct` shows | | ----------------------- | --------------------------------------------------------------------------------- | | It completed | The job as completed | | Someone cancelled it | `CANCELLED`, and by whom | | A restart or scale-down | **Restart All Workers**, **Restart Cluster**, and scale-down all end running jobs | | The node failed | NODE_FAIL. Check the **Nodes** tab. | Set MinJobAge in the extra `slurm.conf`, for example MinJobAge=300, to keep finished jobs in `squeue` and on the **Jobs** tab longer. When everyone logs in as `root`, `scancel -u root` cancels every job on the cluster, so one person clearing their own jobs clears everyone's. Individual logins from Login User Management prevent this, and they also make the **User** column on the **Jobs** tab show who owns each job. See [Manage login users](/tir/tir-slurm-cluster/how-to/manage-login-users/). If the problem continues, contact support with the cluster name, the job ID, and the exact error. See [Troubleshooting TIR Slurm Cluster](/tir/tir-slurm-cluster/troubleshooting/tir-slurm-cluster/#contact-support).