Troubleshooting jobs on TIR Slurm Cluster

Fix jobs that stay pending, fail, run out of memory, run slowly, or disappear. Start with the job’s state and reason from the login node, or from the Jobs tab. See See jobs and logs. For other problems, see Troubleshooting TIR Slurm Cluster, Troubleshooting connecting to TIR Slurm Cluster, and Troubleshooting containers on TIR Slurm Cluster.

My job output or log is missing after a failure

A log written to a node-local path is lost when the node is recreated after a failure, and your only record of the failure goes with it. Point --output and --error at a PFS or SFS mount. To make jobs survive a restart or a node failure:

• Checkpoint to shared storage regularly, and make the script resume from the latest checkpoint.
• Always set --time, so a hung job ends at the limit and frees its GPUs.
• Write --output and --error to PFS or SFS, and use %x-%j in log file names so runs stop overwriting each other.
• Set a partition Max time as a cluster-wide safety net.

A node failure, a preemption, or a restart ends the job, so checkpointing to shared storage is up to you.

My job stays in Pending

Pending is normal until Slurm finds resources that match the request. The reason is in the last column of squeue:

squeue -j <jobid> -o "%.18i %.9P %.30j %.8T %.10M %.30R"

Reason	What it means	What to do
Resources	Too little free capacity right now	Wait, ask for less, or scale up.
Priority	A higher-priority job is ahead	Wait, or use a partition with a higher priority tier.
PartitionNodeLimit, PartitionTimeLimit	The request exceeds the partition’s nodes or maximum time	Fix --nodes or --time, or edit the partition.
ReqNodeNotAvail	A requested node is down, drained, or unknown	Check the Nodes tab and sinfo -N -o "%N %t %E".
PartitionDown, PartitionInactive	The partition is down or inactive	Set its State back to UP.
AssocMaxJobsLimit, or a reason starting QOSMax	An account or QoS limit is reached	Wait for your other jobs to finish.
InvalidQOS, InvalidAccount	The job names a QoS or account that is missing	Fix the job script. TIR does not create QoS entries.
Dependency	A job it depends on is yet to finish	Check that job’s state.
BeginTime	The job is scheduled to start later	Wait.

See Scale a cluster and Manage partitions.

My request asks for more than a node has

--gres counts GPUs per node rather than per job. --gres=gpu:16 on 8-GPU nodes stays pending indefinitely and silently, because Slurm keeps waiting for a node with 16 GPUs, which the cluster will never have. For 16 GPUs across two 8-GPU nodes, ask for:

#SBATCH --nodes=2
#SBATCH --gpus-per-node=8

To see what a node offers and what the job asked for:

sinfo -N -o "%N %c %m %G"
scontrol show node slinky-0
scontrol show job <jobid> | grep -E 'NumNodes|NumCPUs|TRES|MinMemory'

Capacity looks free but nothing schedules

A node in drain is closed to new work. sinfo -N -o "%N %t %E" shows the reason. A reason you recognise from your own Worker Prolog means the prolog is failing and draining nodes one job at a time. Bring a node back with:

scontrol update nodename=slinky-3 state=resume

Jobs aimed at a partition that lost all its nodes after a scale-down stay pending until you edit the partition. If a partition is the constraint, or lost its nodes, see Troubleshooting TIR Slurm Cluster. See also Add prolog and epilog scripts.

My job fails immediately

Read the exit code and your own log first:

sacct -j <jobid> --format=JobID,JobName,State,ExitCode,Elapsed,NodeList,Reason
cat /pfs/logs/<jobname>-<jobid>.err

What you see	Cause	What to do
ExitCode 1:0 with a traceback	Your code	Read the traceback in the .err file.
ExitCode 127	Command not found	The command is missing from the environment. In a container job, check the image has it. #!/bin/bash in a minimal image also gives 127, where sh works.
ExitCode 2 from the shell	A script syntax error or a missing file	Check the paths and that the script is executable.
No output file at all	The --output directory is missing or read-only	Run mkdir -p /pfs/logs, and confirm the path is on a mounted volume.
Permission denied writing output	Output points at a read-only dataset mount	Use a PFS or SFS mount.
slurmstepd: error: execve(): No such file or directory	The command is missing on the node	Fix the path, or run it in a container.
CANCELLED ... DUE TO TIME LIMIT	The job hit --time	Raise --time, within the partition’s Max time.
A kill for host memory	Host RAM rather than GPU memory	Raise --mem, or reduce data-loading workers.
NODE_FAIL	The node died during the job	Check the Nodes tab for XID errors.

To reproduce a failure interactively, which is faster than resubmitting a batch job:

srun --partition=all --nodes=1 --gres=gpu:1 --time=00:30:00 --pty bash

You land on a compute node with the same resources. Run the command by hand and watch it fail.

Invalid generic resource (gres) specification

The full message is srun: error: Unable to allocate resources: Invalid generic resource (gres) specification. Confirm what the nodes offer:

srun --gres=gpu:1 --time=00:05:00 nvidia-smi -L
sinfo -N -o "%N %G"
scontrol show node slinky-0

If Slurm rejects a per-GPU request, take whole nodes instead:

#SBATCH --nodes=1
#SBATCH --exclusive

The job then has every GPU on the node, and nvidia-smi inside it shows all of them. For multi-node work, --nodes=N --exclusive plus --ntasks-per-node behaves as you would expect. If your jobs need per-GPU allocation, raise a support request with the cluster name and image version.

CUDA out of memory

The GPU ran out of memory. Host RAM exhaustion looks different: Slurm kills the job and says so. Try these, in this order:

1. Reduce the batch size.
2. Use gradient accumulation, which keeps the effective batch size while lowering peak memory.
3. Use mixed precision, with torch.autocast or bf16, which roughly halves activation memory.
4. Use gradient checkpointing, which trades compute for memory on deep models.
5. Ask for a bigger GPU. An H200 has 141 GB against an H100’s 80 GB.
6. Shard the model with FSDP, DeepSpeed ZeRO, or tensor parallelism.

Watch usage on the Nodes tab, or from the login node:

srun --jobid=<jobid> --overlap nvidia-smi

A crashed job can leave a process holding GPU memory, which causes a false out-of-memory error at a batch size that worked yesterday. Check for it, and consider a Worker Epilog that clears GPU state between jobs:

srun --nodelist=slinky-0 nvidia-smi

My job is slower than expected

Open the Monitoring tab while the job runs and read GPU Utilisation.

Pattern	Diagnosis	What to do
High and steady	The GPUs are the bottleneck, which is healthy	Optimise the model or add GPUs.
Sawtoothing	The data pipeline is the bottleneck	Add data-loading workers, prefetch, and keep the dataset on a PFS.
Near zero	The GPU sits idle	Check the model and tensors are on the device, and that the job is past start-up.
One node lower than the rest	A straggler rank	Open the per-GPU view for that node.
Multi-node much slower than single-node	Collectives fell back to TCP	Check the NCCL transport; see Troubleshooting containers on TIR Slurm Cluster.
Slow first step only	Image import or dataset cache warming	See Cache container images.

Check utilization on the first run of every new job script. A job that completes at 15% utilization costs the same as one at 95% and takes far longer. For multi-node runs, benchmark the fabric first: see Run multi-node training.

My job disappeared from the queue

sacct keeps a job long after squeue forgets it:

sacct -j <jobid>
sacct -u $USER -S today --format=JobID,JobName,State,ExitCode,Elapsed,End

Cause	What sacct shows
It completed	The job as completed
Someone cancelled it	CANCELLED, and by whom
A restart or scale-down	Restart All Workers, Restart Cluster, and scale-down all end running jobs
The node failed	NODE_FAIL. Check the Nodes tab.

Set MinJobAge in the extra slurm.conf, for example MinJobAge=300, to keep finished jobs in squeue and on the Jobs tab longer.

When everyone logs in as root, scancel -u root cancels every job on the cluster, so one person clearing their own jobs clears everyone’s. Individual logins from Login User Management prevent this, and they also make the User column on the Jobs tab show who owns each job. See Manage login users.

If the problem continues, contact support with the cluster name, the job ID, and the exact error. See Troubleshooting TIR Slurm Cluster.