Jobs Will Not Run

Pending, failing, slow, or vanishing jobs — and what to do about each.

Stays pendingFails immediatelyCUDA OOMSlower than expectedDisappeared

---

A Job Stays Pending

Pending is normal — Slurm has not found matching resources yet. The reason is in squeue:

squeue -j <jobid> -o "%.18i %.9P %.30j %.8T %.10M %.30R"

The last column names the cause.

Reason	Meaning	Fix
Resources	Not enough free nodes or GPUs right now	Wait, ask for less, or scale up
Priority	Higher-priority jobs are ahead	Wait, or use a partition with a higher priority tier
PartitionNodeLimit	The job wants more nodes than the partition holds	Fix --nodes, or add nodes to the partition
PartitionTimeLimit	--time exceeds the partition's Max time	Lower --time, or raise the partition's limit
ReqNodeNotAvail	A requested node is down, drained or unknown	Check the Nodes tab and sinfo -N -o "%N %t %E"
PartitionDown / PartitionInactive	The partition is not accepting jobs	Set its State back to UP
AssocMaxJobsLimit, QOSMax…	An account or QoS limit is reached	Wait for your other jobs to finish
InvalidQOS / InvalidAccount	The job names something that does not exist	Fix the job script. TIR does not create QoS entries
Dependency	A job it depends on has not finished	Check the dependency's state
BeginTime	It is scheduled to start later	Expected

When the request cannot ever be satisfied

The most common trap: asking for more than any single node has.

# What does a node actually offer?
sinfo -N -o "%N %c %m %G"          # CPUs, memory, GRES
scontrol show node slinky-0        # full detail

# What did the job ask for?
scontrol show job <jobid> | grep -E 'NumNodes|NumCPUs|TRES|MinMemory'

--gres=gpu:16 on 8-GPU nodes pends forever

GRES is per node, not per job. For 16 GPUs across two 8-GPU nodes:

#SBATCH --nodes=2
#SBATCH --gpus-per-node=8

Slurm does not report this as an error — it simply waits, indefinitely, for a node that will never exist.

A partition that lost its nodes stops existing

If a scale-down removed every node a partition named, the partition is no longer written to slurm.conf and shows as not live in the partition table. Jobs targeting it never schedule. Repoint it at nodes that exist.

Capacity looks free but nothing schedules

sinfo -N -o "%N %t %E"

Any node in drain shows its reason in %E. A reason you recognise from your own Worker Prolog means the prolog is failing and draining nodes one job at a time. Resume with:

scontrol update nodename=slinky-3 state=resume

---

A Job Fails Immediately

Read the exit code and your own log first:

sacct -j <jobid> --format=JobID,JobName,State,ExitCode,Elapsed,NodeList,Reason
cat /pfs/logs/<jobname>-<jobid>.err

Symptom	Cause	Fix
ExitCode 1:0 with a Python traceback	Your code	Read the traceback in the .err file
ExitCode 127	Command not found	The binary is not in the environment. In a container job, check the image has it; #!/bin/bash in a minimal image also gives 127 — use sh
ExitCode 2 from the shell	Script syntax, or a missing file	Check paths, and that the script is executable
No output file at all	The --output directory does not exist, or is not writable	mkdir -p /pfs/logs, and confirm the path is on a mounted volume
Permission denied writing output	Output is pointed at a dataset mount	Datasets are read-only. Use PFS or SFS
slurmstepd: error: execve(): No such file or directory	The job's command does not exist on the node	Wrong path, or you expected something the node does not have — run it in a container
JOB <id> CANCELLED ... DUE TO TIME LIMIT	It hit --time	Raise --time, within the partition's Max time
Job killed... out of memory (host RAM)	Host memory, not GPU	Raise --mem, or reduce dataloader workers
NODE_FAIL	The node died mid-job	Check the Nodes tab for XID errors
Invalid generic resource (gres) specification	This cluster's Slurm does not sub-allocate GPUs	Request whole nodes with --nodes=N --exclusive instead of --gres=gpu:N. See below

--output outside a mounted volume loses the evidence

If the job's log went to a node-local path, the pod recreation that follows a failure takes it with it — and you are left debugging with nothing. Always point --output and --error at your PFS or SFS mount.

Reproduce interactively

Far faster than resubmitting a batch job:

srun --partition=all --nodes=1 --gres=gpu:1 --time=00:30:00 --pty bash

You are on a compute node with the same resources. Run your command by hand and watch it fail.

---

GPU Requests Are Rejected

srun: error: Unable to allocate resources: Invalid generic resource (gres) specification

The cluster's Slurm is not set up to allocate GPUs individually, so --gres=gpu:N and --gpus-per-node=N are not accepted. Confirm what the nodes advertise:

sinfo -N -o "%N %G"          # the GRES column
scontrol show node slinky-0  # look for Gres= and CfgTRES=

Work around it by taking whole nodes:

#SBATCH --nodes=1
#SBATCH --exclusive

Your job then has every GPU on the node, and nvidia-smi inside it shows all of them. For multi-node work, --nodes=N --exclusive plus --ntasks-per-node behaves as you would expect.

This is a cluster-level setting, not something you can change

It is not fixable from the Slurm Configuration dialog — the relevant directives are platform-managed. If your workload needs per-GPU allocation, raise a support request naming the cluster and its image version.

---

CUDA Out of Memory

The GPU ran out of memory. Host RAM OOM looks different — Slurm kills the job and says so.

Fix	How
Reduce batch size	The first thing to try
Gradient accumulation	Keeps the effective batch size while lowering peak memory
Mixed precision	torch.autocast / bf16 roughly halves activation memory
Gradient checkpointing	Trades compute for memory on deep models
Ask for a bigger GPU	H200 has 141 GB against H100's 80 GB
Shard the model	FSDP, DeepSpeed ZeRO, or tensor parallelism across GPUs

Check what is actually being used while the job runs, on the Nodes tab or from a shell:

srun --jobid=<jobid> --overlap nvidia-smi

A previous job's leftovers can cause a false OOM

A crashed job can leave a process holding GPU memory. If a job OOMs at a batch size that worked yesterday, check for orphans:

srun --nodelist=slinky-0 nvidia-smi

A Worker Epilog that clears GPU state between jobs prevents this class of problem entirely.

---

A Job Is Slower Than Expected

Open Monitoring while it runs and read GPU Utilisation.

Pattern	Diagnosis	Fix
Utilisation high and steady	The GPUs are the bottleneck	This is the healthy case. Optimise the model or add GPUs
Utilisation sawtoothing	Data pipeline bottleneck — GPUs waiting on I/O	More dataloader workers, prefetching, and make sure the dataset is on PFS not a slow path
Utilisation near zero	The work is not on the GPU	Check the model and tensors are .to(device); check the job is not stuck in initialisation
One node lower than the others	A straggler rank	Per-GPU drilldown on that node
Multi-node much slower than single-node	Collectives falling back to TCP	NCCL environment
Slow first step only	Container image import, or dataset cache warming	Cache the image

Check utilisation on every new job script

A job that completes at 15% utilisation costs the same as one at 95% and takes six times as long. Two minutes with the Monitoring tab on a new script is the highest-return debugging you can do on a GPU cluster.

Prove the fabric before blaming your code

For multi-node runs, benchmark the collectives: Validate the fabric first.

---

A Job Disappeared

Cause	How to confirm
It completed and left the queue	sacct -j <jobid> — it lives in accounting long after squeue forgets
Someone cancelled it	sacct -j <jobid> shows CANCELLED and by whom
A restart ended it	Restart All Workers, Restart Cluster and scale-down all end running jobs
The node failed	sacct shows NODE_FAIL. Check the Nodes tab

sacct -u $USER -S today --format=JobID,JobName,State,ExitCode,Elapsed,End

Keep finished jobs in the Jobs tab longer

By default they leave squeue quickly and vanish from the tab. Raise MinJobAge in extra slurm.conf — for example MinJobAge=300.

On a shared cluster, scancel -u root cancels everything

If everyone logs in as root, one person clearing "their" jobs clears the whole cluster's. This is the strongest practical argument for Login User Management.

---

Make Jobs Survivable

Practice	Why
Checkpoint to shared storage regularly	A node failure, preemption or restart ends the job. Progress is only as good as the last checkpoint
Make your script resume from the latest checkpoint	Then a restart costs minutes, not days
Always set --time	An unbounded hung job holds GPUs indefinitely
Write --output and --error to PFS or SFS	Otherwise the evidence dies with the pod
Use %x-%j in log filenames	Runs stop overwriting each other
Set partition Max time	A cluster-wide safety net against hung jobs

---

Related Resources

• Jobs tab
• Container job failures
• Nodes and GPU health
• Manage partitions
• Monitoring