--- title: Jobs Will Not Run sidebar_label: Jobs --- import { Clock, XCircle, Cpu, Activity, AlertTriangle } from 'react-feather'; # Jobs Will Not Run Pending, failing, slow, or vanishing jobs — and what to do about each. }, { href: '#a-job-fails-immediately', label: 'Fails immediately', icon: }, { href: '#cuda-out-of-memory', label: 'CUDA OOM', icon: }, { href: '#a-job-is-slower-than-expected', label: 'Slower than expected', icon: }, { href: '#a-job-disappeared', label: 'Disappeared', icon: }, ]} /> --- ## A Job Stays Pending Pending is normal — Slurm has not found matching resources yet. The reason is in `squeue`: ```bash squeue -j -o "%.18i %.9P %.30j %.8T %.10M %.30R" ``` The last column names the cause. | Reason | Meaning | Fix | |--------|---------|-----| | `Resources` | Not enough free nodes or GPUs right now | Wait, ask for less, or [scale up](/docs/tir/SlurmCluster/manage/actions#scale-cluster) | | `Priority` | Higher-priority jobs are ahead | Wait, or use a partition with a higher priority tier | | `PartitionNodeLimit` | The job wants more nodes than the partition holds | Fix `--nodes`, or [add nodes to the partition](/docs/tir/SlurmCluster/slurm-configuration/partitions#edit-a-partition) | | `PartitionTimeLimit` | `--time` exceeds the partition's **Max time** | Lower `--time`, or raise the partition's limit | | `ReqNodeNotAvail` | A requested node is down, drained or unknown | Check the [Nodes tab](/docs/tir/SlurmCluster/manage/nodes) and `sinfo -N -o "%N %t %E"` | | `PartitionDown` / `PartitionInactive` | The partition is not accepting jobs | Set its **State** back to `UP` | | `AssocMaxJobsLimit`, `QOSMax…` | An account or QoS limit is reached | Wait for your other jobs to finish | | `InvalidQOS` / `InvalidAccount` | The job names something that does not exist | Fix the job script. TIR does not create QoS entries | | `Dependency` | A job it depends on has not finished | Check the dependency's state | | `BeginTime` | It is scheduled to start later | Expected | ### When the request cannot ever be satisfied The most common trap: asking for more than any single node has. ```bash # What does a node actually offer? sinfo -N -o "%N %c %m %G" # CPUs, memory, GRES scontrol show node slinky-0 # full detail ``` ```bash # What did the job ask for? scontrol show job | grep -E 'NumNodes|NumCPUs|TRES|MinMemory' ``` :::danger `--gres=gpu:16` on 8-GPU nodes pends forever GRES is **per node**, not per job. For 16 GPUs across two 8-GPU nodes: ```bash #SBATCH --nodes=2 #SBATCH --gpus-per-node=8 ``` Slurm does not report this as an error — it simply waits, indefinitely, for a node that will never exist. ::: :::warning A partition that lost its nodes stops existing If a scale-down removed every node a partition named, the partition is no longer written to `slurm.conf` and shows as `not live` in the [partition table](/docs/tir/SlurmCluster/manage/overview-tab#slurm-partition-info). Jobs targeting it never schedule. Repoint it at nodes that exist. ::: ### Capacity looks free but nothing schedules ```bash sinfo -N -o "%N %t %E" ``` Any node in `drain` shows its reason in `%E`. A reason you recognise from your own [Worker Prolog](/docs/tir/SlurmCluster/slurm-configuration/prolog-epilog#a-non-zero-exit-from-a-worker-prolog-drains-the-node) means the prolog is failing and draining nodes one job at a time. Resume with: ```bash scontrol update nodename=slinky-3 state=resume ``` --- ## A Job Fails Immediately Read the exit code and your own log first: ```bash sacct -j --format=JobID,JobName,State,ExitCode,Elapsed,NodeList,Reason cat /pfs/logs/-.err ``` | Symptom | Cause | Fix | |---------|-------|-----| | `ExitCode 1:0` with a Python traceback | Your code | Read the traceback in the `.err` file | | `ExitCode 127` | Command not found | The binary is not in the environment. In a container job, check the image has it; `#!/bin/bash` in a minimal image also gives 127 — use `sh` | | `ExitCode 2` from the shell | Script syntax, or a missing file | Check paths, and that the script is executable | | No output file at all | The `--output` directory does not exist, or is not writable | `mkdir -p /pfs/logs`, and confirm the path is on a mounted volume | | `Permission denied` writing output | Output is pointed at a **dataset** mount | Datasets are read-only. Use PFS or SFS | | `slurmstepd: error: execve(): No such file or directory` | The job's command does not exist on the node | Wrong path, or you expected something the node does not have — run it in a [container](/docs/tir/SlurmCluster/containers/) | | `JOB CANCELLED ... DUE TO TIME LIMIT` | It hit `--time` | Raise `--time`, within the partition's **Max time** | | `Job killed... out of memory` (host RAM) | Host memory, not GPU | Raise `--mem`, or reduce dataloader workers | | `NODE_FAIL` | The node died mid-job | Check the [Nodes tab](/docs/tir/SlurmCluster/manage/nodes) for XID errors | | `Invalid generic resource (gres) specification` | This cluster's Slurm does not sub-allocate GPUs | Request whole nodes with `--nodes=N --exclusive` instead of `--gres=gpu:N`. See [below](#gpu-requests-are-rejected) | :::danger `--output` outside a mounted volume loses the evidence If the job's log went to a node-local path, the pod recreation that follows a failure takes it with it — and you are left debugging with nothing. Always point `--output` and `--error` at your PFS or SFS mount. ::: :::tip Reproduce interactively Far faster than resubmitting a batch job: ```bash srun --partition=all --nodes=1 --gres=gpu:1 --time=00:30:00 --pty bash ``` You are on a compute node with the same resources. Run your command by hand and watch it fail. ::: --- ## GPU Requests Are Rejected ``` srun: error: Unable to allocate resources: Invalid generic resource (gres) specification ``` The cluster's Slurm is not set up to allocate GPUs individually, so `--gres=gpu:N` and `--gpus-per-node=N` are not accepted. Confirm what the nodes advertise: ```bash sinfo -N -o "%N %G" # the GRES column scontrol show node slinky-0 # look for Gres= and CfgTRES= ``` Work around it by taking whole nodes: ```bash #SBATCH --nodes=1 #SBATCH --exclusive ``` Your job then has every GPU on the node, and `nvidia-smi` inside it shows all of them. For multi-node work, `--nodes=N --exclusive` plus `--ntasks-per-node` behaves as you would expect. :::info This is a cluster-level setting, not something you can change It is not fixable from the Slurm Configuration dialog — the relevant directives are [platform-managed](/docs/tir/SlurmCluster/slurm-configuration/#what-you-can-and-cannot-change). If your workload needs per-GPU allocation, raise a support request naming the cluster and its image version. ::: --- ## CUDA Out of Memory The GPU ran out of memory. Host RAM OOM looks different — Slurm kills the job and says so. | Fix | How | |-----|-----| | Reduce batch size | The first thing to try | | Gradient accumulation | Keeps the effective batch size while lowering peak memory | | Mixed precision | `torch.autocast` / `bf16` roughly halves activation memory | | Gradient checkpointing | Trades compute for memory on deep models | | Ask for a bigger GPU | H200 has 141 GB against H100's 80 GB | | Shard the model | FSDP, DeepSpeed ZeRO, or tensor parallelism across GPUs | Check what is actually being used while the job runs, on the [Nodes tab](/docs/tir/SlurmCluster/manage/nodes#per-gpu-drilldown) or from a shell: ```bash srun --jobid= --overlap nvidia-smi ``` :::warning A previous job's leftovers can cause a false OOM A crashed job can leave a process holding GPU memory. If a job OOMs at a batch size that worked yesterday, check for orphans: ```bash srun --nodelist=slinky-0 nvidia-smi ``` A [Worker Epilog](/docs/tir/SlurmCluster/slurm-configuration/prolog-epilog#worked-examples) that clears GPU state between jobs prevents this class of problem entirely. ::: --- ## A Job Is Slower Than Expected Open [Monitoring](/docs/tir/SlurmCluster/manage/monitoring) while it runs and read **GPU Utilisation**. | Pattern | Diagnosis | Fix | |---------|-----------|-----| | Utilisation high and steady | The GPUs are the bottleneck | This is the healthy case. Optimise the model or add GPUs | | Utilisation sawtoothing | Data pipeline bottleneck — GPUs waiting on I/O | More dataloader workers, prefetching, and make sure the dataset is on PFS not a slow path | | Utilisation near zero | The work is not on the GPU | Check the model and tensors are `.to(device)`; check the job is not stuck in initialisation | | One node lower than the others | A straggler rank | [Per-GPU drilldown](/docs/tir/SlurmCluster/manage/nodes#per-gpu-drilldown) on that node | | Multi-node much slower than single-node | Collectives falling back to TCP | [NCCL environment](/docs/tir/SlurmCluster/containers/multi-node-training#nccl-environment) | | Slow first step only | Container image import, or dataset cache warming | [Cache the image](/docs/tir/SlurmCluster/containers/image-cache) | :::tip Check utilisation on every new job script A job that completes at 15% utilisation costs the same as one at 95% and takes six times as long. Two minutes with the Monitoring tab on a new script is the highest-return debugging you can do on a GPU cluster. ::: ### Prove the fabric before blaming your code For multi-node runs, benchmark the collectives: [Validate the fabric first](/docs/tir/SlurmCluster/containers/multi-node-training#validate-the-fabric-first). --- ## A Job Disappeared | Cause | How to confirm | |-------|----------------| | It completed and left the queue | `sacct -j ` — it lives in accounting long after `squeue` forgets | | Someone cancelled it | `sacct -j ` shows `CANCELLED` and by whom | | A restart ended it | [Restart All Workers, Restart Cluster](/docs/tir/SlurmCluster/manage/actions#restart-actions) and scale-down all end running jobs | | The node failed | `sacct` shows `NODE_FAIL`. Check the Nodes tab | ```bash sacct -u $USER -S today --format=JobID,JobName,State,ExitCode,Elapsed,End ``` :::tip Keep finished jobs in the Jobs tab longer By default they leave `squeue` quickly and vanish from the tab. Raise `MinJobAge` in [extra `slurm.conf`](/docs/tir/SlurmCluster/slurm-configuration/slurm-conf#useful-settings) — for example `MinJobAge=300`. ::: :::danger On a shared cluster, `scancel -u root` cancels everything If everyone logs in as `root`, one person clearing "their" jobs clears the whole cluster's. This is the strongest practical argument for [Login User Management](/docs/tir/SlurmCluster/connect/login-user-management). ::: --- ## Make Jobs Survivable | Practice | Why | |----------|-----| | Checkpoint to shared storage regularly | A node failure, preemption or restart ends the job. Progress is only as good as the last checkpoint | | Make your script resume from the latest checkpoint | Then a restart costs minutes, not days | | Always set `--time` | An unbounded hung job holds GPUs indefinitely | | Write `--output` and `--error` to PFS or SFS | Otherwise the evidence dies with the pod | | Use `%x-%j` in log filenames | Runs stop overwriting each other | | Set partition **Max time** | A cluster-wide safety net against hung jobs | --- ## Related Resources - [Jobs tab](/docs/tir/SlurmCluster/manage/jobs) - [Container job failures](/docs/tir/SlurmCluster/troubleshoot/containers) - [Nodes and GPU health](/docs/tir/SlurmCluster/manage/nodes) - [Manage partitions](/docs/tir/SlurmCluster/slurm-configuration/partitions) - [Monitoring](/docs/tir/SlurmCluster/manage/monitoring)