--- title: Troubleshooting sidebar_label: Overview --- import { HelpCircle, Play, Box, Wifi, Server, Sliders } from 'react-feather'; # Troubleshooting Start here, then follow the link to the focused page. }, { href: '#the-first-five-checks', label: 'First five checks', icon: }, { href: '#faqs', label: 'FAQs', icon: }, ]} /> | Page | Covers | |------|--------| | [Jobs will not run](/docs/tir/SlurmCluster/troubleshoot/jobs) | Pending, failing, cancelled, or slow jobs | | [Container job failures](/docs/tir/SlurmCluster/troubleshoot/containers) | Enroot and Pyxis: image imports, mounts, GPUs, NCCL | | [Cannot connect](/docs/tir/SlurmCluster/troubleshoot/connectivity) | SSH failures for root and for named users | --- ## By Symptom | Symptom | Go to | |---------|-------| | The cluster is stuck in **Creating** | [First five checks](#the-first-five-checks) | | The cluster is **Failed** | [Cluster problems](#cluster-problems) | | `ssh` times out or is refused | [Cannot connect](/docs/tir/SlurmCluster/troubleshoot/connectivity) | | `Permission denied (publickey)` | [Cannot connect](/docs/tir/SlurmCluster/troubleshoot/connectivity#permission-denied-publickey) | | A job sits in **Pending** | [Jobs](/docs/tir/SlurmCluster/troubleshoot/jobs#a-job-stays-pending) | | A job fails instantly | [Jobs](/docs/tir/SlurmCluster/troubleshoot/jobs#a-job-fails-immediately) | | `srun: unrecognized option '--container-image'` | [Containers](/docs/tir/SlurmCluster/troubleshoot/containers#--container-image-is-not-recognised) | | A container job cannot see my data | [Containers](/docs/tir/SlurmCluster/troubleshoot/containers#the-container-cannot-see-my-files) | | `nvidia-smi` shows no GPUs in a container | [Containers](/docs/tir/SlurmCluster/troubleshoot/containers#no-gpus-inside-the-container) | | A multi-node job hangs at startup | [Containers](/docs/tir/SlurmCluster/troubleshoot/containers#a-multi-node-job-hangs) | | CUDA out of memory | [Jobs](/docs/tir/SlurmCluster/troubleshoot/jobs#cuda-out-of-memory) | | A node shows **XID Error** | [Nodes and GPU health](/docs/tir/SlurmCluster/manage/nodes#xid-errors) | | A node is stuck **UNKNOWN** | [Cluster problems](#cluster-problems) | | A node keeps getting drained | [Prolog scripts](/docs/tir/SlurmCluster/slurm-configuration/prolog-epilog#a-non-zero-exit-from-a-worker-prolog-drains-the-node) | | GPU utilisation is near zero while a job runs | [Monitoring](/docs/tir/SlurmCluster/manage/monitoring#reading-the-charts) | | A partition disappeared after scaling down | [Partitions](/docs/tir/SlurmCluster/slurm-configuration/partitions#a-partition-that-stopped-working-after-a-scale-down) | | A `slurm.conf` setting was rejected | [Extra slurm.conf](/docs/tir/SlurmCluster/slurm-configuration/slurm-conf#validation-messages) | | An action is missing or greyed out | [Action availability](/docs/tir/SlurmCluster/manage/action-availability) | | My data vanished after a restart | [What persists](/docs/tir/SlurmCluster/getting-started/slurm-cluster-concepts#storage-what-persists-and-what-does-not) | | A volume will not unmount | [Storage](/docs/tir/SlurmCluster/manage/storage#when-unmount-is-blocked) | | A named user cannot log in | [Login users](/docs/tir/SlurmCluster/connect/login-user-management#how-a-member-logs-in) | --- ## The First Five Checks Before diving deeper, these five answer most problems. ### 1. Is the cluster Running? The status pill on the [Details tab](/docs/tir/SlurmCluster/manage/). Most tabs and most actions need **Running**. ### 2. Are the nodes healthy? [Cluster Overview](/docs/tir/SlurmCluster/manage/overview-tab) → **Node Health**. Grey (UNKNOWN) or red (DOWN) dots mean those nodes are not usable, which makes jobs pend for no visible reason. ```bash sinfo -N -o "%N %t %E" ``` The `%E` column shows *why* a node was drained. ### 3. Is your storage mounted where you think? ```bash df -h | grep -E '/pfs|/shared' echo test > /pfs/write-check && rm /pfs/write-check && echo "writable" ``` :::danger The commonest root cause of all Work written outside a mounted volume — `/root`, `/tmp`, `/home` without per-user logins — is erased whenever the pod is recreated. If output "disappeared", check where it was written before assuming anything is broken. ::: ### 4. What is Slurm actually saying? ```bash squeue -o "%.18i %.9P %.30j %.8T %.10M %.20R" # the reason column is the answer scontrol show job sacct -j --format=JobID,State,ExitCode,Elapsed,NodeList ``` ### 5. What do the logs say? The [Logs tab](/docs/tir/SlurmCluster/manage/logs) for the worker your job landed on, plus your own `--output` file on shared storage. --- ## Cluster Problems ### Stuck in Creating Provisioning normally takes a few minutes. 1. Wait a little longer — the list refreshes itself while anything is Creating. 2. Try [**Actions → Reconfigure Cluster**](/docs/tir/SlurmCluster/manage/actions#reconfigure-cluster). It is available on a Creating cluster and re-applies the intended state. 3. If it does not move, note the cluster name and contact support. ### Status is Failed A Failed cluster cannot be repaired into a Running one. 1. **Actions → Clone Cluster** — you get the same configuration in a fresh create form. 2. Adjust anything that might have caused it — most often a plan with no inventory. Try a different GPU type, fewer nodes, or a [Private Cluster](/docs/myaccount/private_cluster/). 3. Create the clone, confirm it reaches Running, then **Delete** the failed record. A Failed cluster is not billed for compute, so there is no urgency. ### A node is stuck UNKNOWN **UNKNOWN** means the controller has no report for that node. 1. Give it a few minutes — normal right after a restart, scale-up or image update. 2. Check that worker's log on the [Logs tab](/docs/tir/SlurmCluster/manage/logs). 3. [Reboot that node](/docs/tir/SlurmCluster/manage/nodes#reboot-a-node) from the Nodes tab. 4. If several nodes are affected, [**Restart All Workers**](/docs/tir/SlurmCluster/manage/actions#restart-actions) — but note it ends every running job. ### Everything looks fine but nothing schedules Check whether a partition is the constraint rather than capacity: ```bash sinfo # is the partition UP? scontrol show partition # its nodes, MaxTime, state ``` A partition in `DOWN`, `DRAIN` or `INACTIVE` accepts nothing. A partition whose nodes were removed by a scale-down stops existing altogether — see [Partitions](/docs/tir/SlurmCluster/slurm-configuration/partitions#a-partition-that-stopped-working-after-a-scale-down). --- ## FAQs **Does a Slurm Cluster cost more when it is busy?** No. You pay for the nodes, per hour, whether they are idle or saturated. Jobs are never billed separately. See [Billing](/docs/tir/SlurmCluster/billing). **Can I pause a cluster to stop paying?** No. The only way to stop compute charges is to [terminate](/docs/tir/SlurmCluster/manage/actions#terminate-cluster) it. Clone it first if you want the configuration back later — a Terminated cluster can still be cloned. **Do I need to recreate the cluster to use a different PyTorch or CUDA version?** No. Pass `--container-image` to `srun`/`sbatch` and the job runs in whatever image you name. The cluster's **Image** is its own Slurm software stack, not your job's environment. See [Containers](/docs/tir/SlurmCluster/containers/). **Can two teams use different framework versions on one cluster?** Yes — that is exactly what container jobs are for. Different jobs can use completely different images at the same time. **Can I add a partition while the cluster is being created?** No. A partition names specific nodes, which do not exist yet. Add it from the **Cluster Overview** tab once the cluster is Running. See [Partitions](/docs/tir/SlurmCluster/slurm-configuration/partitions). **Can I edit `slurm.conf` over SSH?** You can, and it will be overwritten. The file is regenerated whenever the cluster changes — a scale, an image update, a volume mount. Use [Slurm Configuration](/docs/tir/SlurmCluster/slurm-configuration/slurm-conf) so the change persists. **Why was my `slurm.conf` line rejected?** Either the key is platform-managed, or the syntax is invalid — most often a space in the value. See [validation messages](/docs/tir/SlurmCluster/slurm-configuration/slurm-conf#validation-messages). **Does scaling down interrupt jobs?** Yes — on the removed nodes, and the remaining workers restart too. Drain first. See [Scale Cluster](/docs/tir/SlurmCluster/manage/actions#scale-cluster). **Will an image update interrupt my jobs?** Yes, but gracefully: nodes are drained one at a time, roughly 3–8 minutes each. Long-running jobs are still affected, so schedule it. **Where should checkpoints go?** A PFS, SFS or Weka mount. Never a node-local path. See [Storage](/docs/tir/SlurmCluster/manage/storage#what-goes-where). **How do I see whose job is whose?** Enable [Login User Management](/docs/tir/SlurmCluster/connect/login-user-management). Without it everyone is `root` and the **User** column is meaningless. **Can I cancel a job from the console?** No. Use `scancel ` over SSH. The [Jobs tab](/docs/tir/SlurmCluster/manage/jobs) is read-only. **Does terminating a cluster delete my data?** No. Volumes are unmounted, not deleted — and they keep billing. Delete them separately under **Storage** when you are done. **Can I get a GPU type that shows "Inventory not available"?** Not at that moment. Retry later, or reserve capacity with a [Private Cluster](/docs/myaccount/private_cluster/). --- ## When to Contact Support Raise a ticket from **Support** in the TIR sidebar, or email [cloud-platform@e2enetworks.com](mailto:cloud-platform@e2enetworks.com). Include the cluster name, the project, and: | Problem | Also include | |---------|--------------| | Stuck in Creating or Terminating | How long, and whether Reconfigure was tried | | Repeated XID errors | The XID code, the `e2e_node_` name, and whether a reboot helped | | A node that will not register | That worker's log from the Logs tab | | Fabric performance far below expectation | Your NCCL benchmark output and the `NCCL_*` variables you set | | A configuration change that fails to apply | The exact configuration and the error message | :::tip Download logs before you restart anything Restarting recreates the pods and the previous logs are gone. Grab them from the [Logs tab](/docs/tir/SlurmCluster/manage/logs#download-logs) first, or you will be asking support to diagnose something with no evidence. ::: --- ## Related Resources - [Manage a Slurm Cluster](/docs/tir/SlurmCluster/manage/) - [Action availability and blockers](/docs/tir/SlurmCluster/manage/action-availability) - [How a Slurm Cluster works](/docs/tir/SlurmCluster/getting-started/slurm-cluster-concepts) - [UI Guide](/docs/tir/SlurmCluster/ui-guide)