Troubleshooting
Start here, then follow the link to the focused page.
| Page | Covers |
|---|---|
| Jobs will not run | Pending, failing, cancelled, or slow jobs |
| Container job failures | Enroot and Pyxis: image imports, mounts, GPUs, NCCL |
| Cannot connect | SSH failures for root and for named users |
By Symptom
| Symptom | Go to |
|---|---|
| The cluster is stuck in Creating | First five checks |
| The cluster is Failed | Cluster problems |
ssh times out or is refused | Cannot connect |
Permission denied (publickey) | Cannot connect |
| A job sits in Pending | Jobs |
| A job fails instantly | Jobs |
srun: unrecognized option '--container-image' | Containers |
| A container job cannot see my data | Containers |
nvidia-smi shows no GPUs in a container | Containers |
| A multi-node job hangs at startup | Containers |
| CUDA out of memory | Jobs |
| A node shows XID Error | Nodes and GPU health |
| A node is stuck UNKNOWN | Cluster problems |
| A node keeps getting drained | Prolog scripts |
| GPU utilisation is near zero while a job runs | Monitoring |
| A partition disappeared after scaling down | Partitions |
A slurm.conf setting was rejected | Extra slurm.conf |
| An action is missing or greyed out | Action availability |
| My data vanished after a restart | What persists |
| A volume will not unmount | Storage |
| A named user cannot log in | Login users |
The First Five Checks
Before diving deeper, these five answer most problems.
1. Is the cluster Running?
The status pill on the Details tab. Most tabs and most actions need Running.
2. Are the nodes healthy?
Cluster Overview → Node Health. Grey (UNKNOWN) or red (DOWN) dots mean those nodes are not usable, which makes jobs pend for no visible reason.
sinfo -N -o "%N %t %E"
The %E column shows why a node was drained.
3. Is your storage mounted where you think?
df -h | grep -E '/pfs|/shared'
echo test > /pfs/write-check && rm /pfs/write-check && echo "writable"
Work written outside a mounted volume — /root, /tmp, /home without per-user logins — is erased
whenever the pod is recreated. If output "disappeared", check where it was written before assuming
anything is broken.
4. What is Slurm actually saying?
squeue -o "%.18i %.9P %.30j %.8T %.10M %.20R" # the reason column is the answer
scontrol show job <jobid>
sacct -j <jobid> --format=JobID,State,ExitCode,Elapsed,NodeList
5. What do the logs say?
The Logs tab for the worker your job landed on, plus your own
--output file on shared storage.
Cluster Problems
Stuck in Creating
Provisioning normally takes a few minutes.
- Wait a little longer — the list refreshes itself while anything is Creating.
- Try Actions → Reconfigure Cluster. It is available on a Creating cluster and re-applies the intended state.
- If it does not move, note the cluster name and contact support.
Status is Failed
A Failed cluster cannot be repaired into a Running one.
- Actions → Clone Cluster — you get the same configuration in a fresh create form.
- Adjust anything that might have caused it — most often a plan with no inventory. Try a different GPU type, fewer nodes, or a Private Cluster.
- Create the clone, confirm it reaches Running, then Delete the failed record.
A Failed cluster is not billed for compute, so there is no urgency.
A node is stuck UNKNOWN
UNKNOWN means the controller has no report for that node.
- Give it a few minutes — normal right after a restart, scale-up or image update.
- Check that worker's log on the Logs tab.
- Reboot that node from the Nodes tab.
- If several nodes are affected, Restart All Workers — but note it ends every running job.
Everything looks fine but nothing schedules
Check whether a partition is the constraint rather than capacity:
sinfo # is the partition UP?
scontrol show partition <name> # its nodes, MaxTime, state
A partition in DOWN, DRAIN or INACTIVE accepts nothing. A partition whose nodes were removed by
a scale-down stops existing altogether — see
Partitions.
FAQs
Does a Slurm Cluster cost more when it is busy? No. You pay for the nodes, per hour, whether they are idle or saturated. Jobs are never billed separately. See Billing.
Can I pause a cluster to stop paying? No. The only way to stop compute charges is to terminate it. Clone it first if you want the configuration back later — a Terminated cluster can still be cloned.
Do I need to recreate the cluster to use a different PyTorch or CUDA version?
No. Pass --container-image to srun/sbatch and the job runs in whatever image you name. The
cluster's Image is its own Slurm software stack, not your job's environment. See
Containers.
Can two teams use different framework versions on one cluster? Yes — that is exactly what container jobs are for. Different jobs can use completely different images at the same time.
Can I add a partition while the cluster is being created? No. A partition names specific nodes, which do not exist yet. Add it from the Cluster Overview tab once the cluster is Running. See Partitions.
Can I edit slurm.conf over SSH?
You can, and it will be overwritten. The file is regenerated whenever the cluster changes — a scale,
an image update, a volume mount. Use
Slurm Configuration so the change persists.
Why was my slurm.conf line rejected?
Either the key is platform-managed, or the syntax is invalid — most often a space in the value. See
validation messages.
Does scaling down interrupt jobs? Yes — on the removed nodes, and the remaining workers restart too. Drain first. See Scale Cluster.
Will an image update interrupt my jobs? Yes, but gracefully: nodes are drained one at a time, roughly 3–8 minutes each. Long-running jobs are still affected, so schedule it.
Where should checkpoints go? A PFS, SFS or Weka mount. Never a node-local path. See Storage.
How do I see whose job is whose?
Enable Login User Management. Without it
everyone is root and the User column is meaningless.
Can I cancel a job from the console?
No. Use scancel <jobid> over SSH. The Jobs tab is read-only.
Does terminating a cluster delete my data? No. Volumes are unmounted, not deleted — and they keep billing. Delete them separately under Storage when you are done.
Can I get a GPU type that shows "Inventory not available"? Not at that moment. Retry later, or reserve capacity with a Private Cluster.
When to Contact Support
Raise a ticket from Support in the TIR sidebar, or email cloud-platform@e2enetworks.com.
Include the cluster name, the project, and:
| Problem | Also include |
|---|---|
| Stuck in Creating or Terminating | How long, and whether Reconfigure was tried |
| Repeated XID errors | The XID code, the e2e_node_<id> name, and whether a reboot helped |
| A node that will not register | That worker's log from the Logs tab |
| Fabric performance far below expectation | Your NCCL benchmark output and the NCCL_* variables you set |
| A configuration change that fails to apply | The exact configuration and the error message |
Restarting recreates the pods and the previous logs are gone. Grab them from the Logs tab first, or you will be asking support to diagnose something with no evidence.