Features
What a TIR Slurm Cluster gives you beyond a scheduler, and where each capability is documented.
Containers with Enroot and Pyxis
Run any Docker or OCI image inside a Slurm job with one extra flag. No cluster rebuild, no Docker daemon, native GPU access.
→Editable Slurm configuration
Append your own slurm.conf directives, add partitions over chosen nodes, and run prolog and epilog scripts — all from the console.
→Custom partitions
Carve the cluster into named queues with their own time limits, defaults and node membership. Up to 20 per cluster.
→Elastic scaling
Add or remove worker nodes on a live cluster. Scale-up leaves running jobs untouched.
→GPU health and DCGM metrics
Per-node and per-GPU utilisation, memory, temperature and power, with XID error detection and single-node reboot.
→Jobs and monitoring without SSH
Live squeue in the Jobs tab, GPU and scheduler charts in Monitoring, controller and worker logs in Logs.
→Per-user logins
Give every teammate their own SSH identity and private home directory on shared storage, managed from the console.
→Shared storage on every node
PFS, SFS, Weka and read-only datasets mounted at the same path on the login node and every worker.
→Feature Reference
| Feature | Summary | Details |
|---|---|---|
| Slurm on Slinky | A real, managed Slurm cluster — sbatch, squeue, sacct, partitions, priorities and accounting all behave as documented. You never install or patch a scheduler | How it works |
| Container jobs | Run any Docker or OCI image inside a job with --container-image, unprivileged, with native GPU access. Two jobs on one cluster can use completely different framework versions | Containers |
Editable slurm.conf | Append your own directives from the console. The platform keeps only the keys that wire the cluster together | Extra slurm.conf |
| Custom partitions | Up to 20 named queues over chosen nodes, each with its own time limits, defaults and node membership | Partitions |
| Prolog & epilog scripts | Four script slots — worker and controller, before and after every job — for setup, cleanup, health gating and auditing | Prolog & Epilog |
| Elastic scaling | Add or remove worker nodes on a live cluster. Scale-up leaves running jobs untouched | Scale Cluster |
| Shared storage everywhere | PFS, SFS, Weka and read-only datasets mounted at the same path on the login node and every worker | Storage |
| GPU health & DCGM metrics | Per-node and per-GPU utilisation, memory, temperature and power, with XID error detection and single-node reboot | Nodes |
| Jobs & monitoring without SSH | Live squeue in the browser, GPU and scheduler charts over selectable time ranges, controller and worker logs | Jobs · Monitoring |
| Alerts | Attach alerts to a cluster so problems find you instead of the other way round | Alerts |
| Per-user logins | Each teammate gets their own SSH identity and a private home directory on shared storage, managed from the console | Login User Management |
| High availability | Node failures are isolated and the cluster recovers automatically. Job-level resilience is yours, via checkpointing | High availability |
| Network control | Reserved and VPC IPs, and security groups you manage per cluster | Network & security |
| Full API coverage | Every console action maps to a REST call, so provisioning and management can be automated end to end | API reference |
Slurm Cluster vs a Single GPU Node
| Slurm Cluster | Single node | |
|---|---|---|
| Multi-node training | Yes, with an RDMA fabric between nodes | No |
| Job queue and priorities | Yes | No |
| Sharing between people | Yes, with per-user identities and partitions | Awkward |
| Container per job | Yes | You manage the container yourself |
| Billing | Per node, per hour; jobs free | Per node, per hour |
| Best for | Distributed training, shared team capacity, batch workloads | Interactive development, single-GPU experiments |
Best Practices for Slurm Clusters
A worker or login pod's own filesystem is scratch — it is wiped on restart, scale, image update and node reboot. Checkpoints, logs, datasets and Python environments belong on a PFS or SFS mount, which is identical on the login node and every worker.
Run enroot import once on the login node into a shared path, then point jobs at the .sqsh file. Otherwise every node and every array task pulls the image again.
A node failure, a preemption or a restart ends the job. Your progress is only as good as the last checkpoint written to shared storage.
Slurm drains the node when a prolog fails. Finish every worker prolog with exit 0 unless draining is exactly what you intend.
No action inspects the Slurm queue. Restart, scale-down and terminate all proceed with jobs in flight and lose them — only Update Image drains nodes first.
With everyone as root, job ownership is meaningless and scancel -u root cancels the whole cluster. Enable Login User Management for per-user identities and home directories.
A partition with a one-hour Max time keeps interactive debugging out from behind a 24-hour training run, without anyone having to coordinate.
Terminating a cluster unmounts your volumes but does not delete them. Review PFS, SFS and datasets separately when you clean up.