How a Slurm Cluster Works
A TIR Slurm Cluster is a genuine Slurm installation — sinfo, sbatch, squeue, sacct all behave
the way the Slurm documentation says. What TIR manages is the installation itself: the cluster runs as
Kubernetes workloads, provisioned through the Slinky Slurm
operator, so you never install or patch a scheduler.
The Parts of a Cluster
you your Slurm Cluster
─── ──────────────────
ssh root@<ip> ────────────► ┌─ Login node ────────────────────────────┐
│ sshd; sbatch / srun / squeue / sinfo │
│ your PFS, SFS and dataset mounts │
└────────────────┬───────────────────────┘
│ submit
┌────────────────▼───────────────────────┐
│ Controller (slurmctld) │
│ schedules jobs, owns slurm.conf │
│ + accounting database (sacct) │
└────────────────┬───────────────────────┘
│ allocate
┌────────────────────────────▼────────────────────────────┐
│ Worker nodes slinky-0 slinky-1 … slinky-(N-1) │
│ 8 GPUs each (plan-dependent), RDMA fabric between them │
│ the same PFS / SFS / dataset mounts, same paths │
└─────────────────────────────────────────────────────────┘
| Part | What it does | How you interact with it |
|---|---|---|
| Login node | Where SSH lands. You edit scripts, submit jobs, and import container images here | ssh root@<cluster-ip>. Restart it with Restart Login Service |
| Controller | Runs slurmctld. Holds the queue, makes scheduling decisions, owns slurm.conf | Indirectly — through Slurm Configuration and the Jobs tab |
| Worker nodes | The GPU machines that run your jobs, one Slurm node each | --nodes, --gres=gpu:N; visible on the Nodes tab |
| Accounting database | Records every job so sacct works | sacct on the login node |
| Storage | Your PFS, SFS, Weka and dataset volumes, mounted at the same path on the login node and every worker | The Volumes tab |
It exists to submit from. It has no GPUs allocated to your jobs, and running training there directly
does not use the cluster — it just loads the login node until it falls over. Everything real goes
through sbatch or srun.
It is the only place with both your storage and outbound network. Import container images there,
stage datasets there, and keep your job scripts there — on a mounted volume, not in /root.
Node Names
Slurm knows your GPU workers as:
slinky-0, slinky-1, slinky-2, … slinky-(N-1)
Some clusters also carry a CPU worker group, named cpu-workers-0 … cpu-workers-7.
These are the names you use everywhere Slurm expects a node: --nodelist, scontrol show node, and
the Nodes in this partition field when
defining a partition. Bracket
ranges work — slinky-[0-3], slinky-[0-1,5,8-10].
The console shows two names side by side, for example e2e_node_682 (slinky-0):
| Name | What it is |
|---|---|
slinky-0 | The Slurm node name. Use this in job scripts and partitions |
e2e_node_682 | The physical machine identifier. Use it when reporting a hardware problem to support |
Scaling up adds the next ordinals, so a 4-node cluster scaled to 6 gains slinky-4 and slinky-5.
Partitions
A partition is a named queue over a set of nodes. Every cluster starts with partitions the platform manages:
| Partition | Covers | Notes |
|---|---|---|
all | Every node | The default partition — where jobs land without --partition |
slinky | The GPU worker group | Created alongside the node group |
cpu-workers | The CPU worker group, where present |
Depending on the region, a cluster may also carry additional platform partitions such as llm,
asr, vision and tts.
So the first sinfo shows more partitions than you created — that is expected:
sinfo
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
all* up infinite 4 idle slinky-[0-3]
slinky up infinite 4 idle slinky-[0-3]
The * marks the default partition. all and slinky both cover the same nodes here; all is the
one to submit to unless you have a reason not to.
You can add up to 20 of your own partitions, each with its own node membership, time limits and defaults. That is how you separate short interactive work from long training runs, or reserve nodes for one team. See Manage partitions.
scontrol show partition all # everything Slurm knows about one partition
Storage: What Persists and What Does Not
This is the single most important thing to understand before you run anything real.
| Location | Persists? | Shared across nodes? | Use it for |
|---|---|---|---|
PFS mount (e.g. /pfs) | Yes | Yes | Datasets, checkpoints, logs, container squash files, job scripts |
SFS mount (e.g. /shared) | Yes | Yes | The same, where you want POSIX shared storage |
| Weka mount | Yes | Yes | High-performance shared storage where available |
| Dataset mount | Yes, read-only | Yes | Input data you never write to |
/root, /home (without per-user logins) | No | No | Nothing you care about |
/tmp on a worker | No | No | Per-job scratch, cleaned up by an epilog |
/dev/shm | No — memory-backed | No | Fast per-job shared memory |
Restarting workers, restarting the login service, scaling, updating the image and rebooting a node all
recreate pods and erase their local filesystems. That includes anything you pip installed
interactively, any checkpoint written outside a mounted volume, and anything in /root.
Write everything you want to keep to a PFS or SFS mount. This is the most common cause of lost work on a Slurm Cluster.
Put your Python environment on shared storage — a virtualenv under /pfs/envs/…, or better, bake it
into a container image. Then a restart costs you nothing, and
every node sees the identical environment.
With Login User Management enabled, each user gets a home directory on shared storage, which does survive restarts.
Container Jobs
Slurm jobs can run inside a container image without the cluster knowing anything about it in advance:
srun --container-image=nvcr.io/nvidia/pytorch:25.09-py3 \
--container-mounts=/pfs:/pfs \
python train.py
This is Enroot (an unprivileged container runtime) plus
Pyxis (the Slurm plugin that adds the --container-* flags). Both
are part of the cluster image; nothing to install.
| Question | Answer |
|---|---|
| Does it need root? | No — Enroot is unprivileged |
| Do GPUs work inside? | Yes. nvidia-smi shows exactly the GPUs the job was allocated |
| Can two jobs use different images? | Yes — the image is per job, not per cluster |
| Does it work across nodes? | Yes. One srun starts the container on every node in the allocation |
| Is my storage visible inside? | Only what you pass with --container-mounts |
The Image you pick when creating the cluster is a different thing entirely: it is the cluster's own Slurm software stack, not your job's container. See Containers with Enroot and Pyxis.
Cluster Statuses
| Status | Meaning | What you can do |
|---|---|---|
| Creating | Provisioning. Usually a few minutes | Wait. Reconfigure Cluster if it stalls |
| Running | Live and accepting jobs | Everything |
| Failed | Creation did not complete | Clone it into a new cluster, then delete the failed record |
| Terminating / Terminated | Shutting down / shut down. The record remains | Clone or delete it |
| Deleting / Deleted | Being removed / removed from the list | Nothing |
Most actions require Running — see Action availability.
How a cluster is changed
Every change you make — scaling, an image update, mounting a volume, editing slurm.conf, updating
SSH keys — is applied by re-rendering the cluster's desired state and reconciling it. Two consequences
worth internalising:
- Editing files on the controller by hand does not last.
slurm.confin particular is regenerated on the next change of any kind. Use the Slurm Configuration dialog. - A failed change is rolled back automatically. If applying a configuration fails, the previous one is restored and you get an error rather than a half-changed cluster.
High Availability, and What It Does Not Cover
| Guaranteed | Not guaranteed |
|---|---|
| A node failure does not take down the controller or other nodes | A job running on the failed node survives |
| The cluster recovers on its own after a node restart | Anything on that node's local filesystem survives |
| The queue and job history survive a controller restart | A job whose node vanished is restarted for you |
Node isolation protects the cluster, not your run. A failure, a preemption or a restart ends the job — so checkpoint to shared storage regularly and make your training script resume from the last checkpoint. On a multi-day run this is the difference between losing minutes and losing days.
Node health, including NVIDIA XID hardware errors, is surfaced on the Nodes tab, where a single bad node can be rebooted without touching the rest of the cluster.