How a Slurm Cluster works

A TIR Slurm Cluster is a Slurm installation that E2E runs for you. It has a login node to submit from, a controller that schedules, and GPU worker nodes that run the work. sinfo, sbatch, squeue, and sacct behave the way the Slurm documentation says they do. E2E installs and patches the scheduler for you. Knowing how the parts fit together tells you where to run a command and where to keep a file.

The parts of a cluster

Diagram as text

flowchart TD
    You["You, over SSH"] --> Login["Login node: submit jobs, your storage mounts"]
    Login --> Controller["Controller: job queue, scheduling, accounting"]
    Controller --> Workers["Worker nodes slinky-0 to slinky-N: GPUs, RDMA fabric between them"]
    Storage["Shared storage: PFS, SFS, datasets"] --- Login
    Storage --- Workers

You connect to the login node over SSH. The controller holds the job queue and decides where jobs run. The worker nodes run the jobs. Shared storage is mounted at the same path on the login node and on every worker node.

Part	What it does	How you reach it
Login node	Where SSH lands. It runs the SSH service and the Slurm client commands (sbatch, srun, squeue, sinfo) and carries your storage mounts. It is the only place with both your storage and outbound network.	ssh root@<cluster-ip>. See Connect to a cluster. Restart it with Restart Login Service.
Controller	Runs slurmctld, the Slurm scheduler. It holds the job queue, makes scheduling decisions, and owns slurm.conf.	Indirectly, through the Slurm Configuration dialog and the Jobs tab.
Worker nodes	The GPU machines that run your jobs, one Slurm node each, with an RDMA fabric between them. A plan sets the number of GPUs on each.	--nodes and --gres=gpu:N in your job. The Nodes tab shows them.
Accounting database	Records every job so that sacct works. E2E manages it fully.	sacct on the login node.

The login node is where you submit work from. The GPUs for your jobs are on the worker nodes, and training started directly on the login node does not use them. Send training through sbatch or srun, which run it there. Use the login node to import container images and stage datasets, and keep job scripts on a mounted volume rather than in /root.

How a job moves through the cluster

Diagram as text

flowchart LR
    Submit["Submit with sbatch or srun on the login node"] --> Queue["Slurm puts the job in the queue"]
    Queue --> Allocate["Slurm allocates nodes and GPUs"]
    Allocate --> Run["The job runs on the worker nodes"]
    Run --> Output["Output is written to shared storage"]

You submit a job on the login node. Slurm puts it in the queue, allocates nodes and GPUs when they are free, and runs the job on the worker nodes. A job that writes its output to shared storage keeps it there after the job ends. For your first job, see Get started with TIR Slurm Cluster.

Node names

Slurm knows your GPU workers by the names slinky-0, slinky-1, and so on up to slinky-(N-1). These are the names you use wherever Slurm expects a node: --nodelist, scontrol show node, and partition definitions. Bracket ranges such as slinky-[0-3] and slinky-[0-1,5,8-10] work.

The console shows a node’s Slurm name beside its physical machine identifier, for example e2e_node_682 (slinky-0). Use the Slurm name in job scripts and partitions. Give the physical machine identifier to support when you report a hardware problem.

Scaling a cluster up adds the next names in order: a four-node cluster scaled to six gains slinky-4 and slinky-5.

Partitions

A partition is a named queue over a set of nodes. Every cluster has two partitions that E2E manages for you:

• all covers every node. It is the default partition, so a job runs there unless it sets --partition.
• slinky covers the GPU worker nodes.

Your first sinfo already shows them:

PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
all*         up   infinite      4   idle slinky-[0-3]
slinky       up   infinite      4   idle slinky-[0-3]

The asterisk marks the default partition. Here all and slinky cover the same nodes, and all is the usual one to submit to. You cannot edit these two, but you can add up to 20 partitions of your own to separate short interactive work from long training runs, or to set nodes aside for one team. To see everything Slurm knows about one partition, run scontrol show partition all. See Manage partitions.

What is kept after a restart

Location	Kept after a restart?	Shared across nodes?	Use it for
A Parallel File System or Shared File System mount	Yes	Yes	Datasets, checkpoints, logs, container squash files, and job scripts.
A dataset mount	Yes, and it is read-only	Yes	Input data that jobs only read.
/root, and /home while Login User Management is off	No	No	Files you can afford to lose.
/tmp on a worker	No	No	Scratch space for one job.
/dev/shm	No, it is memory-backed	No	Fast memory-backed space for one job.

A node’s own disk is erased whenever the node is recreated. That happens when you restart workers, restart the login service, scale the cluster down, update its image, or mount or unmount a volume. The erase takes anything you installed by hand with pip, any checkpoint written outside a mounted volume, and anything in /root. The login node’s own disk is erased the same way, by a login service restart, an image update, or a reconfigure. This is the most common cause of lost work on a Slurm Cluster, so write everything you want to keep to a Parallel File System or Shared File System mount.

Put your Python environment on shared storage too, for example a virtual environment on a mounted volume, or better, build it into a container image. A restart then leaves your environment in place, and every node sees the same environment.

With Login User Management turned on, each user’s home directory lives on a Shared File System or Parallel File System volume and is kept across restarts. See Manage login users.

Container jobs

A Slurm job can run inside a container image that you name in the job, and the cluster needs no preparation:

srun --container-image=nvcr.io/nvidia/pytorch:25.09-py3 \
     --container-mounts=/shared:/shared \
     python train.py

Enroot, an unprivileged container runtime, and Pyxis, the Slurm plugin that adds the --container-* flags, are part of the cluster image. Because each job names its own image, two teams can use different framework versions on one cluster at the same time. The Image you choose when you create the cluster is a different thing: it is the cluster’s own Slurm, drivers, and operating system. See Run containers in jobs.

How a cluster is changed

E2E applies every change to a running cluster from the cluster’s stored settings. That covers scaling, an image update, mounting a volume, editing slurm.conf, and updating SSH keys. The next change of any kind overwrites a hand edit of slurm.conf on the controller, so make your changes in the Slurm Configuration dialog. See Change the Slurm configuration.

When a change fails to apply, TIR refuses it with an error and puts the stored settings back as they were.

A cluster also has a status, which says what you can do with it at a given moment. See Cluster statuses and actions.

What high availability covers

The controller and the other nodes keep running when a node fails, and the cluster recovers on its own after a node restart. The job queue and the job history survive a controller restart. The login service and the Slurm REST API each run as one copy, so the login node is not covered.

A failure, a preemption, or a restart ends the jobs running on that node. Restarting those jobs is up to you. Checkpoint to shared storage often, and make your training script resume from the latest checkpoint; on a multi-day run, that is the difference between losing minutes and losing days. The Nodes tab shows node health, including XID hardware errors, and lets you reboot a single node and leave the rest of the cluster running. See Check cluster and node health.