--- title: "How a Slurm Cluster works" description: "How a TIR Slurm Cluster is laid out and how a job moves through it, from the login node and controller to worker nodes, partitions, and storage." content_type: "concept" product: "tir-slurm-cluster" platform: "TIR" url: "/tir/tir-slurm-cluster/concepts/how-a-slurm-cluster-works/" --- # How a Slurm Cluster works A TIR Slurm Cluster is a Slurm installation that E2E runs for you. It has a login node to submit from, a controller that schedules, and GPU worker nodes that run the work. `sinfo`, `sbatch`, `squeue`, and `sacct` behave the way the Slurm documentation says they do. E2E installs and patches the scheduler for you. Knowing how the parts fit together tells you where to run a command and where to keep a file. ## The parts of a cluster ```mermaid flowchart TD You["You, over SSH"] --> Login["Login node: submit jobs, your storage mounts"] Login --> Controller["Controller: job queue, scheduling, accounting"] Controller --> Workers["Worker nodes slinky-0 to slinky-N: GPUs, RDMA fabric between them"] Storage["Shared storage: PFS, SFS, datasets"] --- Login Storage --- Workers ``` You connect to the login node over SSH. The controller holds the job queue and decides where jobs run. The worker nodes run the jobs. Shared storage is mounted at the same path on the login node and on every worker node. | Part | What it does | How you reach it | | ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | | Login node | Where SSH lands. It runs the SSH service and the Slurm client commands (`sbatch`, `srun`, `squeue`, `sinfo`) and carries your storage mounts. It is the only place with both your storage and outbound network. | `ssh root@`. See [Connect to a cluster](/tir/tir-slurm-cluster/how-to/connect-to-a-cluster/). Restart it with **Restart Login Service**. | | Controller | Runs `slurmctld`, the Slurm scheduler. It holds the job queue, makes scheduling decisions, and owns `slurm.conf`. | Indirectly, through the Slurm Configuration dialog and the **Jobs** tab. | | Worker nodes | The GPU machines that run your jobs, one Slurm node each, with an RDMA fabric between them. A plan sets the number of GPUs on each. | `--nodes` and `--gres=gpu:N` in your job. The **Nodes** tab shows them. | | Accounting database | Records every job so that `sacct` works. E2E manages it fully. | `sacct` on the login node. | The login node is where you submit work from. The GPUs for your jobs are on the worker nodes, and training started directly on the login node does not use them. Send training through `sbatch` or `srun`, which run it there. Use the login node to import container images and stage datasets, and keep job scripts on a mounted volume rather than in `/root`. ## How a job moves through the cluster ```mermaid flowchart LR Submit["Submit with sbatch or srun on the login node"] --> Queue["Slurm puts the job in the queue"] Queue --> Allocate["Slurm allocates nodes and GPUs"] Allocate --> Run["The job runs on the worker nodes"] Run --> Output["Output is written to shared storage"] ``` You submit a job on the login node. Slurm puts it in the queue, allocates nodes and GPUs when they are free, and runs the job on the worker nodes. A job that writes its output to shared storage keeps it there after the job ends. For your first job, see [Get started with TIR Slurm Cluster](/tir/tir-slurm-cluster/get-started/). ## Node names Slurm knows your GPU workers by the names `slinky-0`, `slinky-1`, and so on up to `slinky-(N-1)`. These are the names you use wherever Slurm expects a node: `--nodelist`, `scontrol show node`, and partition definitions. Bracket ranges such as `slinky-[0-3]` and `slinky-[0-1,5,8-10]` work. The console shows a node's Slurm name beside its physical machine identifier, for example `e2e_node_682 (slinky-0)`. Use the Slurm name in job scripts and partitions. Give the physical machine identifier to support when you report a hardware problem. Scaling a cluster up adds the next names in order: a four-node cluster scaled to six gains `slinky-4` and `slinky-5`. ## Partitions A partition is a named queue over a set of nodes. Every cluster has two partitions that E2E manages for you: - `all` covers every node. It is the default partition, so a job runs there unless it sets `--partition`. - `slinky` covers the GPU worker nodes. Your first `sinfo` already shows them: ```text PARTITION AVAIL TIMELIMIT NODES STATE NODELIST all* up infinite 4 idle slinky-[0-3] slinky up infinite 4 idle slinky-[0-3] ``` The asterisk marks the default partition. Here `all` and `slinky` cover the same nodes, and `all` is the usual one to submit to. You cannot edit these two, but you can add up to 20 partitions of your own to separate short interactive work from long training runs, or to set nodes aside for one team. To see everything Slurm knows about one partition, run `scontrol show partition all`. See [Manage partitions](/tir/tir-slurm-cluster/how-to/manage-partitions/). ## What is kept after a restart | Location | Kept after a restart? | Shared across nodes? | Use it for | | ------------------------------------------------------- | ------------------------ | -------------------- | --------------------------------------------------------------------- | | A Parallel File System or Shared File System mount | Yes | Yes | Datasets, checkpoints, logs, container squash files, and job scripts. | | A dataset mount | Yes, and it is read-only | Yes | Input data that jobs only read. | | `/root`, and `/home` while Login User Management is off | No | No | Files you can afford to lose. | | `/tmp` on a worker | No | No | Scratch space for one job. | | `/dev/shm` | No, it is memory-backed | No | Fast memory-backed space for one job. | A node's own disk is erased whenever the node is recreated. That happens when you restart workers, restart the login service, scale the cluster down, update its image, or mount or unmount a volume. The erase takes anything you installed by hand with `pip`, any checkpoint written outside a mounted volume, and anything in `/root`. The login node's own disk is erased the same way, by a login service restart, an image update, or a reconfigure. This is the most common cause of lost work on a Slurm Cluster, so write everything you want to keep to a Parallel File System or Shared File System mount. Put your Python environment on shared storage too, for example a virtual environment on a mounted volume, or better, build it into a container image. A restart then leaves your environment in place, and every node sees the same environment. With Login User Management turned on, each user's home directory lives on a Shared File System or Parallel File System volume and is kept across restarts. See [Manage login users](/tir/tir-slurm-cluster/how-to/manage-login-users/). ## Container jobs A Slurm job can run inside a container image that you name in the job, and the cluster needs no preparation: ```bash srun --container-image=nvcr.io/nvidia/pytorch:25.09-py3 \ --container-mounts=/shared:/shared \ python train.py ``` Enroot, an unprivileged container runtime, and Pyxis, the Slurm plugin that adds the `--container-*` flags, are part of the cluster image. Because each job names its own image, two teams can use different framework versions on one cluster at the same time. The **Image** you choose when you create the cluster is a different thing: it is the cluster's own Slurm, drivers, and operating system. See [Run containers in jobs](/tir/tir-slurm-cluster/how-to/run-containers-in-jobs/). ## How a cluster is changed E2E applies every change to a running cluster from the cluster's stored settings. That covers scaling, an image update, mounting a volume, editing `slurm.conf`, and updating SSH keys. The next change of any kind overwrites a hand edit of `slurm.conf` on the controller, so make your changes in the Slurm Configuration dialog. See [Change the Slurm configuration](/tir/tir-slurm-cluster/how-to/change-the-slurm-configuration/). When a change fails to apply, TIR refuses it with an error and puts the stored settings back as they were. A cluster also has a status, which says what you can do with it at a given moment. See [Cluster statuses and actions](/tir/tir-slurm-cluster/concepts/cluster-statuses-and-actions/). ## What high availability covers The controller and the other nodes keep running when a node fails, and the cluster recovers on its own after a node restart. The job queue and the job history survive a controller restart. The login service and the Slurm REST API each run as one copy, so the login node is not covered. A failure, a preemption, or a restart ends the jobs running on that node. Restarting those jobs is up to you. Checkpoint to shared storage often, and make your training script resume from the latest checkpoint; on a multi-day run, that is the difference between losing minutes and losing days. The **Nodes** tab shows node health, including XID hardware errors, and lets you reboot a single node and leave the rest of the cluster running. See [Check cluster and node health](/tir/tir-slurm-cluster/how-to/check-cluster-and-node-health/).