--- title: "TIR Slurm Cluster" description: "TIR Slurm Cluster is a dedicated pool of GPU nodes with a managed Slurm scheduler for multi-GPU and multi-node training, with container jobs." content_type: "overview" product: "tir-slurm-cluster" platform: "TIR" url: "/tir/tir-slurm-cluster/" --- # TIR Slurm Cluster A TIR Slurm Cluster is a dedicated pool of GPU worker nodes with a managed Slurm scheduler in front of them. You submit jobs to a queue, Slurm allocates nodes and GPUs, and your work runs. E2E manages the Slurm installation, so the scheduler is installed and patched for you. A cluster gives you three parts: a login node to submit jobs from, a controller that runs the Slurm scheduler, and the GPU worker nodes that run the work. [How a Slurm Cluster works](/tir/tir-slurm-cluster/concepts/how-a-slurm-cluster-works/) explains each one. ## Where to find it In TIR, open your project. In the side menu, under **PRODUCTS**, select **Training Cluster**. In the portal, this product is called **Training Cluster** on the menu, the list page, and the create button. These docs call it a Slurm Cluster. Some accounts see a **Training Cluster** group in the side menu with **Slurm Cluster** inside it. The screens are the same. A Slurm Cluster belongs to one project. The project selector at the top of the console decides which clusters you see. ## What you can do ### Create and connect - [Create a Slurm Cluster](/tir/tir-slurm-cluster/how-to/create-a-slurm-cluster/) from an image, a GPU plan, SSH keys, and storage. - [Connect to a cluster](/tir/tir-slurm-cluster/how-to/connect-to-a-cluster/) over SSH and submit jobs from the login node. - [Manage login users](/tir/tir-slurm-cluster/how-to/manage-login-users/) so every team member connects with their own login name, keys, and home directory. ### Run jobs - [Run containers in jobs](/tir/tir-slurm-cluster/how-to/run-containers-in-jobs/): any Docker or OCI image, chosen per job with `--container-image` and run by the built-in Enroot runtime, with the cluster left as it is. - [Run multi-node training](/tir/tir-slurm-cluster/how-to/run-multi-node-training/) across several nodes over the cluster's RDMA fabric. - [Cache container images](/tir/tir-slurm-cluster/how-to/cache-container-images/) on shared storage so every node and every job reads the same image. ### Configure Slurm - [Change the Slurm configuration](/tir/tir-slurm-cluster/how-to/change-the-slurm-configuration/) by adding your own directives to the cluster's `slurm.conf` from the console. - [Manage partitions](/tir/tir-slurm-cluster/how-to/manage-partitions/) to define your own named queues over chosen nodes, each with its own limits and members. - [Add prolog and epilog scripts](/tir/tir-slurm-cluster/how-to/add-prolog-and-epilog-scripts/) to run your own shell scripts around every job, on the worker nodes and on the controller. ### Storage and network - [Mount and unmount volumes](/tir/tir-slurm-cluster/how-to/mount-and-unmount-volumes/): storage appears at the same path on the login node and every worker node. - [Manage network and security](/tir/tir-slurm-cluster/how-to/manage-network-and-security/): the cluster's public IP, a private VPC address, and security groups. ### Watch the cluster - [Check cluster and node health](/tir/tir-slurm-cluster/how-to/check-cluster-and-node-health/) with per-node and per-GPU utilization, memory, temperature, and power, XID error detection, and a single-node reboot. - [See jobs and logs](/tir/tir-slurm-cluster/how-to/see-jobs-and-logs/) in the console, straight from the browser. - [Monitor a cluster](/tir/tir-slurm-cluster/how-to/monitor-a-cluster/) with GPU and scheduler charts over time, and attach alerts that warn you about problems. ### Change, copy, and end a cluster - [Update a cluster](/tir/tir-slurm-cluster/how-to/update-a-cluster/): replace its root SSH keys, move it to a new software image, or restart its workers, login service, or the whole cluster. - [Scale a cluster](/tir/tir-slurm-cluster/how-to/scale-a-cluster/) by adding or removing worker nodes. - [Clone a cluster](/tir/tir-slurm-cluster/how-to/clone-a-cluster/) to start a new one from an existing setup. - [Convert a cluster to committed](/tir/tir-slurm-cluster/how-to/convert-a-cluster-to-committed/) once its size is settled. - [Terminate or delete a cluster](/tir/tir-slurm-cluster/how-to/terminate-or-delete-a-cluster/) when you are done with it. ## Understand the cluster - [How a Slurm Cluster works](/tir/tir-slurm-cluster/concepts/how-a-slurm-cluster-works/): the login node, the controller, the workers, node names, partitions, and which storage survives a restart. - [Cluster statuses and actions](/tir/tir-slurm-cluster/concepts/cluster-statuses-and-actions/): what each status allows, and what each action does to running jobs. - [The cluster details page](/tir/tir-slurm-cluster/concepts/the-cluster-details-page/): where each tab and control is. - [Billing and plans](/tir/tir-slurm-cluster/concepts/billing-and-plans/): on-demand, savings, and committed plans, and how to keep the bill down. ## Who it is for | A Slurm Cluster suits you when | Something else fits better when | | ----------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- | | You train across several GPUs or several nodes. | You want a single GPU for interactive work: use a [TIR Instance](/tir/tir-instance/). | | A team shares GPU capacity through a queue, with priorities and fair sharing. | One person runs one job at a time. | | Your jobs must run in specific container images. | You serve a model: use a [model endpoint](/tir/model-endpoints/). | | You want a predictable cost for a fixed pool of nodes. | You need a process that stays up on its own rather than a queue. | Before your first long run, read the [FAQ answers on keeping your work safe](/tir/tir-slurm-cluster/faq/#keep-your-work-safe): where to write checkpoints, what to check before a restart, and why each person needs their own login. ## Where to start Start with [Get started with TIR Slurm Cluster](/tir/tir-slurm-cluster/get-started/), which creates a cluster, runs a first GPU job and a container job, and ends the charges when you are done. For help with SSH, jobs, or containers, [Troubleshooting TIR Slurm Cluster](/tir/tir-slurm-cluster/troubleshooting/tir-slurm-cluster/) starts from the symptom you see, and the [TIR Slurm Cluster FAQ](/tir/tir-slurm-cluster/faq/) has short answers.