TIR Slurm Cluster

A TIR Slurm Cluster is a dedicated pool of GPU worker nodes with a managed Slurm scheduler in front of them. You submit jobs to a queue, Slurm allocates nodes and GPUs, and your work runs. E2E manages the Slurm installation, so the scheduler is installed and patched for you.

A cluster gives you three parts: a login node to submit jobs from, a controller that runs the Slurm scheduler, and the GPU worker nodes that run the work. How a Slurm Cluster works explains each one.

Where to find it

In TIR, open your project. In the side menu, under PRODUCTS, select Training Cluster. In the portal, this product is called Training Cluster on the menu, the list page, and the create button. These docs call it a Slurm Cluster. Some accounts see a Training Cluster group in the side menu with Slurm Cluster inside it. The screens are the same.

A Slurm Cluster belongs to one project. The project selector at the top of the console decides which clusters you see.

What you can do

Create and connect

• Create a Slurm Cluster from an image, a GPU plan, SSH keys, and storage.
• Connect to a cluster over SSH and submit jobs from the login node.
• Manage login users so every team member connects with their own login name, keys, and home directory.

Run jobs

• Run containers in jobs: any Docker or OCI image, chosen per job with --container-image and run by the built-in Enroot runtime, with the cluster left as it is.
• Run multi-node training across several nodes over the cluster’s RDMA fabric.
• Cache container images on shared storage so every node and every job reads the same image.

Configure Slurm

• Change the Slurm configuration by adding your own directives to the cluster’s slurm.conf from the console.
• Manage partitions to define your own named queues over chosen nodes, each with its own limits and members.
• Add prolog and epilog scripts to run your own shell scripts around every job, on the worker nodes and on the controller.

Storage and network

• Mount and unmount volumes: storage appears at the same path on the login node and every worker node.
• Manage network and security: the cluster’s public IP, a private VPC address, and security groups.

Watch the cluster

• Check cluster and node health with per-node and per-GPU utilization, memory, temperature, and power, XID error detection, and a single-node reboot.
• See jobs and logs in the console, straight from the browser.
• Monitor a cluster with GPU and scheduler charts over time, and attach alerts that warn you about problems.

Change, copy, and end a cluster

• Update a cluster: replace its root SSH keys, move it to a new software image, or restart its workers, login service, or the whole cluster.
• Scale a cluster by adding or removing worker nodes.
• Clone a cluster to start a new one from an existing setup.
• Convert a cluster to committed once its size is settled.
• Terminate or delete a cluster when you are done with it.

Understand the cluster

• How a Slurm Cluster works: the login node, the controller, the workers, node names, partitions, and which storage survives a restart.
• Cluster statuses and actions: what each status allows, and what each action does to running jobs.
• The cluster details page: where each tab and control is.
• Billing and plans: on-demand, savings, and committed plans, and how to keep the bill down.

Who it is for

A Slurm Cluster suits you when	Something else fits better when
You train across several GPUs or several nodes.	You want a single GPU for interactive work: use a TIR Instance.
A team shares GPU capacity through a queue, with priorities and fair sharing.	One person runs one job at a time.
Your jobs must run in specific container images.	You serve a model: use a model endpoint.
You want a predictable cost for a fixed pool of nodes.	You need a process that stays up on its own rather than a queue.

Before your first long run, read the FAQ answers on keeping your work safe: where to write checkpoints, what to check before a restart, and why each person needs their own login.

Where to start

Start with Get started with TIR Slurm Cluster, which creates a cluster, runs a first GPU job and a container job, and ends the charges when you are done. For help with SSH, jobs, or containers, Troubleshooting TIR Slurm Cluster starts from the symptom you see, and the TIR Slurm Cluster FAQ has short answers.