Create a Slurm Cluster

Create a Slurm Cluster in your project: a login node to submit jobs from, a controller, and the GPU worker nodes you choose. The create form is a single page, and you fill it in from top to bottom.

Before you begin

• At least one SSH key in TIR. The form requires one, and it is how you log in as root. Add a key with the Click here link beside the SSH Keys field, or see Add an SSH key.
• At least one storage volume to mount: a Parallel File System, a Shared File System, or a dataset. Your data, checkpoints, and container images live there.
• A security group with inbound TCP 22 open, so you can reach the login node over SSH. The default SSH security group works. To make another, see Create a security group.
• Enough credit for the plan and node count you choose. TIR checks this when you submit.

If you expect to create and delete clusters often, or need the capacity guaranteed, create a TIR Private Cluster first. A Slurm Cluster built on one draws its nodes from that reserved pool instead of on-demand inventory.

Open the create form

In TIR, open your project. In the side menu, under PRODUCTS, select Training Cluster. The list page is headed Manage Training Clusters. Select Create Training Cluster.

Name the cluster

Cluster Name is required. It starts with a generated name such as training-cluster-<timestamp>, which you can replace with something that tells your team what the cluster is for.

A name can be up to 50 characters. It starts with a lowercase letter, then uses only lowercase letters, digits, and hyphens, and it ends with a letter or a digit.

Choose an image

Choose an Image family, then a version within it.

Each image version comes with GPU drivers, CUDA, NCCL, the Slurm runtime, and the Enroot and Pyxis container tools already installed. The families include:

Family	Use it for
Ubuntu Slurm	A general-purpose Slurm environment. Start here, and bring your framework in a container.
NeMo Framework	NVIDIA NeMo and Megatron-style training, with the framework already installed on the nodes.

A version is named for what it holds, for example v2-slurm25.11-ubuntu24.04 for Slurm 25.11 on Ubuntu 24.04. The Details tab shows the version afterwards.

The image you choose here is the cluster’s own software: Slurm, the drivers, and the operating system. You choose what each job runs in separately, with --container-image, so a different PyTorch or CUDA version runs on the cluster you have. An Ubuntu Slurm cluster with a container per job lets two teams use different framework versions on the same cluster, and a framework upgrade happens entirely in the container image. See Run containers in jobs.

You can change the cluster’s image later with Update Image. See Update a cluster.

Choose a plan

Under Plan Configuration, the tabs say where the capacity comes from:

• GPU is on-demand inventory.
• Private Cluster is capacity your account has already reserved.

On the GPU tab

Each plan names the GPU model and the shape of one node: GPU count, vCPU count, RAM, and GPU memory, for example “8 GPU, 240 CPU, 3000 GB RAM, 141GB GPU Memory”. A plan shows Inventory not available while every node of its type is taken, and you can select it only when nodes are free. Choose another GPU type, try again later, or reserve capacity with a TIR Private Cluster.

Set Workers, the number of worker nodes, to at least one. Every GPU plan is a whole node with 8 GPUs, so the smallest cluster is one node with 8 GPUs. The price shown for a plan is per node, so two nodes cost twice that price.

The Pricing section offers On-Demand, hourly billing that you can end any time, and the committed terms the plan offers. A term appears with its saving:

Term	Saving
1 Month Savings Plan	Save 10%
3 Month Savings Plan	Save 14%
6 Month Savings Plan	Save 20%
1 Year Savings Plan	Save 25%
2 Year Savings Plan	Save 30%
3 Year Savings Plan	Save 35%

E2E’s price list sets which terms a plan offers, so a plan can show only some of these. For the rates, see E2E Networks pricing. To compare the options, see Billing and plans.

Size the cluster before you commit to a term, because a committed cluster keeps its node count for the whole term. On On-Demand, scaling down restarts the workers that remain and ends their jobs, so it is worth getting the size close from the start. If you are unsure, start with fewer nodes on On-Demand and scale up once the queue shows what you need; see Scale a cluster.

On the Private Cluster tab

Choose one of your private clusters, then set Workers to between one and the number of nodes it has free.

A Slurm Cluster on a private cluster has no separate hourly rate. It is billed as part of your private cluster plan.

Add SSH keys

Under Access, SSH Keys is required. Select one or more keys. They give root access to the login node.

Add a second key now, for example a colleague’s or a backup key. A cluster with one key is one lost laptop away from being unreachable. Replace the keys later with Update SSH Keys, and Manage login users gives each person their own access.

Add storage

You need at least one of a Parallel File System, a Shared File System, or a dataset, and you can choose more than one. Each volume you choose gets a mount path, and it appears at that path on the login node and on every worker node. Choose a short, obvious path such as /shared, and keep it the same across clusters so job scripts move between them unchanged.

• Parallel File System is offered in Delhi; in Chennai, use a Shared File System. The list offers only volumes that are in a healthy state.
• Shared File System (optional) is read-write.
• Dataset Storage (optional) is mounted read-only, so it suits input data. Jobs can only read from it: a write, such as a checkpoint or a log, fails with a permission error.

A mount path must start with / and name a folder below it, such as /shared. Separate folders with a single /, end with the folder name, and leave out spaces. Every mount path on the cluster must be unique across your Parallel File System, Shared File System, and dataset volumes.

If a team will use Login User Management, attach a Shared File System or a Parallel File System now. Mounting a volume later restarts the cluster’s nodes and ends running jobs. See Mount and unmount volumes.

Set the advanced options

Expand Advanced Settings for the rest.

Security Group is required. Once you have selected SSH keys, the form selects your SSH security group for you, and you can choose only a group with port 22 open; the others show Port 22 disabled. Keep a group with inbound TCP 22 open, because that is how you reach the login node. To limit who can connect, narrow the group’s source range later and keep the rule itself. See Manage network and security.

(Optional) Lifecycle Script runs once on each node after the cluster is created. Select Add Script to write it in the code editor, or Upload Script to load a shell script or a Python file. Use it for one-time node setup, such as installing a package or writing a config file. You can set it only when you create the cluster. It is separate from a worker prolog, which runs before every job and which you can change while the cluster is Running, so the two run independently. Keep credentials out of it, because the script is stored with the cluster configuration and copied into any clone. Read a secret from a file on your mounted storage instead.

(Optional) Slurm Configuration lets you set up the cluster’s extra slurm.conf directives and its prolog and epilog scripts before the cluster exists. Select Configure Slurm, use the slurm.conf and Prolog & Epilog tabs, and confirm with Save Configuration. The text beside the button reads Not configured, or lists what you set, such as extra slurm.conf, 1 script. Clear Configuration discards it. Partitions come later, because a partition names specific nodes, and the nodes exist only once the cluster is running. You can also skip it and change everything in the dialog later, while the cluster is Running. See Change the Slurm configuration and Manage partitions.

Create the cluster

Billing starts when the create succeeds and is dated from the moment you submit the form, before any job runs. A cluster costs the same idle as busy, so create it when you are ready to use it. A create that fails is free of compute charges.

Select Create Training Cluster. The button becomes available once every field is valid. TIR checks that you chose a plan and at least one storage volume, that your mount paths are unique, and that you have enough credit for the plan and node count. If something is missing, the form names it under the field, with messages such as:

• “Cluster name is required” or “Invalid name format”
• “Please select a plan configuration”
• “Please select at least one SSH key”
• “Please select a Security Group”, or “Security Group with port 22 is required when SSH Key is added”
• “Please select at least one storage (PFS, SFS, or Dataset)” or “Mount path is required”
• “Node count cannot be less than 1”
• “Please upload a valid .sh or .py file” for a lifecycle script

Watch it come up

The cluster is Creating in the list while it sets up, which usually takes a few minutes. The status then moves to Initializing Worker Nodes, which means the controller and login node are up and the workers are still joining, and then to Running. From Initializing Worker Nodes you can already select Connect. For all the statuses, see Cluster statuses and actions.

If a cluster stays in Creating for much longer than a few minutes, or ends in Failed, follow the steps in Troubleshooting TIR Slurm Cluster. A failed create is free.

Once it is Running, your cluster is ready for work. Connect to a cluster to reach the login node, or follow Get started with TIR Slurm Cluster for a first job. Now that the nodes exist, split them into queues with Manage partitions, and save your images once with Cache container images. If a team will share the cluster, give each person a login with Manage login users.