--- title: Create a Slurm Cluster sidebar_label: Create a cluster --- import { Tag, Box, Cpu, Key, HardDrive, Settings, CheckCircle } from 'react-feather'; # Create a Slurm Cluster The create form is one page. This walks it top to bottom — every field, what it accepts, and the decisions worth thinking about before you click. }, { href: '#cluster-name', label: 'Cluster Name', icon: }, { href: '#image', label: 'Image', icon: }, { href: '#plan-configuration', label: 'Plan Configuration', icon: }, { href: '#access', label: 'Access', icon: }, { href: '#advanced-settings', label: 'Advanced Settings', icon: }, { href: '#create-the-cluster', label: 'Create', icon: }, ]} /> --- ## Open the Create Form 1. Log in to the [TIR console](https://tir.e2enetworks.com/). 2. Select your project in the top-left project selector. 3. Click **Training Cluster** in the sidebar. 4. Click **Create Training Cluster**. The page is headed **Training Clusters** — *"Create and manage your training clusters for distributed machine learning workloads."* :::warning Billing starts at creation, not at first job The form says so plainly: **"Charges apply from the moment the cluster is created."** An idle cluster costs the same as a busy one. Do not create a cluster the day before you need it. ::: --- ## Cluster Name | | | |--|--| | Required | Yes | | Default | A generated name such as `training-cluster-260910092046` | | Rules | Lowercase letters, digits and hyphens; must start with a letter; must not end with a hyphen; at most 50 characters | | Errors | *Cluster name is required* · *Name is too long, max 50 characters allowed.* · *Invalid name format* | The name is how you and your team recognise the cluster in the list, and it appears in the cluster's internal resource names. Pick something meaningful — `llama-pretrain-sep`, not the default. --- ## Image Two dropdowns under the label **Select image and version**, plus a refresh icon. | Field | What it selects | |-------|-----------------| | **Image** | The image family, e.g. **Ubuntu Slurm** or **NeMo Framework** | | **Version** | A specific build, e.g. `v2-slurm25.11-ubuntu24.04` | The version encodes the Slurm version and base OS, and is shown on the cluster's **Details** tab afterwards. Each image comes with GPU drivers, CUDA, NCCL, the Slurm runtime, and the [Enroot and Pyxis](/docs/tir/SlurmCluster/containers/) container stack pre-installed. | Family | Use it for | |--------|------------| | **Ubuntu Slurm** | The general-purpose choice. A clean Slurm environment; bring your framework in a container | | **NeMo Framework** | NVIDIA NeMo / Megatron-style LLM training, with the framework stack already in the cluster image | :::danger This is not the container image for your jobs The **Image** field selects the **cluster's own** software — Slurm, the drivers, the OS. It does not constrain what your jobs run in. To run a specific PyTorch, JAX or custom image, pass `--container-image` to `srun`/`sbatch`; see [Containers with Enroot and Pyxis](/docs/tir/SlurmCluster/containers/). If you have been told to create a new cluster to get a different framework version, you almost certainly do not need to. ::: :::tip Prefer Ubuntu Slurm plus a container A container per job means two teams can use different CUDA and framework versions on one cluster, and upgrading your framework never means touching the cluster. Reach for **NeMo Framework** only when you specifically want that stack baked into the nodes. ::: The version can be changed later with [**Update Image**](/docs/tir/SlurmCluster/manage/actions#update-image), which does a rolling restart. --- ## Plan Configuration Tabs select where the capacity comes from: **GPU** (on-demand inventory) or **Private Cluster** (capacity you have already reserved). Some accounts also see an **IndiaAI** tab for IndiaAI-allocated capacity. ### GPU tab Plan cards show the GPU model and per-node resources, for example: ``` NVIDIA H200 SXM 8 GPU, 240 CPU, 3000 GB RAM, 141GB GPU Memory ₹3,484.00 /hr ``` A greyed-out card with **Inventory not available** means no free nodes of that type right now. | Field | Notes | |-------|-------| | **Configuration** | The per-node shape, e.g. `8 GPU - 240 vCPUs - 3000GB RAM` | | **Workers** | Number of worker nodes, set with the − / + stepper. Minimum 1 | Below that, **Pricing** offers **On-Demand** (*"Flexible pay-as-you-go hourly pricing"*) and, where available, savings plans — **1 Month** (Save 10%), **3 Month** (Save 14%) and **6 Month** (Save 20%). :::info The price shown is per node The rate on the card is for one node. Two H200 nodes cost twice the card price. The savings plans commit you for their term. ::: ### Private Cluster tab **Select Private Cluster** — *"Choose a pre-existing private cluster. Click here to create new one."* Each card shows the cluster name, GPU model, and chips for `Total Nodes` and `Available Nodes`. Then set **Workers** from the available nodes. | Error | Meaning | |-------|---------| | *The requested node count exceeds current availability. Please reduce the node count.* | You asked for more nodes than the private cluster has free | | *Node count cannot be less than 1* | At least one worker is required | :::tip Reserve capacity if you create clusters often A [Private Cluster](/docs/myaccount/private_cluster/) holds nodes for your account. Creation always succeeds, provisioning is faster, and there are no separate hourly charges for the Slurm cluster on top. This is the right pattern if you tear down and recreate clusters regularly. ::: :::warning Sizing decisions you cannot undo cheaply - **Committed and savings plans cannot be scaled.** Size the cluster before you commit. - **Scaling down restarts the remaining workers**, so it is not a free adjustment. If you are unsure, start with fewer nodes on On-Demand and [scale up](/docs/tir/SlurmCluster/manage/actions#scale-cluster) once the queue tells you what you need. ::: Missing plan selection raises *Please select a plan configuration*. --- ## Access ### SSH Keys | | | |--|--| | Required | **Yes** | | Helper | *"Choose a pre-existing SSH key for seamless access by uploading the key file. Click here to create a new one."* | | Error | *Please select at least one SSH key* | Select one or more keys; **Click here** creates a new one inline. These keys give `root` access to the login node. :::tip Add a second key now A cluster with one key is one lost laptop away from being unreachable. Add a colleague's key or a backup key at create time. Keys can be changed later with [**Update SSH Keys**](/docs/tir/SlurmCluster/manage/actions#update-ssh-keys), and per-person access is better handled with [Login User Management](/docs/tir/SlurmCluster/connect/login-user-management). ::: ### Parallel File System | | | |--|--| | Required | **Yes** — and at least one storage volume is required overall | | Helper | *"Select a parallel file system and specify the path where it will be mounted on the cluster workers. Click here to create new one."* | Select a PFS volume and give it a **mount path**, e.g. `/pfs`. Only volumes in a healthy state are offered. | Mount path rule | Detail | |-----------------|--------| | Must start with `/` | `pfs` is invalid, `/pfs` is valid | | Cannot be `/` | The root itself is not allowed | | No `//`, no trailing `/` | `/a//b` and `/pfs/` are invalid | | No spaces | The space key is blocked in the field | | Must be unique | Errors with *All mount paths must be unique* across your PFS, SFS and dataset paths | | Errors | *Mount path is required* · *Invalid path format* | :::info This path is where your work lives The mount appears at the same path on the login node **and every worker**, so it is where datasets, checkpoints, logs, job scripts and container squash files belong. Anything outside a mounted volume is scratch. Choose a short, obvious path and keep it consistent across clusters. ::: --- ## Advanced Settings Expand **Advanced Settings** for the rest. One field here is required. ### Security Group | | | |--|--| | Required | **Yes** | | Helper | *"Choose a pre-existing Security Group to manage network access. Click here to create a new one."* | | Error | *Please select a Security Group* | Once you select SSH keys, the form auto-selects a security group that has SSH open, and groups **without port 22** are disabled with a `Port 22 disabled` chip. If SSH keys are selected but no suitable group is, an inline note reads *"Security Group with port 22 is required when SSH Key is added"*. :::warning A group without port 22 makes the cluster unreachable Provisioning still succeeds — you simply cannot SSH in. Leave the auto-selected SSH group in place unless you have a specific reason, and tighten the source range rather than removing the rule. Groups can be changed later from [Network & Security](/docs/tir/SlurmCluster/manage/network-security#security-groups). ::: ### Lifecycle Script (Optional) > Lifecycle configuration script that you want to run on each instance in the instance group after > cluster creation Click **Add Script** to open a code editor, or **Upload Script** to load a `.sh` or `.py` file. **✕ Remove Script** clears it. Uploading any other file type raises *Please upload a valid .sh or .py file*. Use it for one-time node setup — installing a package, writing a config file, mounting something extra. :::info Lifecycle Script runs once; a prolog runs per job The lifecycle script runs on each node **after the cluster is created**, and it is a create-time field you cannot change later. For code that must run around **every job**, use [Worker Prolog](/docs/tir/SlurmCluster/slurm-configuration/prolog-epilog) instead — that is editable at any time. The two coexist; a prolog never replaces the lifecycle script. ::: :::danger No secrets in the lifecycle script It is stored with the cluster configuration and carried into clones. Read credentials from a file on your mounted storage at runtime instead. ::: ### Slurm Configuration (Optional) > Extra `slurm.conf` settings and prolog/epilog scripts for the cluster. Partitions are added from the > cluster's Overview tab once it is running, since a partition selects specific nodes. Click **Configure Slurm** to open the dialog with two tabs — **slurm.conf** and **Prolog & Epilog** — and confirm with **Save Configuration**. The summary next to the button reads **Not configured**, or a list such as `extra slurm.conf, 1 script`. **Clear Configuration** discards it. There is no **Partitions** tab here. Add partitions once the cluster is Running, from the [partition table](/docs/tir/SlurmCluster/slurm-configuration/partitions#add-a-partition). :::tip You can skip this and come back Everything in this dialog is editable on a Running cluster. Setting it at create time only saves you one reconfigure later. See [Slurm Configuration](/docs/tir/SlurmCluster/slurm-configuration/). ::: ### Shared File System (Optional) Select an SFS volume and a mount path (placeholder `e.g. /shared`). Read-write, and mounted on every node. The same path rules as PFS apply. :::tip Add an SFS volume now if a team will share this cluster [Login User Management](/docs/tir/SlurmCluster/connect/login-user-management) — per-user SSH logins and private home directories — provisions those home directories on an **SFS** volume, and its storage picker lists SFS only. Attaching one here costs nothing; attaching one later means a node restart that ends running jobs. ::: ### Dataset Storage (Optional) Select a dataset and a mount path (placeholder `e.g. /data`). :::warning Datasets are mounted read-only Perfect for input data, useless for checkpoints and logs. Those need PFS, SFS or Weka. A job that tries to write to a dataset mount fails with a permission error. ::: ### Weka Storage In regions where Weka is available it replaces PFS as the required volume, with the same shape and path rules (placeholder `e.g. /weka`). --- ## Create the Cluster Click **Create Training Cluster**. The button stays disabled while any field is invalid. Checks that run on submit: | Check | Message | |-------|---------| | A plan is selected | *Please select a plan configuration* | | At least one volume | *Please select at least one storage (PFS, SFS, or Dataset)* — or *(Weka, SFS or Datasets)* in Weka regions | | Mount paths are unique | *All mount paths must be unique* | | Partitions are not set at create | *Partitions can only be configured after the cluster is running, because a partition selects specific nodes and those do not exist yet* | | Enough credit for the plan and node count | A credit message from the billing service | ### After you submit 1. The cluster appears in the list at status **Creating**. The list refreshes itself every few seconds while anything is Creating. 2. Provisioning normally takes a few minutes: nodes are reserved, the Slurm cluster is installed, and the login endpoint is published. 3. When every component is ready, the status becomes **Running**. :::info If it stays in Creating Try [**Reconfigure Cluster**](/docs/tir/SlurmCluster/manage/actions#reconfigure-cluster) — it is available on a Creating cluster and re-applies the intended state. If the cluster ends up **Failed**, it cannot be repaired: use **Clone Cluster** to get the same configuration in a fresh form, then delete the failed record. A Failed cluster is not billed for compute. ::: ### Next 1. [Submit your first job](/docs/tir/SlurmCluster/getting-started/first-job) — connect and run something. 2. [Add partitions](/docs/tir/SlurmCluster/slurm-configuration/partitions) — now that nodes exist. 3. [Cache a container image](/docs/tir/SlurmCluster/containers/image-cache) — before your first real run. 4. [Enable Login User Management](/docs/tir/SlurmCluster/connect/login-user-management) — if a team will share the cluster. --- ## Create via the API Every field above maps to the create endpoint, so cluster provisioning can be scripted. See the [Training Cluster API reference](/api/tir/clusters/training-cluster/post-distributed-jobs-v2-cluster/) for the payload, including the optional `slurm_config` object. --- ## Related Resources - [How a Slurm Cluster works](/docs/tir/SlurmCluster/getting-started/slurm-cluster-concepts) - [Submit your first job](/docs/tir/SlurmCluster/getting-started/first-job) - [Cluster actions](/docs/tir/SlurmCluster/manage/actions) - [Storage and volumes](/docs/tir/SlurmCluster/manage/storage) - [Billing](/docs/tir/SlurmCluster/billing)