---
title: "Get started with TIR Slurm Cluster"
description: "Create a TIR Slurm Cluster, connect over SSH, run a GPU job and a container job, follow them in the console, and stop the charges."
content_type: "get-started"
product: "tir-slurm-cluster"
platform: "TIR"
url: "/tir/tir-slurm-cluster/get-started/"
---

# Get started with TIR Slurm Cluster

<!-- cspell:ignore nvcr sqsh RDMA -->

Create a one-node Slurm Cluster, connect to it over SSH, run a batch job on a GPU, then run the same
job inside a container. You need:

- A TIR project with enough credit for one GPU node. The cluster is billed by the hour until you terminate it.
- An SSH key in TIR. To add one, see [Add an SSH key](/tir/ssh-keys/how-to/add-an-ssh-key/).
- A [Shared File System](/tir/tir-sfs/) or a [Parallel File System](/tir/tir-pfs/) in the project, to
  hold your scripts and job output. To make one, see [Create an
  SFS](/tir/tir-sfs/how-to/create-an-sfs/).

## Create a cluster

1. In TIR, open your project. In the side menu, under **PRODUCTS**, select **Training Cluster**, then
   select **Create Training Cluster**. The portal calls this product a Training Cluster.
2. Keep the generated **Cluster Name**, or type your own in lowercase letters, digits, and hyphens.
3. Under **Image**, choose an image family, such as **Ubuntu Slurm**, and then a version. This is the
   cluster's own Slurm and driver software; each job chooses its own container later.
4. Under **Plan Configuration**, stay on the **GPU** tab and choose a plan. Each plan is one node with
   8 GPUs. Leave **Workers** at 1 and **Pricing** on **On-Demand**, the hourly plan.
5. Under **SSH Keys**, select your key.
6. Choose your Shared File System or Parallel File System and set its mount path to `/shared`. The
   commands on this page use that path.
7. Expand **Advanced Settings**. Once you have chosen a key, TIR picks your SSH security group, which
   opens port 22. Keep it.
8. Select **Create Training Cluster**.

The cluster appears in the list as **Creating**. Billing starts once the create succeeds, counted from
the moment you submitted the form. Within a few minutes the status moves to **Initializing Worker
Nodes** and then **Running**. Wait for **Running** before you submit jobs.

For every option, see [Create a Slurm
Cluster](/tir/tir-slurm-cluster/how-to/create-a-slurm-cluster/).

## Connect to the login node

Select the cluster's name, then select **Connect**, and copy the SSH command. Run it in your
terminal:

```bash
ssh root@<cluster-ip>
```

Replace `<cluster-ip>` with the cluster's public IP address. The same command is on the **Details**
tab under **Connection Details**.

You land on the login node. You submit jobs from here, and your storage is at `/shared`. The GPUs for
your work are on the worker nodes, and you send work to them with `sbatch` or `srun`. If SSH is
refused or times out, see [Troubleshooting connecting to TIR Slurm
Cluster](/tir/tir-slurm-cluster/troubleshooting/connecting/).

## Look around

```bash
sinfo
```

`sinfo` lists partitions and node states: `idle` is free, `alloc` is busy, `mix` is partly busy, and
`drain` or `down` is out of service. You see the `all` partition covering your node.

```bash
sinfo -N -o "%N %P %t %C %G"
squeue
df -h /shared
```

The first command adds each node's CPU use and its GPUs, for example `gpu:h100:8`. `squeue` shows the
job queue, which is empty on a new cluster. `df` confirms that your storage is mounted.

## Run a job on a GPU

Make a working directory on shared storage and write a job script:

```bash
mkdir -p /shared/first-job/logs
cd /shared/first-job
```

```bash
cat > hello.sh <<'EOF'
#!/bin/bash
#SBATCH --job-name=hello
#SBATCH --partition=all
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --gres=gpu:1
#SBATCH --time=00:05:00
#SBATCH --output=/shared/first-job/logs/%x-%j.out
#SBATCH --error=/shared/first-job/logs/%x-%j.err

echo "Job $SLURM_JOB_ID running on $(hostname)"
echo "Allocated GPUs: $CUDA_VISIBLE_DEVICES"
nvidia-smi
EOF
```

| Directive             | What it does                                                        |
| --------------------- | ------------------------------------------------------------------- |
| `--job-name`          | Sets the name shown in `squeue` and on the **Jobs** tab.            |
| `--partition=all`     | Uses the default partition, which covers every node.                |
| `--nodes`, `--ntasks` | Ask for one node and one task.                                      |
| `--gres=gpu:1`        | Asks for one GPU. A job gets GPUs only when it asks for them.       |
| `--time`              | Ends the job after this long, so a hung job releases the GPU then.  |
| `--output`, `--error` | Set where output goes. `%x` is the job name and `%j` is the job ID. |

Submit it and watch the queue:

```bash
sbatch hello.sh
squeue -u $USER
```

`sbatch` prints `Submitted batch job <jobid>`. When the job has left the queue, read its output,
with that number in place of `<jobid>`:

```bash
cat /shared/first-job/logs/hello-<jobid>.out
```

`nvidia-smi` lists one GPU, the one Slurm allocated to the job. Output written to `/shared` stays
there, while a file on a node's own disk is erased when that node is recreated.

## Run the same job in a container

A container job takes its software from an image you name, so a different PyTorch or CUDA version
runs on the cluster as it is:

```bash
cat > hello-container.sh <<'EOF'
#!/bin/bash
#SBATCH --job-name=hello-container
#SBATCH --partition=all
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --gres=gpu:1
#SBATCH --time=00:15:00
#SBATCH --output=/shared/first-job/logs/%x-%j.out
#SBATCH --error=/shared/first-job/logs/%x-%j.err

srun --container-image=nvcr.io/nvidia/pytorch:25.09-py3 \
     --container-mounts=/shared:/shared \
     --container-workdir=/shared/first-job \
     bash -lc '
       echo "Inside the container on $(hostname)"
       nvidia-smi
       python -c "import torch; print(torch.__version__, torch.cuda.is_available(), torch.cuda.device_count())"
     '
EOF
```

| Flag                                 | What it does                                                                            |
| ------------------------------------ | --------------------------------------------------------------------------------------- |
| `--container-image`                  | Names the image to run: a registry reference or the path to a squash file.              |
| `--container-mounts=/shared:/shared` | Makes your storage visible inside the container, which sees only the paths passed here. |
| `--container-workdir`                | Sets the working directory inside the container.                                        |

Submit it, and read the output once the job has left `squeue -u $USER`:

```bash
sbatch hello-container.sh
cat /shared/first-job/logs/hello-container-<jobid>.out
```

The first run spends a couple of minutes importing the image. The output shows the container's own
PyTorch version and `True` for CUDA. To import the image once for every job, save it
on shared storage. See [Cache container images](/tir/tir-slurm-cluster/how-to/cache-container-images/).

## Follow the job in the console

You can also follow jobs from the console. Open the cluster in TIR:

| Tab                  | What it shows                                                   |
| -------------------- | --------------------------------------------------------------- |
| **Jobs**             | The live queue with state, nodes, GPUs, run time, and priority. |
| **Cluster Overview** | GPU allocation, idle nodes, node health, and partitions.        |
| **Monitoring**       | GPU utilization, memory, temperature, and power over time.      |
| **Nodes**            | Per-GPU metrics and hardware errors.                            |
| **Logs**             | Login node and worker logs.                                     |

To confirm that a job uses its GPUs, open **Monitoring** while it runs and read **GPU Utilisation**.
A flat line near zero means the GPUs are sitting idle.

## Stop the charges

A cluster is billed for its nodes every hour, busy or idle. When you have finished, select
**Actions**, then **Terminate Cluster**, and confirm. Terminating ends every job on the cluster and
stops its compute charges. Your Shared File System or Parallel File System is unmounted and kept,
with your scripts and logs in it, and it keeps billing until you delete it.

The terminated cluster stays in the list. To pick up where you left off, select **Actions**, then
**Clone Cluster**, which opens the create form filled in from it. See [Terminate or delete a
cluster](/tir/tir-slurm-cluster/how-to/terminate-or-delete-a-cluster/).

## What to do next

- [Run multi-node training](/tir/tir-slurm-cluster/how-to/run-multi-node-training/) to train across
  several nodes over the RDMA fabric.
- [Run containers in jobs](/tir/tir-slurm-cluster/how-to/run-containers-in-jobs/) for interactive
  container shells, job arrays, and every container flag.
- [Manage partitions](/tir/tir-slurm-cluster/how-to/manage-partitions/) to give short jobs their own
  queue.
- [Manage login users](/tir/tir-slurm-cluster/how-to/manage-login-users/) to give your team
  individual logins.

If a job stays pending or fails, see [Troubleshooting jobs on TIR Slurm
Cluster](/tir/tir-slurm-cluster/troubleshooting/jobs/). For every `sbatch` option, see the [Slurm
`sbatch` reference](https://slurm.schedmd.com/sbatch.html).