Get started with TIR Slurm Cluster

View as Markdown

Create a one-node Slurm Cluster, connect to it over SSH, run a batch job on a GPU, then run the same job inside a container. You need:

A TIR project with enough credit for one GPU node. The cluster is billed by the hour until you terminate it.
An SSH key in TIR. To add one, see Add an SSH key.
A Shared File System or a Parallel File System in the project, to hold your scripts and job output. To make one, see Create an SFS.

Create a cluster

In TIR, open your project. In the side menu, under PRODUCTS, select Training Cluster, then select Create Training Cluster. The portal calls this product a Training Cluster.
Keep the generated Cluster Name, or type your own in lowercase letters, digits, and hyphens.
Under Image, choose an image family, such as Ubuntu Slurm, and then a version. This is the cluster’s own Slurm and driver software; each job chooses its own container later.
Under Plan Configuration, stay on the GPU tab and choose a plan. Each plan is one node with 8 GPUs. Leave Workers at 1 and Pricing on On-Demand, the hourly plan.
Under SSH Keys, select your key.
Choose your Shared File System or Parallel File System and set its mount path to /shared. The commands on this page use that path.
Expand Advanced Settings. Once you have chosen a key, TIR picks your SSH security group, which opens port 22. Keep it.
Select Create Training Cluster.

The cluster appears in the list as Creating. Billing starts once the create succeeds, counted from the moment you submitted the form. Within a few minutes the status moves to Initializing Worker Nodes and then Running. Wait for Running before you submit jobs.

For every option, see Create a Slurm Cluster.

Select the cluster’s name, then select Connect, and copy the SSH command. Run it in your terminal:

ssh root@<cluster-ip>

Replace <cluster-ip> with the cluster’s public IP address. The same command is on the Details tab under Connection Details.

You land on the login node. You submit jobs from here, and your storage is at /shared. The GPUs for your work are on the worker nodes, and you send work to them with sbatch or srun. If SSH is refused or times out, see Troubleshooting connecting to TIR Slurm Cluster.

Look around

sinfo

sinfo lists partitions and node states: idle is free, alloc is busy, mix is partly busy, and drain or down is out of service. You see the all partition covering your node.

sinfo -N -o "%N %P %t %C %G"
squeue
df -h /shared

The first command adds each node’s CPU use and its GPUs, for example gpu:h100:8. squeue shows the job queue, which is empty on a new cluster. df confirms that your storage is mounted.

Run a job on a GPU

Make a working directory on shared storage and write a job script:

mkdir -p /shared/first-job/logs
cd /shared/first-job

cat > hello.sh <<'EOF'
#!/bin/bash
#SBATCH --job-name=hello
#SBATCH --partition=all
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --gres=gpu:1
#SBATCH --time=00:05:00
#SBATCH --output=/shared/first-job/logs/%x-%j.out
#SBATCH --error=/shared/first-job/logs/%x-%j.err

echo "Job $SLURM_JOB_ID running on $(hostname)"
echo "Allocated GPUs: $CUDA_VISIBLE_DEVICES"
nvidia-smi
EOF

Directive	What it does
`--job-name`	Sets the name shown in `squeue` and on the Jobs tab.
`--partition=all`	Uses the default partition, which covers every node.
`--nodes`, `--ntasks`	Ask for one node and one task.
`--gres=gpu:1`	Asks for one GPU. A job gets GPUs only when it asks for them.
`--time`	Ends the job after this long, so a hung job releases the GPU then.
`--output`, `--error`	Set where output goes. `%x` is the job name and `%j` is the job ID.

Submit it and watch the queue:

sbatch hello.sh
squeue -u $USER

sbatch prints Submitted batch job <jobid>. When the job has left the queue, read its output, with that number in place of <jobid>:

cat /shared/first-job/logs/hello-<jobid>.out

nvidia-smi lists one GPU, the one Slurm allocated to the job. Output written to /shared stays there, while a file on a node’s own disk is erased when that node is recreated.

Run the same job in a container

A container job takes its software from an image you name, so a different PyTorch or CUDA version runs on the cluster as it is:

cat > hello-container.sh <<'EOF'
#!/bin/bash
#SBATCH --job-name=hello-container
#SBATCH --partition=all
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --gres=gpu:1
#SBATCH --time=00:15:00
#SBATCH --output=/shared/first-job/logs/%x-%j.out
#SBATCH --error=/shared/first-job/logs/%x-%j.err

srun --container-image=nvcr.io/nvidia/pytorch:25.09-py3 \
     --container-mounts=/shared:/shared \
     --container-workdir=/shared/first-job \
     bash -lc '
       echo "Inside the container on $(hostname)"
       nvidia-smi
       python -c "import torch; print(torch.__version__, torch.cuda.is_available(), torch.cuda.device_count())"
     '
EOF

Flag	What it does
`--container-image`	Names the image to run: a registry reference or the path to a squash file.
`--container-mounts=/shared:/shared`	Makes your storage visible inside the container, which sees only the paths passed here.
`--container-workdir`	Sets the working directory inside the container.

Submit it, and read the output once the job has left squeue -u $USER:

sbatch hello-container.sh
cat /shared/first-job/logs/hello-container-<jobid>.out

The first run spends a couple of minutes importing the image. The output shows the container’s own PyTorch version and True for CUDA. To import the image once for every job, save it on shared storage. See Cache container images.

Follow the job in the console

You can also follow jobs from the console. Open the cluster in TIR:

Tab	What it shows
Jobs	The live queue with state, nodes, GPUs, run time, and priority.
Cluster Overview	GPU allocation, idle nodes, node health, and partitions.
Monitoring	GPU utilization, memory, temperature, and power over time.
Nodes	Per-GPU metrics and hardware errors.
Logs	Login node and worker logs.

To confirm that a job uses its GPUs, open Monitoring while it runs and read GPU Utilisation. A flat line near zero means the GPUs are sitting idle.

Stop the charges

A cluster is billed for its nodes every hour, busy or idle. When you have finished, select Actions, then Terminate Cluster, and confirm. Terminating ends every job on the cluster and stops its compute charges. Your Shared File System or Parallel File System is unmounted and kept, with your scripts and logs in it, and it keeps billing until you delete it.

The terminated cluster stays in the list. To pick up where you left off, select Actions, then Clone Cluster, which opens the create form filled in from it. See Terminate or delete a cluster.

What to do next

Run multi-node training to train across several nodes over the RDMA fabric.
Run containers in jobs for interactive container shells, job arrays, and every container flag.
Manage partitions to give short jobs their own queue.
Manage login users to give your team individual logins.

If a job stays pending or fails, see Troubleshooting jobs on TIR Slurm Cluster. For every sbatch option, see the Slurm sbatch reference.