---
title: Submit Your First Job
sidebar_label: First job
---

import { Terminal, Play, Box, Eye, CheckCircle } from 'react-feather';

# Submit Your First Job

Ten minutes, start to finish: connect, look around, run a plain job, then run the same job inside a
container.

<QuickNav items={[
{ href: '#1-connect', label: 'Connect', icon: <Terminal size={13} /> },
{ href: '#2-look-around', label: 'Look around', icon: <Eye size={13} /> },
{ href: '#3-run-a-plain-job', label: 'A plain job', icon: <Play size={13} /> },
{ href: '#4-run-a-container-job', label: 'A container job', icon: <Box size={13} /> },
{ href: '#5-watch-it-from-the-console', label: 'Watch from the console', icon: <Eye size={13} /> },
{ href: '#what-to-do-next', label: 'What next', icon: <CheckCircle size={13} /> },
]} />

---

## 1. Connect

Open the cluster in the TIR console and click **Connect** in the top-right area. The panel is headed
**Use Shell to run freeform commands** and gives you the exact command, with a copy button:

```bash
ssh root@<cluster-ip>
```

The same command, and the cluster's IP, are on the **Details** tab under **Connection Details**.

| Problem | Fix |
|---------|-----|
| `Permission denied (publickey)` | The private key matching a cluster SSH key is not being offered. Try `ssh -i ~/.ssh/<key> root@<ip>` |
| `Connection refused` or a timeout | Port 22 is not open in the security group, or the cluster is not Running yet. See [Network & Security](/docs/tir/SlurmCluster/manage/network-security) |
| It worked yesterday, not today | The IP may have changed. A **Floating** IP is not stable — [convert it to reserved](/docs/tir/SlurmCluster/manage/network-security#public-ip) |

More in [Cannot connect](/docs/tir/SlurmCluster/troubleshoot/connectivity).

:::info You land on the login node
This is the submit host, not a compute node. It has your storage mounts and outbound network, but no
GPUs for your work. Everything real goes through `sbatch` or `srun`.
:::

---

## 2. Look Around

The **Connect** panel offers these as copyable commands; here is what to read in the output.

```bash
sinfo
```

Shows partitions and node states. `idle` means free, `alloc` busy, `mix` partly busy, `drain` or
`down` unavailable. You should see the `all` partition covering your nodes.

```bash
sinfo -N -o "%N %P %t %C %G"
```

Per node: name, partition, state, CPU allocation, and GRES — the GRES column is where you confirm the
GPUs Slurm knows about, e.g. `gpu:h100:8`.

```bash
scontrol show nodes
```

Everything Slurm knows about each node.

```bash
squeue
```

The queue. Empty on a fresh cluster.

```bash
df -h | grep -E '/pfs|/shared|/mnt'
```

Your storage mounts. Confirm your PFS path is there and writable — this is where your work belongs.

### Confirm how your cluster hands out GPUs

Before you write job scripts around it, check that a GPU request is accepted:

```bash
srun --gres=gpu:1 --time=00:05:00 nvidia-smi -L
```

One GPU listed means per-GPU requests work, and `--gres=gpu:N` / `--gpus-per-node=N` are the flags to
use throughout.

:::warning If `--gres` is rejected
`Invalid generic resource (gres) specification` means this cluster's Slurm is not configured to
sub-allocate GPUs. Request whole nodes instead:

```bash
srun --nodes=1 --exclusive --time=00:05:00 nvidia-smi -L
```

That gives your job every GPU on the node. Note it in your job templates, and raise a support
request if you need per-GPU allocation on that cluster — it is a cluster-level configuration, not
something you can change from the console.
:::

:::tip Prove your storage is shared before you trust it
```bash
echo "hello from $(hostname)" > /pfs/hello.txt
srun --nodes=1 bash -c 'cat /pfs/hello.txt'
```
If the worker can read what the login node wrote, the mount is genuinely shared. Do this once per new
cluster — it takes seconds and rules out the most confusing class of multi-node failure.
:::

---

## 3. Run a Plain Job

Create a working directory on shared storage and a job script.

```bash
mkdir -p /pfs/first-job/logs
cd /pfs/first-job
```

```bash
cat > hello.sh <<'EOF'
#!/bin/bash
#SBATCH --job-name=hello
#SBATCH --partition=all
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --gres=gpu:1
#SBATCH --time=00:05:00
#SBATCH --output=/pfs/first-job/logs/%x-%j.out
#SBATCH --error=/pfs/first-job/logs/%x-%j.err

echo "Job $SLURM_JOB_ID running on $(hostname)"
echo "Allocated GPUs: $CUDA_VISIBLE_DEVICES"
nvidia-smi
EOF
```

Submit it:

```bash
sbatch hello.sh
```

You get `Submitted batch job <jobid>`. Watch it:

```bash
squeue -u $USER
```

And read the output when it finishes:

```bash
cat /pfs/first-job/logs/hello-<jobid>.out
```

`nvidia-smi` should list exactly one GPU — the one Slurm allocated, not all eight on the node.

### What each directive did

| Directive | Effect |
|-----------|--------|
| `--job-name` | The name in `squeue` and the [Jobs tab](/docs/tir/SlurmCluster/manage/jobs) |
| `--partition=all` | The default partition covering every node |
| `--nodes` / `--ntasks` | One node, one task |
| `--gres=gpu:1` | One GPU. Without it, the job gets **no** GPU |
| `--time` | Kill the job after this long. Always set it |
| `--output` / `--error` | Where output goes. `%x` is the job name, `%j` the job ID |

:::warning Always set `--time`, and always write output to shared storage
A job without `--time` inherits the partition's Max time, which may be `UNLIMITED` — a hung job then
holds GPUs indefinitely. And output written outside a mounted volume is lost when the pod is
recreated, taking your only record of what went wrong with it.
:::

---

## 4. Run a Container Job

The same job, but the environment comes from a container image instead of the node.

```bash
cat > hello-container.sh <<'EOF'
#!/bin/bash
#SBATCH --job-name=hello-container
#SBATCH --partition=all
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --gres=gpu:1
#SBATCH --time=00:15:00
#SBATCH --output=/pfs/first-job/logs/%x-%j.out
#SBATCH --error=/pfs/first-job/logs/%x-%j.err

srun --container-image=nvcr.io/nvidia/pytorch:25.09-py3 \
     --container-mounts=/pfs:/pfs \
     --container-workdir=/pfs/first-job \
     bash -lc '
       echo "Inside the container on $(hostname)"
       nvidia-smi
       python -c "import torch; print(torch.__version__, torch.cuda.is_available(), torch.cuda.device_count())"
     '
EOF
```

```bash
sbatch hello-container.sh
squeue -u $USER
```

The first run spends a couple of minutes importing the image, so give it longer than the plain job.
When it finishes:

```bash
cat /pfs/first-job/logs/hello-container-<jobid>.out
```

You should see the container's own PyTorch version and `True` for CUDA availability. That is the whole
mechanism — one flag, and the job environment is whatever image you name.

| The flag | Why it is there |
|----------|-----------------|
| `--container-image` | The image to run. A registry reference, or a path to a squash file |
| `--container-mounts=/pfs:/pfs` | Without it the container cannot see your storage |
| `--container-workdir` | Working directory inside the container |

:::tip Cache the image before your real runs
Every job that names a registry reference imports the image again on each node. Import once to shared
storage and point jobs at the file instead:

```bash
mkdir -p /pfs/images
enroot import -o /pfs/images/pytorch-25.09.sqsh docker://nvcr.io/nvidia/pytorch:25.09-py3
```

Then use `--container-image=/pfs/images/pytorch-25.09.sqsh`. See
[Cache and manage images](/docs/tir/SlurmCluster/containers/image-cache).
:::

### An interactive container shell

For poking around rather than batch work:

```bash
srun --partition=all --nodes=1 --gres=gpu:1 --time=00:30:00 --pty \
     --container-image=nvcr.io/nvidia/pytorch:25.09-py3 \
     --container-mounts=/pfs:/pfs \
     bash
```

You are inside the container on a compute node, with a GPU and your data. `exit` releases the
allocation.

:::warning An interactive allocation holds GPUs until you exit
Set `--time`, and exit when you are done. `squeue -u $USER` shows what you are still holding.
:::

---

## 5. Watch It from the Console

You do not need SSH to follow a job.

| Tab | What it tells you |
|-----|-------------------|
| [**Jobs**](/docs/tir/SlurmCluster/manage/jobs) | Live queue with state, nodes, GPUs, run time and priority |
| [**Cluster Overview**](/docs/tir/SlurmCluster/manage/overview-tab) | GPU allocation, idle nodes, node health, partitions |
| [**Monitoring**](/docs/tir/SlurmCluster/manage/monitoring) | GPU utilisation, memory, temperature and power over time |
| [**Nodes**](/docs/tir/SlurmCluster/manage/nodes) | Per-GPU DCGM metrics and XID hardware errors |
| [**Logs**](/docs/tir/SlurmCluster/manage/logs) | Controller and worker logs |

:::tip Confirm your job is actually using the GPUs
A common surprise is a job that runs happily at 0% GPU utilisation — usually a dataloader bottleneck
or a model that never left the CPU. Open **Monitoring** while the job runs and look at **GPU
Utilisation**. Flat near zero means the GPUs are idle and you are paying for nothing.
:::

---

## What to Do Next

| Goal | Page |
|------|------|
| Train across several nodes | [Multi-node container training](/docs/tir/SlurmCluster/containers/multi-node-training) |
| Give short jobs their own queue | [Manage partitions](/docs/tir/SlurmCluster/slurm-configuration/partitions) |
| Run setup or cleanup around every job | [Prolog and epilog scripts](/docs/tir/SlurmCluster/slurm-configuration/prolog-epilog) |
| Tune the scheduler | [Extra slurm.conf settings](/docs/tir/SlurmCluster/slurm-configuration/slurm-conf) |
| Give your team individual logins | [Login User Management](/docs/tir/SlurmCluster/connect/login-user-management) |
| A job will not start | [Troubleshoot jobs](/docs/tir/SlurmCluster/troubleshoot/jobs) |

---

## Command Cheat Sheet

```bash
sbatch job.sh                    # submit a batch job
srun --pty bash                  # interactive shell on a node
squeue                           # the queue
squeue -u $USER                  # your jobs
scancel <jobid>                  # cancel a job
scontrol show job <jobid>        # full job detail
sstat -j <jobid>                 # live usage of a running job
sacct -j <jobid>                 # job history and exit code
sinfo                            # partitions and node states
scontrol show nodes              # node detail
```

---

## Related Resources

- [How a Slurm Cluster works](/docs/tir/SlurmCluster/getting-started/slurm-cluster-concepts)
- [Run containers in jobs](/docs/tir/SlurmCluster/containers/run-containers)
- [Jobs tab](/docs/tir/SlurmCluster/manage/jobs)
- [UI Guide](/docs/tir/SlurmCluster/ui-guide)
- [Slurm `sbatch` reference](https://slurm.schedmd.com/sbatch.html)