Submit Your First Job
Ten minutes, start to finish: connect, look around, run a plain job, then run the same job inside a container.
1. Connect
Open the cluster in the TIR console and click Connect in the top-right area. The panel is headed Use Shell to run freeform commands and gives you the exact command, with a copy button:
ssh root@<cluster-ip>
The same command, and the cluster's IP, are on the Details tab under Connection Details.
| Problem | Fix |
|---|---|
Permission denied (publickey) | The private key matching a cluster SSH key is not being offered. Try ssh -i ~/.ssh/<key> root@<ip> |
Connection refused or a timeout | Port 22 is not open in the security group, or the cluster is not Running yet. See Network & Security |
| It worked yesterday, not today | The IP may have changed. A Floating IP is not stable — convert it to reserved |
More in Cannot connect.
This is the submit host, not a compute node. It has your storage mounts and outbound network, but no
GPUs for your work. Everything real goes through sbatch or srun.
2. Look Around
The Connect panel offers these as copyable commands; here is what to read in the output.
sinfo
Shows partitions and node states. idle means free, alloc busy, mix partly busy, drain or
down unavailable. You should see the all partition covering your nodes.
sinfo -N -o "%N %P %t %C %G"
Per node: name, partition, state, CPU allocation, and GRES — the GRES column is where you confirm the
GPUs Slurm knows about, e.g. gpu:h100:8.
scontrol show nodes
Everything Slurm knows about each node.
squeue
The queue. Empty on a fresh cluster.
df -h | grep -E '/pfs|/shared|/mnt'
Your storage mounts. Confirm your PFS path is there and writable — this is where your work belongs.
Confirm how your cluster hands out GPUs
Before you write job scripts around it, check that a GPU request is accepted:
srun --gres=gpu:1 --time=00:05:00 nvidia-smi -L
One GPU listed means per-GPU requests work, and --gres=gpu:N / --gpus-per-node=N are the flags to
use throughout.
--gres is rejectedInvalid generic resource (gres) specification means this cluster's Slurm is not configured to
sub-allocate GPUs. Request whole nodes instead:
srun --nodes=1 --exclusive --time=00:05:00 nvidia-smi -L
That gives your job every GPU on the node. Note it in your job templates, and raise a support request if you need per-GPU allocation on that cluster — it is a cluster-level configuration, not something you can change from the console.
echo "hello from $(hostname)" > /pfs/hello.txt
srun --nodes=1 bash -c 'cat /pfs/hello.txt'
If the worker can read what the login node wrote, the mount is genuinely shared. Do this once per new cluster — it takes seconds and rules out the most confusing class of multi-node failure.
3. Run a Plain Job
Create a working directory on shared storage and a job script.
mkdir -p /pfs/first-job/logs
cd /pfs/first-job
cat > hello.sh <<'EOF'
#!/bin/bash
#SBATCH --job-name=hello
#SBATCH --partition=all
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --gres=gpu:1
#SBATCH --time=00:05:00
#SBATCH --output=/pfs/first-job/logs/%x-%j.out
#SBATCH --error=/pfs/first-job/logs/%x-%j.err
echo "Job $SLURM_JOB_ID running on $(hostname)"
echo "Allocated GPUs: $CUDA_VISIBLE_DEVICES"
nvidia-smi
EOF
Submit it:
sbatch hello.sh
You get Submitted batch job <jobid>. Watch it:
squeue -u $USER
And read the output when it finishes:
cat /pfs/first-job/logs/hello-<jobid>.out
nvidia-smi should list exactly one GPU — the one Slurm allocated, not all eight on the node.
What each directive did
| Directive | Effect |
|---|---|
--job-name | The name in squeue and the Jobs tab |
--partition=all | The default partition covering every node |
--nodes / --ntasks | One node, one task |
--gres=gpu:1 | One GPU. Without it, the job gets no GPU |
--time | Kill the job after this long. Always set it |
--output / --error | Where output goes. %x is the job name, %j the job ID |
--time, and always write output to shared storageA job without --time inherits the partition's Max time, which may be UNLIMITED — a hung job then
holds GPUs indefinitely. And output written outside a mounted volume is lost when the pod is
recreated, taking your only record of what went wrong with it.
4. Run a Container Job
The same job, but the environment comes from a container image instead of the node.
cat > hello-container.sh <<'EOF'
#!/bin/bash
#SBATCH --job-name=hello-container
#SBATCH --partition=all
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --gres=gpu:1
#SBATCH --time=00:15:00
#SBATCH --output=/pfs/first-job/logs/%x-%j.out
#SBATCH --error=/pfs/first-job/logs/%x-%j.err
srun --container-image=nvcr.io/nvidia/pytorch:25.09-py3 \
--container-mounts=/pfs:/pfs \
--container-workdir=/pfs/first-job \
bash -lc '
echo "Inside the container on $(hostname)"
nvidia-smi
python -c "import torch; print(torch.__version__, torch.cuda.is_available(), torch.cuda.device_count())"
'
EOF
sbatch hello-container.sh
squeue -u $USER
The first run spends a couple of minutes importing the image, so give it longer than the plain job. When it finishes:
cat /pfs/first-job/logs/hello-container-<jobid>.out
You should see the container's own PyTorch version and True for CUDA availability. That is the whole
mechanism — one flag, and the job environment is whatever image you name.
| The flag | Why it is there |
|---|---|
--container-image | The image to run. A registry reference, or a path to a squash file |
--container-mounts=/pfs:/pfs | Without it the container cannot see your storage |
--container-workdir | Working directory inside the container |
Every job that names a registry reference imports the image again on each node. Import once to shared storage and point jobs at the file instead:
mkdir -p /pfs/images
enroot import -o /pfs/images/pytorch-25.09.sqsh docker://nvcr.io/nvidia/pytorch:25.09-py3
Then use --container-image=/pfs/images/pytorch-25.09.sqsh. See
Cache and manage images.
An interactive container shell
For poking around rather than batch work:
srun --partition=all --nodes=1 --gres=gpu:1 --time=00:30:00 --pty \
--container-image=nvcr.io/nvidia/pytorch:25.09-py3 \
--container-mounts=/pfs:/pfs \
bash
You are inside the container on a compute node, with a GPU and your data. exit releases the
allocation.
Set --time, and exit when you are done. squeue -u $USER shows what you are still holding.
5. Watch It from the Console
You do not need SSH to follow a job.
| Tab | What it tells you |
|---|---|
| Jobs | Live queue with state, nodes, GPUs, run time and priority |
| Cluster Overview | GPU allocation, idle nodes, node health, partitions |
| Monitoring | GPU utilisation, memory, temperature and power over time |
| Nodes | Per-GPU DCGM metrics and XID hardware errors |
| Logs | Controller and worker logs |
A common surprise is a job that runs happily at 0% GPU utilisation — usually a dataloader bottleneck or a model that never left the CPU. Open Monitoring while the job runs and look at GPU Utilisation. Flat near zero means the GPUs are idle and you are paying for nothing.
What to Do Next
| Goal | Page |
|---|---|
| Train across several nodes | Multi-node container training |
| Give short jobs their own queue | Manage partitions |
| Run setup or cleanup around every job | Prolog and epilog scripts |
| Tune the scheduler | Extra slurm.conf settings |
| Give your team individual logins | Login User Management |
| A job will not start | Troubleshoot jobs |
Command Cheat Sheet
sbatch job.sh # submit a batch job
srun --pty bash # interactive shell on a node
squeue # the queue
squeue -u $USER # your jobs
scancel <jobid> # cancel a job
scontrol show job <jobid> # full job detail
sstat -j <jobid> # live usage of a running job
sacct -j <jobid> # job history and exit code
sinfo # partitions and node states
scontrol show nodes # node detail