Connect to a cluster

Sign in to your cluster’s login node over SSH. That is where you edit scripts, submit jobs with sbatch, and import container images.

Before you begin

• A cluster in Initializing Worker Nodes or Running, the statuses where Connect is available.
• The private key that matches one of the SSH keys you selected when you created the cluster.
• A security group on the cluster with inbound TCP 22 open.

Connect as root

Every cluster starts with root access, using the SSH keys you selected at creation:

ssh root@<cluster-ip>

Replace <cluster-ip> with the cluster’s public IP address. If you use a key other than your default identity, name it:

ssh -i ~/.ssh/<your-key> root@<cluster-ip>

Replace <your-key> with the file name of your private key.

To get the command, open the cluster and select Connect. It also gives these commands to copy:

Command	What it does
ssh root@<ip>	Connects to the login node.
sinfo	Shows nodes and partitions.
scontrol show nodes	Shows the full detail of every node.
sbatch <PATH OF SCRIPT>	Submits a batch script for later execution.
sstat -j <Job ID>	Tracks the resource usage of a running job.
scontrol show job <Job ID>	Tracks one job.

The login node is the place to submit work from. It has your storage mounts and outbound network access, but no GPUs allocated to your own work, so run training through sbatch or srun.

Find the cluster’s address

The cluster’s IP address is in two places:

• The Details tab, under Connection Details, next to the SSH command.
• The Network & Security tab, under Network.

The Connect panel lists one SSH command for each address the cluster has: its public IP, labelled Floating IP, and any reserved IP attached to it.

The IP is labelled Floating or Reserved. Floating means it is assigned from a pool and can change if the cluster’s network resources are recreated. Reserved means it is held for your account and stays the same.

Convert a floating IP to reserved before anyone scripts against it. A script, a CI job, an SSH config entry, or a bookmark that hard-codes a floating IP breaks silently the moment the address changes. On the Network & Security tab, select Convert to reserved. See Manage network and security.

Add an SSH config entry

To connect with a short name, add an entry to your SSH config:

cat >> ~/.ssh/config <<'EOF'
Host slurm-train
  HostName <cluster-ip>
  User root
  IdentityFile ~/.ssh/<your-key>
  ServerAliveInterval 30
EOF

Then connect with ssh slurm-train. The last line sends a keep-alive message every 30 seconds, so idle sessions stay open.

Submit long work with sbatch

A dropped connection takes any process in your foreground shell with it, and restarting the login service drops every open session. Submit long work with sbatch, and it survives both. tmux or screen can hold a shell open across a dropped connection, though the session still ends if the login node itself is recreated.

Keep your files on a mounted volume

Store job scripts, datasets, checkpoints, and container images on a mounted volume rather than in /root or anywhere else on the login node’s own disk. That disk is erased whenever the login node is recreated, for example by a login service restart, an image update, or a reconfigure. Your storage mounts are at the paths you chose and are the same on the login node and on every worker node.

Replace the root keys

To change which keys have root access, select Actions, then Update SSH Keys. It replaces the whole set in one step, so a key you leave out of the new selection loses access. Keep at least two keys on every cluster, so a lost laptop is only an inconvenience: the other key still gets you in. See Update a cluster.

Give your team individual logins

One shared root key works for one person. It works less well once a team shares the cluster:

With a shared root login	With individual logins
Every job is submitted by root, so the Jobs tab shows the same user on every job.	Jobs carry the real user in the Jobs tab.
scancel -u root cancels every job on the cluster.	Each person’s commands affect only their own jobs.
Everyone shares one home directory, and it is erased when the login node is recreated.	Each person gets a private home directory on shared storage.
Removing one person means changing the key for everybody.	You revoke one person, and everyone else keeps their access.

Login User Management does this from the console. You add a member with their own login name and SSH key, and they connect as themselves with ssh <login-name>@<cluster-ip>. See Manage login users.

Keep these commands at hand

These run on the login node. Replace <jobid> with a job’s ID from squeue or sbatch:

sinfo                            # partitions and node states
scontrol show nodes              # full detail on every node
squeue                           # the queue
squeue -u $USER                  # your jobs
sbatch train.sh                  # submit a batch job
srun --partition=all --nodes=1 --gres=gpu:1 --time=00:30:00 --pty bash   # shell on a compute node
scontrol show job <jobid>        # full detail on one job
sstat -j <jobid>                 # live resource use of a running job
sacct -j <jobid>                 # a job's history and exit code
sacct -u $USER -S today          # your jobs today
scancel <jobid>                  # cancel a job
scontrol show config             # the live slurm.conf
enroot import -o /shared/images/<name>.sqsh docker://<image>   # cache a container image

In the last command, <name> is a file name you choose and <image> is the image’s registry reference.

To run your first jobs, see Get started with TIR Slurm Cluster. If the connection fails, see Troubleshooting connecting to TIR Slurm Cluster. The usual causes are a security group with TCP 22 closed, a key missing from the cluster, a changed floating IP, or a stale host key after the login node was recreated.