Troubleshooting connecting to TIR Slurm Cluster

View as Markdown

Fix SSH failures on a TIR Slurm Cluster, for root and for named login users. For other problems with the cluster, see Troubleshooting TIR Slurm Cluster. For job and container problems, see Troubleshooting jobs on TIR Slurm Cluster and Troubleshooting containers on TIR Slurm Cluster. For the steps to connect, see Connect to a cluster.

Start here

Run SSH with verbose output. It usually names the problem and shows which key was offered:

ssh -v root@<cluster-ip>

Replace <cluster-ip> with the address from Connection Details on the cluster’s details page. Then check three things in the console before anything else:

The cluster status is Running, on the Details tab.
The IP you are using is the current one, from Connection Details or Network & Security.
A security group allows inbound TCP 22, under Network & Security, then Security Groups.

The login node’s own disk is wiped whenever the login node is recreated, for example by Restart Login Service, an image update, or Reconfigure Cluster. A session that drops also takes any process in its foreground shell with it, and a login service restart drops every open session. Store job scripts, datasets, checkpoints, and container images on a mounted volume, and submit long work with sbatch. See Connect to a cluster.

SSH is refused or times out

The cluster looks healthy but you cannot log in. Match the cause:

Cause	How to confirm	Fix
Port 22 is closed	Network & Security, then Security Groups. Expand the group and read Inbound Rules.	Attach a group with an SSH rule for TCP 22.
The cluster is not Running	The status on the Details tab	Wait, or select Reconfigure Cluster if it is stuck in Creating.
The IP changed	Compare with Connection Details	Use the current IP, then convert it to reserved.
The login service is restarting	You ran Restart Login Service a moment ago, or an image update started	Wait a minute and try again.
Your own network blocks outbound port 22	Try from a different network	Use your VPN, or ask your network team.

A security group missing an inbound TCP 22 rule is the most common cause of “the cluster is broken”. The cluster itself is fine, and the whole fix is to attach a group that allows port 22 from Network & Security.

A floating IP can change when the cluster’s network resources are recreated, which breaks every SSH config entry and script that uses it. Select Convert to reserved once, and the IP stays stable. See Manage network and security.

Permission denied (publickey)

The cluster is reachable, but it rejected your key. Work through these checks in order.

Is the right key offered? Look for the offered key, then name the key explicitly:

ssh -v root@<cluster-ip> 2>&1 | grep -i 'offering\|Offering public key'
ssh -i ~/.ssh/id_ed25519 root@<cluster-ip>

Is that key on the cluster? On the Details tab, Connection Details lists the keys by label. If yours is missing, select Actions, then Update SSH Keys. Update SSH Keys replaces the whole set, so a key left out loses access. Keep at least two keys on a cluster, so a second key still works when one machine is lost. See Update a cluster.
Are your local key permissions right? SSH refuses a private key that others can read:
```
chmod 700 ~/.ssh
chmod 600 ~/.ssh/id_ed25519
```
Are you the right user? The cluster’s own keys log in as root. A Login User Management member logs in with their own login name:
```
ssh root@<cluster-ip>
ssh <login-name>@<cluster-ip>
```
Are too many keys offered? An agent with many keys can reach the server’s attempt limit before the right one. Offer only one:
```
ssh -o IdentitiesOnly=yes -i ~/.ssh/<the-right-key> root@<cluster-ip>
```

To use the right key every time, add an entry to ~/.ssh/config, replacing <cluster-ip>:

Host slurm-train
  HostName <cluster-ip>
  User root
  IdentityFile ~/.ssh/id_ed25519
  IdentitiesOnly yes
  ServerAliveInterval 30

Then connect with ssh slurm-train.

WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED

This warning is expected after the login node is recreated by a restart, an image update, or a reconfigure. Remove the stale entry, then reconnect and accept the new key:

ssh-keygen -R <cluster-ip>

Clear the entry only when you know why it changed. The same message is what a real interception looks like. If the cluster has run uninterrupted and kept the same IP, investigate instead.

root works but a Login User Management member does not. Access is granted per group rather than per member, so a new member needs a group and that group needs cluster access.

Cause	Check	Fix
The member’s SSH key is missing	Members, then the SSH keys count	Open the member’s actions menu and update the SSH keys.
Their group lacks cluster access	The Groups tab	Allow access for the group.
They have yet to join a group	Members, filtered by group	Add the member to a group that has access.
Login User Management is still off	The tab shows the empty state to set it up	Turn it on. See Manage login users.
It was turned on moments ago	The tab shows an in-progress state	Wait for it to finish.
The login name is wrong	The Login Name column	Use the exact name, in lowercase.

ssh -v <login-name>@<cluster-ip>

For a member, Permission denied (publickey) usually means one of the first three rows. Members log in with keys only, so someone who asks for a password needs their public key attached to their member record instead.

My SSH session keeps dropping

Cause	Fix
An idle timeout on a network device	Add ServerAliveInterval 30 to `~/.ssh/config`.
The login service restarted	Expected. It recreates the login node and drops every session.
An image update or reconfigure	The same.
The login node ran out of resources	Someone is running heavy work on the login node instead of submitting it.

Run training through Slurm rather than in a foreground shell on the login node. The login node only submits work, and the GPUs allocated to your jobs are on the workers. A dropped connection or a login service restart ends a process running there. A job submitted with sbatch survives disconnects, restarts, and image updates. tmux or screen keeps a shell open across a dropped connection, though the session still ends if the login node is recreated.

Check that you are in

Once you are connected, these commands confirm that the cluster is working:

sinfo
df -h | grep -E '/pfs|/shared'
echo test > /pfs/write-check && rm /pfs/write-check && echo writable
squeue
srun --help | grep -i container

They show that the cluster is scheduling, your storage is mounted and writable, the queue answers, and container support is present.

If the problem remains, contact support with the cluster name, the project, and the output of ssh -v. See Troubleshooting TIR Slurm Cluster.