--- title: "Troubleshooting connecting to TIR Slurm Cluster" description: "Fix SSH problems with a TIR Slurm Cluster, for root and for named login users, from timeouts and refused connections to rejected keys and dropped sessions." content_type: "troubleshooting" product: "tir-slurm-cluster" platform: "TIR" url: "/tir/tir-slurm-cluster/troubleshooting/connecting/" relationships: {"troubleshooting-for":["/tir/tir-slurm-cluster/how-to/connect-to-a-cluster/","/tir/tir-slurm-cluster/how-to/manage-login-users/","/tir/tir-slurm-cluster/how-to/manage-network-and-security/"]} --- # Troubleshooting connecting to TIR Slurm Cluster Fix SSH failures on a TIR Slurm Cluster, for `root` and for named login users. For other problems with the cluster, see [Troubleshooting TIR Slurm Cluster](/tir/tir-slurm-cluster/troubleshooting/tir-slurm-cluster/). For job and container problems, see [Troubleshooting jobs on TIR Slurm Cluster](/tir/tir-slurm-cluster/troubleshooting/jobs/) and [Troubleshooting containers on TIR Slurm Cluster](/tir/tir-slurm-cluster/troubleshooting/containers/). For the steps to connect, see [Connect to a cluster](/tir/tir-slurm-cluster/how-to/connect-to-a-cluster/). ## Start here Run SSH with verbose output. It usually names the problem and shows which key was offered: ```bash ssh -v root@ ``` Replace `` with the address from **Connection Details** on the cluster's details page. Then check three things in the console before anything else: - The cluster status is **Running**, on the **Details** tab. - The IP you are using is the current one, from **Connection Details** or **Network & Security**. - A security group allows inbound TCP 22, under **Network & Security**, then **Security Groups**. ## My work on the login node is gone after I reconnected The login node's own disk is wiped whenever the login node is recreated, for example by **Restart Login Service**, an image update, or **Reconfigure Cluster**. A session that drops also takes any process in its foreground shell with it, and a login service restart drops every open session. Store job scripts, datasets, checkpoints, and container images on a mounted volume, and submit long work with `sbatch`. See [Connect to a cluster](/tir/tir-slurm-cluster/how-to/connect-to-a-cluster/). ## SSH is refused or times out The cluster looks healthy but you cannot log in. Match the cause: | Cause | How to confirm | Fix | | ---------------------------------------- | ---------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------- | | Port 22 is closed | **Network & Security**, then **Security Groups**. Expand the group and read **Inbound Rules**. | Attach a group with an SSH rule for TCP 22. | | The cluster is not Running | The status on the **Details** tab | Wait, or select **Reconfigure Cluster** if it is stuck in **Creating**. | | The IP changed | Compare with **Connection Details** | Use the current IP, then convert it to reserved. | | The login service is restarting | You ran **Restart Login Service** a moment ago, or an image update started | Wait a minute and try again. | | Your own network blocks outbound port 22 | Try from a different network | Use your VPN, or ask your network team. | A security group missing an inbound TCP 22 rule is the most common cause of "the cluster is broken". The cluster itself is fine, and the whole fix is to attach a group that allows port 22 from **Network & Security**. A floating IP can change when the cluster's network resources are recreated, which breaks every SSH config entry and script that uses it. Select **Convert to reserved** once, and the IP stays stable. See [Manage network and security](/tir/tir-slurm-cluster/how-to/manage-network-and-security/). ## Permission denied (publickey) The cluster is reachable, but it rejected your key. Work through these checks in order. 1. **Is the right key offered?** Look for the offered key, then name the key explicitly: ```bash ssh -v root@ 2>&1 | grep -i 'offering\|Offering public key' ssh -i ~/.ssh/id_ed25519 root@ ``` 2. **Is that key on the cluster?** On the **Details** tab, **Connection Details** lists the keys by label. If yours is missing, select **Actions**, then **Update SSH Keys**. Update SSH Keys replaces the whole set, so a key left out loses access. Keep at least two keys on a cluster, so a second key still works when one machine is lost. See [Update a cluster](/tir/tir-slurm-cluster/how-to/update-a-cluster/). 3. **Are your local key permissions right?** SSH refuses a private key that others can read: ```bash chmod 700 ~/.ssh chmod 600 ~/.ssh/id_ed25519 ``` 4. **Are you the right user?** The cluster's own keys log in as `root`. A Login User Management member logs in with their own login name: ```bash ssh root@ ssh @ ``` 5. **Are too many keys offered?** An agent with many keys can reach the server's attempt limit before the right one. Offer only one: ```bash ssh -o IdentitiesOnly=yes -i ~/.ssh/ root@ ``` To use the right key every time, add an entry to `~/.ssh/config`, replacing ``: ```text Host slurm-train HostName User root IdentityFile ~/.ssh/id_ed25519 IdentitiesOnly yes ServerAliveInterval 30 ``` Then connect with `ssh slurm-train`. ## WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED This warning is expected after the login node is recreated by a restart, an image update, or a reconfigure. Remove the stale entry, then reconnect and accept the new key: ```bash ssh-keygen -R ``` Clear the entry only when you know why it changed. The same message is what a real interception looks like. If the cluster has run uninterrupted and kept the same IP, investigate instead. ## A named login user cannot log in `root` works but a Login User Management member does not. Access is granted per group rather than per member, so a new member needs a group and that group needs cluster access. | Cause | Check | Fix | | ---------------------------------- | ------------------------------------------ | ---------------------------------------------------------------------------------------- | | The member's SSH key is missing | **Members**, then the SSH keys count | Open the member's actions menu and update the SSH keys. | | Their group lacks cluster access | The **Groups** tab | Allow access for the group. | | They have yet to join a group | **Members**, filtered by group | Add the member to a group that has access. | | Login User Management is still off | The tab shows the empty state to set it up | Turn it on. See [Manage login users](/tir/tir-slurm-cluster/how-to/manage-login-users/). | | It was turned on moments ago | The tab shows an in-progress state | Wait for it to finish. | | The login name is wrong | The **Login Name** column | Use the exact name, in lowercase. | ```bash ssh -v @ ``` For a member, `Permission denied (publickey)` usually means one of the first three rows. Members log in with keys only, so someone who asks for a password needs their public key attached to their member record instead. ## My SSH session keeps dropping | Cause | Fix | | ----------------------------------- | ------------------------------------------------------------------------- | | An idle timeout on a network device | Add ServerAliveInterval 30 to `~/.ssh/config`. | | The login service restarted | Expected. It recreates the login node and drops every session. | | An image update or reconfigure | The same. | | The login node ran out of resources | Someone is running heavy work on the login node instead of submitting it. | Run training through Slurm rather than in a foreground shell on the login node. The login node only submits work, and the GPUs allocated to your jobs are on the workers. A dropped connection or a login service restart ends a process running there. A job submitted with `sbatch` survives disconnects, restarts, and image updates. `tmux` or `screen` keeps a shell open across a dropped connection, though the session still ends if the login node is recreated. ## Check that you are in Once you are connected, these commands confirm that the cluster is working: ```bash sinfo df -h | grep -E '/pfs|/shared' echo test > /pfs/write-check && rm /pfs/write-check && echo writable squeue srun --help | grep -i container ``` They show that the cluster is scheduling, your storage is mounted and writable, the queue answers, and container support is present. If the problem remains, contact support with the cluster name, the project, and the output of `ssh -v`. See [Troubleshooting TIR Slurm Cluster](/tir/tir-slurm-cluster/troubleshooting/tir-slurm-cluster/#contact-support).