--- title: Cannot Connect sidebar_label: Connectivity --- import { Wifi, Key, Users, Shield } from 'react-feather'; # Cannot Connect SSH failures, for `root` and for named login users. }, { href: '#permission-denied-publickey', label: 'Permission denied', icon: }, { href: '#host-key-verification-failed', label: 'Host key changed', icon: }, { href: '#a-named-user-cannot-log-in', label: 'A named user', icon: }, { href: '#the-session-keeps-dropping', label: 'Session drops', icon: }, ]} /> --- ## Start Here ```bash ssh -v root@ ``` The verbose output usually names the problem — and confirms which key was offered. Check three things in the console before anything else: | Check | Where | |-------|-------| | The cluster status is **Running** | [Details tab](/docs/tir/SlurmCluster/manage/) | | The IP you are using is the current one | **Details** → **Connection Details**, or **Network & Security** | | A security group allows inbound TCP 22 | **Network & Security** → **Security Groups** | --- ## Connection Refused or Timeout | Cause | How to confirm | Fix | |-------|----------------|-----| | Port 22 is not open | **Network & Security** → **Security Groups** → expand the group → **Inbound Rules** | Attach a group with an `SSH / TCP / 22` rule | | The cluster is not Running | The status pill | Wait, or [Reconfigure Cluster](/docs/tir/SlurmCluster/manage/actions#reconfigure-cluster) if it is stuck Creating | | The IP changed | Compare with **Connection Details** | Use the current IP, then [convert it to reserved](/docs/tir/SlurmCluster/manage/network-security#public-ip) | | The login service is restarting | You just ran **Restart Login Service** or an image update | Wait a minute and retry | | Your own network blocks outbound 22 | Try from a different network | Use your VPN, or ask your network team | :::danger A security group without inbound TCP 22 makes the cluster unreachable Provisioning succeeds and the cluster runs perfectly — you simply cannot log in. This is the single most common cause of "the cluster is broken". You do not need to recreate anything: attach a group that allows port 22 from **Network & Security**. ::: :::tip Convert the IP to reserved once, and stop chasing it A **Floating** IP can change when the cluster's network resources are recreated, breaking every `~/.ssh/config` entry and CI job that hard-codes it. One click on **Convert to reserved** removes the whole problem. ::: --- ## Permission Denied (publickey) The cluster is reachable; your key was not accepted. ### 1. Is the right key being offered? ```bash ssh -v root@ 2>&1 | grep -i 'offering\|Offering public key' ``` Be explicit: ```bash ssh -i ~/.ssh/id_ed25519 root@ ``` ### 2. Is that key on the cluster? **Details** → **Connection Details** → **SSH Keys** lists them by label. If the key you are using is not there, add it with [**Actions → Update SSH Keys**](/docs/tir/SlurmCluster/manage/actions#update-ssh-keys). :::danger Update SSH Keys replaces the whole set Keys left out of the selection lose access. The dialog pre-selects the current keys — deselecting one removes it. Keep at least two keys on every cluster so one lost machine is not a lockout. ::: ### 3. Local key permissions ```bash chmod 700 ~/.ssh chmod 600 ~/.ssh/id_ed25519 ``` SSH refuses to use a private key that is group- or world-readable. ### 4. Are you the right user? ```bash ssh root@ # the cluster's own keys ssh janedoe@ # a Login User Management member ``` A member's key gets them in as **themselves**, not as `root`. ### 5. Too many keys offered An agent with many keys can exhaust the server's attempt limit before reaching the right one: ```bash ssh -o IdentitiesOnly=yes -i ~/.ssh/ root@ ``` :::tip Put it in `~/.ssh/config` once ``` Host slurm-train HostName User root IdentityFile ~/.ssh/id_ed25519 IdentitiesOnly yes ServerAliveInterval 30 ``` Then `ssh slurm-train`, with the right key, every time. ::: --- ## Host Key Verification Failed ``` WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! ``` Expected after the login node is recreated — a **Restart Login Service**, an image update, or a reconfigure. Remove the stale entry: ```bash ssh-keygen -R ``` Then reconnect and accept the new key. :::warning Only clear it when you know why it changed The message is also what a genuine interception looks like. If nothing on the cluster was restarted and the IP has not changed, do not clear the entry — investigate. ::: --- ## A Named User Cannot Log In `root` works, a [Login User Management](/docs/tir/SlurmCluster/connect/login-user-management) member does not. | Cause | Check | Fix | |-------|-------|-----| | No SSH key attached to the member | **Login User Management** → **Members** → the **SSH Keys** column | ⋮ → **Update SSH Keys** | | Their group has no cluster access | **Groups** tab → **Cluster Access** column | ⋮ → **Allow Access** | | They are in no group | **Members** filtered by group | ⋮ → **Manage Members** on a group with access | | Login User Management is not enabled | The tab shows the **Set up Login User Management** empty state | [Enable it](/docs/tir/SlurmCluster/connect/login-user-management#enable-it) | | It was enabled moments ago | The panel shows an in-progress state | Wait for it to finish | | The login name is wrong | The **Login Name** column | Use the exact name — lowercase, no typos | ```bash ssh -v janedoe@ ``` `Permission denied (publickey)` for a member usually means one of the first three rows above. :::info Members log in with keys only, never a password There is no password to issue or reset. If someone asks for one, they need their public key attached to their member record. ::: :::warning Access is granted per group, not per member Adding a member does not grant them anything on its own. They need a group, and that group needs **Allowed** cluster access. This trips people up on the first user they add. ::: --- ## The Session Keeps Dropping | Cause | Fix | |-------|-----| | Idle timeout on a network device | Add `ServerAliveInterval 30` to `~/.ssh/config` | | The login service restarted | Expected — it recreates the pod and drops every session | | An image update or reconfigure | Same | | The login node ran out of resources | Someone is running heavy work on the login node instead of submitting it | :::danger Never run training in a foreground shell on the login node The login node is a submit host with no GPUs allocated to your work. Heavy processes there load it until everyone's sessions suffer, and a dropped connection or a login-service restart kills the process outright. Use `sbatch`. A Slurm job survives disconnects, restarts and image updates; a shell does not. ::: :::tip `tmux` helps, but only within the pod's life `tmux` or `screen` survives a dropped connection. It does **not** survive the pod being recreated by a restart or an image update. For anything that must not be lost, it has to be a Slurm job. ::: --- ## Verify Once You Are In ```bash sinfo # the cluster is scheduling df -h | grep -E '/pfs|/shared' # your storage is mounted echo test > /pfs/write-check && rm /pfs/write-check && echo writable squeue # the queue is reachable srun --help | grep -i container # container support is present ``` --- ## Related Resources - [Connect to the cluster](/docs/tir/SlurmCluster/connect/) - [Login User Management](/docs/tir/SlurmCluster/connect/login-user-management) - [Network and security](/docs/tir/SlurmCluster/manage/network-security) - [Cluster actions](/docs/tir/SlurmCluster/manage/actions)