--- title: "Connect to a cluster" description: "Connect to a TIR Slurm Cluster's login node over SSH, find its address, and keep your work safe across dropped connections." content_type: "how-to" product: "tir-slurm-cluster" platform: "TIR" url: "/tir/tir-slurm-cluster/how-to/connect-to-a-cluster/" topic: ["create-and-connect"] --- # Connect to a cluster Sign in to your cluster's login node over SSH. That is where you edit scripts, submit jobs with `sbatch`, and import container images. ## Before you begin - A cluster in **Initializing Worker Nodes** or **Running**, the statuses where **Connect** is available. - The private key that matches one of the SSH keys you selected when you created the cluster. - A security group on the cluster with inbound TCP 22 open. ## Connect as root Every cluster starts with `root` access, using the SSH keys you selected at creation: ```bash ssh root@ ``` Replace `` with the cluster's public IP address. If you use a key other than your default identity, name it: ```bash ssh -i ~/.ssh/ root@ ``` Replace `` with the file name of your private key. To get the command, open the cluster and select **Connect**. It also gives these commands to copy: | Command | What it does | | ---------------------------- | ------------------------------------------- | | `ssh root@` | Connects to the login node. | | `sinfo` | Shows nodes and partitions. | | `scontrol show nodes` | Shows the full detail of every node. | | `sbatch ` | Submits a batch script for later execution. | | `sstat -j ` | Tracks the resource usage of a running job. | | `scontrol show job ` | Tracks one job. | The login node is the place to submit work from. It has your storage mounts and outbound network access, but no GPUs allocated to your own work, so run training through `sbatch` or `srun`. ## Find the cluster's address The cluster's IP address is in two places: - The **Details** tab, under **Connection Details**, next to the SSH command. - The **Network & Security** tab, under **Network**. The **Connect** panel lists one SSH command for each address the cluster has: its public IP, labelled **Floating IP**, and any reserved IP attached to it. The IP is labelled **Floating** or **Reserved**. **Floating** means it is assigned from a pool and can change if the cluster's network resources are recreated. **Reserved** means it is held for your account and stays the same. Convert a floating IP to reserved before anyone scripts against it. A script, a CI job, an SSH config entry, or a bookmark that hard-codes a floating IP breaks silently the moment the address changes. On the **Network & Security** tab, select **Convert to reserved**. See [Manage network and security](/tir/tir-slurm-cluster/how-to/manage-network-and-security/). ### Add an SSH config entry To connect with a short name, add an entry to your SSH config: ```bash cat >> ~/.ssh/config <<'EOF' Host slurm-train HostName User root IdentityFile ~/.ssh/ ServerAliveInterval 30 EOF ``` Then connect with `ssh slurm-train`. The last line sends a keep-alive message every 30 seconds, so idle sessions stay open. ## Submit long work with sbatch A dropped connection takes any process in your foreground shell with it, and restarting the login service drops every open session. Submit long work with `sbatch`, and it survives both. `tmux` or `screen` can hold a shell open across a dropped connection, though the session still ends if the login node itself is recreated. ## Keep your files on a mounted volume Store job scripts, datasets, checkpoints, and container images on a mounted volume rather than in `/root` or anywhere else on the login node's own disk. That disk is erased whenever the login node is recreated, for example by a login service restart, an image update, or a reconfigure. Your storage mounts are at the paths you chose and are the same on the login node and on every worker node. ## Replace the root keys To change which keys have root access, select **Actions**, then **Update SSH Keys**. It replaces the whole set in one step, so a key you leave out of the new selection loses access. Keep at least two keys on every cluster, so a lost laptop is only an inconvenience: the other key still gets you in. See [Update a cluster](/tir/tir-slurm-cluster/how-to/update-a-cluster/#update-ssh-keys). ## Give your team individual logins One shared `root` key works for one person. It works less well once a team shares the cluster: | With a shared `root` login | With individual logins | | --------------------------------------------------------------------------------------- | ------------------------------------------------------------ | | Every job is submitted by `root`, so the **Jobs** tab shows the same user on every job. | Jobs carry the real user in the **Jobs** tab. | | `scancel -u root` cancels every job on the cluster. | Each person's commands affect only their own jobs. | | Everyone shares one home directory, and it is erased when the login node is recreated. | Each person gets a private home directory on shared storage. | | Removing one person means changing the key for everybody. | You revoke one person, and everyone else keeps their access. | **Login User Management** does this from the console. You add a member with their own login name and SSH key, and they connect as themselves with `ssh @`. See [Manage login users](/tir/tir-slurm-cluster/how-to/manage-login-users/). ## Keep these commands at hand These run on the login node. Replace `` with a job's ID from `squeue` or `sbatch`: ```bash sinfo # partitions and node states scontrol show nodes # full detail on every node squeue # the queue squeue -u $USER # your jobs sbatch train.sh # submit a batch job srun --partition=all --nodes=1 --gres=gpu:1 --time=00:30:00 --pty bash # shell on a compute node scontrol show job # full detail on one job sstat -j # live resource use of a running job sacct -j # a job's history and exit code sacct -u $USER -S today # your jobs today scancel # cancel a job scontrol show config # the live slurm.conf enroot import -o /shared/images/.sqsh docker:// # cache a container image ``` In the last command, `` is a file name you choose and `` is the image's registry reference. To run your first jobs, see [Get started with TIR Slurm Cluster](/tir/tir-slurm-cluster/get-started/). If the connection fails, see [Troubleshooting connecting to TIR Slurm Cluster](/tir/tir-slurm-cluster/troubleshooting/connecting/). The usual causes are a security group with TCP 22 closed, a key missing from the cluster, a changed floating IP, or a stale host key after the login node was recreated.