Manage network and security

Control how your cluster is reached: its public IP, an optional private VPC address, and the security groups that decide which traffic is allowed. All of it is on the Network & Security tab, which has the sub-tabs Network and Security Groups.

Before you begin

Make sure that your cluster is Running. Network & Security is available in that state.

Keep the public IP stable

The IP on the Network sub-tab is the address you connect to over SSH. It is labelled Floating or Reserved:

• Floating means the address is assigned from a pool and can change if the cluster’s network resources are recreated.
• Reserved means the address is held for your account and stays the same.

To keep a floating IP, select Convert to reserved on its row. Do this before anyone scripts against the address: a floating IP can change at any time, and every SSH config entry, CI job, bookmark, and runbook that hard-codes it then stops working.

To use an address you have already reserved for your account, for example one your corporate firewall already allows, select + Add Reserved IP. To reserve one, see Reserve an IP. New connections must then use the new address, so tell your team and check the Connection Details on the Details tab. Anyone already connected on the old address stays connected until they reconnect.

A cluster must always keep at least one IP, reserved or floating. To swap addresses, attach the new one first, then detach the old one.

The public IP lands on the login node only. Worker nodes have addresses inside the cluster only.

Attach a VPC IP

A VPC IP puts the cluster on your private network, so it can reach other resources, such as a database or an internal registry, over the private network instead of the internet. It sits alongside the public IP, so SSH still uses the public address.

1. On the Network sub-tab, choose an existing private address from Select VPC IP, then select Attach.
2. To create a new private address, select + Reserve New VPC IP.

To set up the VPC itself, see Create a VPC.

Manage security groups

On the Security Groups sub-tab, choose a group from Select Security Groups and select Attach. To make a group with your own rules first, see Create a security group. The sub-tab lists the attached groups. Expand one to see its Inbound Rules and Outbound Rules, each with a Type, Protocol, Ports, Source, and Description. A cluster must keep at least one security group, so attach a replacement before you remove the last one.

Open inbound TCP 22 to reach the login node over SSH. The cluster keeps running either way, but SSH times out until a group allows port 22. If SSH times out, check this tab first and attach such a group; that is the only change the cluster needs.

Traffic between nodes, including Slurm and NCCL, is handled inside the cluster, so the only inbound rule a Slurm Cluster needs is TCP 22. Keep every other port closed. The cluster’s nodes sit together on a high-speed RDMA fabric. For a container job to use it, mount /dev/infiniband into the container. See Run multi-node training.

To narrow who can connect, restrict the source of the SSH rule to your office or VPN range rather than removing the rule. With Any Network on port 22, the login node’s log fills with login attempts from the internet. The login node accepts SSH keys only, so these attempts are rejected, and seeing them in the log is normal. Narrowing the source removes the noise and the risk: create a group with your own range, attach it, and detach the open one.

A security group also governs outbound traffic, so a group that restricts it can break image pulls from a registry. See Run containers in jobs.

Harden the cluster

• Restrict inbound SSH to your own address ranges.
• Convert the public IP to reserved, so an allow-list stays valid.
• Use individual logins instead of one shared root key. See Manage login users.
• Keep two root SSH keys. Fewer keys are fewer ways in, and a second key still gets you in when one laptop is lost.
• Attach a VPC IP for private data sources, to keep that traffic off the public internet.
• Review the attached groups when a cluster changes hands, because a group added for a one-off test tends to stay attached.

When someone asks for a password to the cluster, add an SSH key for them instead, preferably as their own login user, or else to the root keys.

To connect to the cluster, see Connect to a cluster. If SSH fails, see Troubleshooting connecting to TIR Slurm Cluster.