---
title: "Use node affinity"
description: "Group a private cluster's nodes and point a workload's scheduling at the group, or keep it off the group's nodes."
content_type: "how-to"
product: "tir-private-cluster"
platform: "TIR"
url: "/tir/tir-private-cluster/how-to/use-node-affinity/"
topic: ["manage-nodes"]
---

# Use node affinity

Group a private cluster's nodes, then keep a workload on that group's nodes, or off them, when you
launch it. By default, TIR can place a workload on any node in the cluster. With node
groups you decide, for example, that inference runs only on the nodes set aside for it, or that
experiments stay away from the nodes running production work.

```mermaid
flowchart LR
    N[Private cluster nodes] --> G["Node group: inference-nodes"]
    G -->|Affinity| R1[Workload runs only on these nodes]
    N -->|Anti-Affinity on inference-nodes| R2[Workload runs on any other node in the cluster]
    N -->|All Nodes, or rule left unset| R3[Workload runs on any node in the cluster]
```

## Before you begin

- A private cluster with nodes allocated to your project. See [Allocate nodes to
  projects](/tir/tir-private-cluster/how-to/allocate-nodes-to-projects/).
- Update access to every project whose nodes you want in the group. You need the same access to
  edit or delete the group later.

## Plan your groups

Name a group for what it does, such as `inference-nodes`, `gpu-nodes-1`, `worker-1`, `testing-workers`, or `high-workload-nodes`, rather than by node ID. The name then stays meaningful as the cluster's layout changes. Keep each group small and specific to the workloads that
use it.

Then choose a rule for each workload:

| Scheduling Rule   | Where the workload runs                                     | Use it for                                                                     |
| ----------------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------ |
| **All Nodes**     | Any node in the cluster, the same as leaving the rule unset | Workloads that can run on any node                                             |
| **Affinity**      | Only the group's nodes                                      | Work that needs particular hardware, such as GPU-specific or high-memory nodes |
| **Anti-Affinity** | Any node in the cluster except the group's nodes            | Experimental or bursty jobs you want kept off stable production nodes          |

## Create a node group

1. Open the cluster and select the **Node Affinity** tab. The page is titled **Node Groups**, with
   the cluster's **Total Nodes**, the number of **Groups**, and the groups you already have.
2. Select **+ New group**.
3. Name the group, up to 64 characters. Each group on a cluster needs its own name.
4. Select one or more of the cluster's nodes for it, and confirm.

The group appears on the page with its name, its number of nodes, and up to four of their names.

## Apply a group to a workload

On the service's create form, under the optional **Affinity Group** section, choose a **Scheduling
Rule**. For **Affinity** or **Anti-Affinity**, also choose the node group. See [Launch services on a
private cluster](/tir/tir-private-cluster/how-to/launch-services-on-a-private-cluster/).

## Edit a node group

Before you edit a group, check which workloads use it, so that you know which ones a change affects.

1. On the **Node Affinity** tab, select **Edit** on the group.
2. Change the name, or the set of nodes. The nodes you select become the group's complete new list.
3. Save the change.

An instance or model endpoint already running under the group's rule stays where it is. To move it
onto the updated group, remove the rule from it, update the group, then apply the rule again.

## Delete a node group

1. Remove the group's rule from every instance and model endpoint that uses it.
2. On the **Node Affinity** tab, select **Delete** on the group, and confirm.

While an instance or model endpoint still uses the group, TIR refuses the delete with
"Private-cloud affinity is being used by notebook or inference services."

Your workloads now land on the nodes you meant for them. If one does not start on the nodes you
expect, see [Troubleshooting TIR Private
Cluster](/tir/tir-private-cluster/troubleshooting/tir-private-cluster/).