Set node labels and taints

Node labels and taints decide which pods run on a worker pool’s nodes. A label lets a pod ask for the pool’s nodes, with a node selector or an affinity rule. A taint keeps pods off the nodes unless they carry a matching toleration, which is how you reserve a pool for one kind of work. You set them on the worker pool, and TIR applies them to every node in it, including nodes added later.

Before you begin

• A role with the update permission on Kubernetes. See Roles and permissions.
• To change an existing pool, a pool in the Ready state, in a cluster that is Ready or Worker Joined.

How keys and values work

Every label and taint key starts with the fixed prefix node.e2enetworks.com/, and you type the rest. Pods use the full key: a label you enter as team with the value ml is node.e2enetworks.com/team: ml on the node.

• A key, and a label’s value, can be up to 63 characters long, using letters, numbers, hyphens, underscores, and dots. Each starts and ends with a letter or number.
• A label needs a value. A taint’s value can be empty.
• Each label key can appear once in a pool. A taint’s key and effect together can appear once, so the same key can carry two different effects.
• A pool can have up to 20 labels and 10 taints. Add label and Add taint stay available until the pool reaches those limits.

A taint’s effect decides what happens to pods that lack a matching toleration:

• NoSchedule, the default, keeps new pods off the nodes. Pods already running stay.
• PreferNoSchedule keeps new pods off the nodes when there is room elsewhere.
• NoExecute also evicts running pods from the nodes.

These effects work as they do on any Kubernetes cluster.

Set labels and taints on a new pool

When you add a pool, the Add worker node pool dialog has a section for labels and taints. You open the dialog from the create form, or with Add Worker Pool on the Node Pool tab.

1. Select Add label, type the key after the prefix, and type its value. Repeat for each label.
2. Select Add taint, type the key after the prefix, optionally a value, and choose the effect. Repeat for each taint.
3. Finish the rest of the pool and select Add node pool.

For the rest of the dialog, see Add and delete worker pools or Create a Kubernetes cluster.

Change labels and taints on an existing pool

TIR applies your changes in place on the pool’s existing nodes, which keep running. Only a NoExecute taint affects running pods.

Adding a NoExecute taint immediately evicts every running pod on the pool’s nodes that lacks a matching toleration. That includes system pods such as storage and monitoring agents. Add the tolerations your pods need before you add the taint.

1. Open the cluster’s details page and select the Node Pool tab.
2. On the pool’s row, select Manage labels and taints.
3. Add, change, or remove labels and taints.
4. Select Save changes.

Change them in TIR

TIR manages these labels and taints on every node of the pool. If you remove or change one directly on a node, for example with kubectl label or kubectl taint, TIR puts it back. Make every change in Manage labels and taints instead.

Use them from your pods

To run a pod only on a pool’s nodes, give it a node selector with the full label key. To let it run on a tainted pool, give it a toleration that matches the taint’s full key, value, and effect. This example runs on nodes labelled team: ml and tolerates a dedicated: training taint with the NoSchedule effect:

apiVersion: v1
kind: Pod
metadata:
  name: trainer
spec:
  nodeSelector:
    node.e2enetworks.com/team: ml
  tolerations:
    - key: node.e2enetworks.com/dedicated
      operator: Equal
      value: training
      effect: NoSchedule
  containers:
    - name: trainer
      image: nginx

Replace nginx with your own image. A pod with only the toleration can still run on other pools. To keep a pool for one kind of work, use the label and the taint together.

Check the labels and taints

On the Node Pool tab, open the pool. Node labels and Node taints list each one with its full key, and you can copy each one for use in a manifest. To check them from your machine, run:

kubectl get nodes --show-labels
kubectl describe nodes | grep -A3 Taints

Your pool’s nodes now carry the labels and taints you set, and pods land where you meant them to. To change the number of nodes in the pool, see Resize or autoscale a worker pool. If pods stay Pending after you add a taint, see Troubleshooting TIR Kubernetes.