--- title: "How a TIR Kubernetes cluster works" description: "The parts of a TIR Kubernetes cluster, its control plane, worker pools, networking, add-ons, and storage, what each status means, and who can do what." content_type: "concept" product: "tir-kubernetes" platform: "TIR" url: "/tir/tir-kubernetes/concepts/how-a-cluster-works/" --- # How a TIR Kubernetes cluster works A TIR Kubernetes cluster is a Kubernetes cluster whose control plane E2E sets up and runs for you. The platform also sets up the network plugin and the add-ons. Everything you deploy on top works as it does on any Kubernetes cluster. You choose the cluster's parts on the [create form](/tir/tir-kubernetes/how-to/create-a-cluster/), and its statuses tell you where it stands from then on. ## The control plane The control plane runs the Kubernetes API server and the parts that schedule and manage your pods. E2E sets it up and runs it for you. You manage the cluster only through its kubeconfig and the Kubernetes API, rather than over SSH to the nodes. The API server listens on port 6443, and the kubeconfig you download already points at it. You choose two things for the control plane when you create the cluster: - The number of nodes, 1 or 3. One node suits a cluster you are trying out. Three nodes give the control plane redundancy, so choose 3 for a cluster you depend on. - The plan. The control plane runs on CPU plans only, hourly or committed. The control plane nodes are Kubernetes nodes, so `kubectl get nodes` lists them beside the workers. Your own pods run on the worker nodes only. The labels and taints you set in TIR are for worker pools only. The node count and the plan are fixed once you create the cluster: a control plane created with 1 node stays at 1 node, and its plan stays the one you chose. ## Worker pools Worker nodes run your pods. They always belong to a worker pool: a group of nodes that share one plan and one node count. The pool is how you add capacity, scale it, and pay for it. - A cluster needs at least one worker pool and can hold as many as you need. - A pool has from 1 to 25 nodes. - Each pool in a cluster has its own name. You can reuse the name of a pool you have deleted. - Each pool has its own plan, hourly or committed, so one cluster can mix the two. Pools let you run different kinds of work on the right hardware. For example, a small pool can run system services and a larger one your applications. You can add, resize, autoscale, and delete each pool independently of the others. To send pods to a particular pool, give the pool node labels and taints, and use node selectors, affinity rules, or tolerations in your pods. See [Set node labels and taints](/tir/tir-kubernetes/how-to/set-node-labels-and-taints/). ## Networking ### The VPC The cluster's nodes run as virtual machines in a VPC that you choose when you create the cluster. They take their addresses from the VPC's first subnet, so the cluster sits on your private network beside your other TIR resources. The VPC must already exist and be **Active**, and the cluster stays in it for its whole life. See [VPC](/tir/tir-vpc/). The cluster screens have no security group setting. ### Pod and Service address ranges Every cluster uses two fixed address ranges of its own: | Used for | Range | | -------- | --------------- | | Pods | `172.30.0.0/16` | | Services | `172.31.0.0/16` | Keep your VPC ranges, and any network you route to from the cluster, clear of both. Traffic to an address in either range stays inside the cluster instead of reaching your network. ### The network plugin Cilium provides pod and Service networking on every cluster, and the platform chooses and manages it for you. Keep Cilium as the only network plugin, because a second one breaks the cluster's networking. ### Reaching your Services A Service of type LoadBalancer takes its address from the cluster's LB IP pool. The pool holds public reserved IPs that you attach on the **LB IP Pool** tab. A new cluster's pool is empty, and a LoadBalancer Service shows its `EXTERNAL-IP` as `` until the pool has a free address for it. Deleting the Service frees its address for another Service in the same cluster. See [Expose services with a load balancer](/tir/tir-kubernetes/how-to/expose-services-with-a-load-balancer/). ClusterIP and NodePort Services work out of the box, as on any Kubernetes cluster. You can reach a NodePort Service only from networks that can route to the nodes' VPC addresses. ## Add-ons The platform chooses and installs the cluster's add-ons, including Cilium, while it builds the cluster. ## Storage Every cluster comes with Ceph block storage behind a default StorageClass. A pod asks for storage with a volume claim. A claim that leaves out the class gets a persistent volume from the default class, ready to use from the start. The volume keeps its data after the pod restarts. That class supports ReadWriteOnce volumes, which one node mounts at a time. TIR's Shared File System and Parallel File System are not mounted into Kubernetes clusters. Deleting the cluster deletes its volumes. See [Use persistent volumes](/tir/tir-kubernetes/how-to/use-persistent-volumes/). ## Cluster, pool, and node statuses TIR builds a cluster in the background. The cluster is listed at once, and its status moves on by itself. ### Cluster statuses | Status | What it means | | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------- | | **Queued** | TIR has accepted the request and is waiting to start. | | **Cluster Provisioning** | The control plane is being set up. | | **Worker Joining** | Worker nodes are joining the cluster. A **Ready** cluster also returns to this status while a pool you add comes up. | | **Worker Joined** | The workers have joined. This lasts a short time before **Ready**. | | **Ready** | The cluster is ready to use. | | **Failed** | Provisioning stopped before the cluster was ready. TIR shows only the status. To try again, delete it and create a new cluster. | | **Deleting** | The cluster is being deleted. It leaves the list when it is gone. | Provisioning usually takes 10 to 30 minutes. If provisioning runs into trouble, the cluster ends at **Failed** instead of waiting indefinitely. Its control plane is billed until you delete it; see [Billing options for TIR Kubernetes](/tir/tir-kubernetes/concepts/billing-options/). ### What each status allows | Cluster status | Download kubeconfig | Change worker pools | Add reserved IPs to the LB IP pool | Delete the cluster | | ------------------------ | ------------------- | ------------------- | ---------------------------------- | ------------------ | | **Queued** | No | No | No | Yes | | **Cluster Provisioning** | No | No | No | Yes | | **Worker Joining** | No | No | Yes | Yes | | **Worker Joined** | Yes | Yes | Yes | Yes | | **Ready** | Yes | Yes | Yes | Yes | | **Failed** | No | No | No | Yes | | **Deleting** | No | No | No | No | Changing worker pools covers adding, resizing, autoscaling, labels and taints, committed plans, and deleting. When a status blocks an action, its button tells you why. ### Worker pool statuses Each pool has its own status, separate from the cluster's: | Status | What it means | | ------------ | ------------------------------------------------------------------------------------------- | | **Queued** | The pool has been requested and is waiting to start. | | **Joining** | The pool's nodes are being set up and joining. | | **Ready** | The pool is serving. | | **Resizing** | The pool is being set to a new fixed size. | | **Scaling** | The pool's autoscaling is changing its node count. | | **Failed** | The pool ran into a problem, such as nodes that failed to start. TIR shows only the status. | | **Deleting** | The pool is being deleted. | Resizing or autoscaling a pool leaves the cluster **Ready**; only the pool shows **Resizing** or **Scaling**. A pool that fails also leaves the cluster **Ready**, as long as another pool is serving. A new cluster whose first pools all fail ends at **Failed**. ### Node states Each node in a pool has its own state, on the **Node Pool** tab. The states are **Pending**, **Control Plane Provisioning**, **Network Provisioning**, **Network Ready**, **Provisioning**, **Joining**, **Joined**, **Ready**, **Failed**, **Removing**, or **Removed**. A node is **Unknown** when no state is reported for it. ## Who can do what Kubernetes has one permission in a project's policies, with four actions: read, create, update, and delete. A Project Lead has all four, and a Member has none by default. You see only the actions your role allows. When the status blocks one, TIR says why, for example that the action waits until the cluster or pool is ready. | To do this | You need | | ----------------------------------------------------------------------------------- | ---------------------------------------------- | | See the list, the details page and all its tabs, and download the kubeconfig | Read | | Create a cluster | Create | | Add, resize, or autoscale a worker pool, change its labels and taints, or commit it | Update | | Delete a worker pool | Update and delete | | Attach reserved IPs to the LB IP pool | Update on Kubernetes, and create on Reserve IP | | Detach reserved IPs from the LB IP pool | Update on Kubernetes, and delete on Reserve IP | | Delete a cluster | Delete | If your role lacks the read permission, the Kubernetes pages show **Access Denied!**. To change a role, see [Roles and permissions](/tir/tir-iam/concepts/roles-and-permissions/). ## What is billed You pay for a cluster per pool: the control plane is a pool on its own plan, and each worker pool has its own plan. You pay only for the nodes; the pods, Services, and jobs you run on them are included. For when billing starts and stops, and how committed pools are charged, see [Billing options for TIR Kubernetes](/tir/tir-kubernetes/concepts/billing-options/).