--- title: "Monitor a Kubernetes cluster" description: "Check a TIR Kubernetes cluster's details, node health, CPU, memory, disk, and network in the portal, and run Prometheus and Grafana for what runs inside it." content_type: "how-to" product: "tir-kubernetes" platform: "TIR" url: "/tir/tir-kubernetes/how-to/monitor-a-cluster/" topic: ["monitor-and-delete"] --- # Monitor a Kubernetes cluster Keep an eye on a cluster from its details page. The **Overview** tab says what the cluster is, how much capacity it has, and which network it is on. The **Monitoring** tab, built into every cluster, has the health and resource use of every node. For what runs inside the cluster, such as pods and your own apps, run Prometheus and Grafana in it. ## Before you begin - Permission to read Kubernetes in the project. Both tabs need only that. - For Prometheus and Grafana: - A cluster in the **Ready** state and its kubeconfig set up for `kubectl`. See [Connect to a Kubernetes cluster](/tir/tir-kubernetes/how-to/connect-to-a-cluster/). - Helm installed on your computer. See the [Helm install guide](https://helm.sh/docs/intro/install/). ## Open the cluster's details page In **Kubernetes**, select the cluster's name. To look at another cluster, choose it in **Select Kubernetes Cluster** at the top of the page. ## Read the Overview tab **Overview** gives the cluster's name, status, Kubernetes version, and who created it and when; its capacity, as control plane nodes and as worker pools and nodes; and its VPC, with the VPC's CIDR. **Workers** counts the nodes your pools are set to have, which can differ from the nodes that are ready. To see how many nodes are running, use **Nodes Ready** on the **Monitoring** tab, or the node states on the **Node Pool** tab. ## Watch the nodes on the Monitoring tab The **Monitoring** tab reads its figures from the virtual machines that run the nodes. It works on every cluster, alongside any monitoring you run yourself. ### Choose a time range and a scope At the top of the tab, choose a time range: **5m**, **15m**, **1h**, **6h**, **1d**, **7d**, or **30d**. It starts on **5m**. Then choose what to look at: - **Cluster Overview** covers health and totals across every node. - **Control Plane** covers the control plane nodes. - **Workers** covers the worker nodes, and lets you narrow it to one **Worker pool**. It is available once the cluster has worker nodes. With **Control Plane** or **Workers**, you can also choose one node to look at. A dash means the figure was missing from the report, which is different from zero. ### Read the cluster's health **Cluster Overview** gives: - **Nodes Ready**: the nodes that are ready out of those reporting, such as `4/5`. - **CPU (cores)**, **Memory**, and **Ephemeral Storage**: each a gauge of what is in use against what the nodes have. Ephemeral storage is the nodes' local disk, whose contents are lost when a node is removed. - **Network In** and **Network Out**: the traffic coming into and going out of the nodes. Four panels below break these down: - **CPU Breakdown**: **User**, **System**, and **vCPU Wait**. vCPU Wait is the time a virtual CPU was ready to run but waited for the host. - **Memory Detail**: **Page Cache** in GB, which is memory the system can reclaim, and **Major Faults**, which are memory reads that had to go to disk. - **Disk I/O**: **Read** and **Write**, **Read IOPS** and **Write IOPS**, and **Read Latency** and **Write Latency** in ms. - **Network Detail**: **Packets In** and **Packets Out** in pkt/s, **Errors**, and **Dropped**. The tab flags **vCPU Wait**, **Major Faults**, **Errors**, and **Dropped** whenever they are above zero. When things feel slow, look at these four first: - A high **vCPU Wait** points at the host rather than your workload. - A rising **Major Faults** count means real memory pressure. - **Errors** or **Dropped** above zero mean the node's network is failing or losing packets. ### Read the nodes table The nodes table has a row for each node: | Column | Shows | | ----------- | --------------------------------------------------------- | | **Node** | The node's hostname and plan | | **Role** | **Control plane** or **Worker** | | **Health** | **Up**, **Down**, or **Unknown** when the node is silent | | **CPU** | CPU use | | **Memory** | Memory in use against what the node has | | **Storage** | Disk in use against what the node has | | **Load** | The node's load average, read against its number of cores | | **Uptime** | How long the node has been up | Select a row to open that node. ### Look at one node A node's view has its **Health**, **CPU Utilisation**, **Load Average** against the node's number of cores, and **Uptime**. Below them are the same gauges and detail panels as for the cluster, scoped to this node. Then come charts of **CPU Utilisation**, **Memory Utilisation**, **Disk Utilisation**, **Load Average**, and **Network Throughput**. Select a chart to open it larger. Until a new cluster's nodes start reporting, the tab shows "No nodes are reporting metrics for this cluster yet." ## See what runs inside the cluster The **Monitoring** tab covers the nodes only. For pods, namespaces, and workloads, such as your Deployments and jobs, use `kubectl`, or run Prometheus and Grafana in the cluster: | Question | Where to look | | ----------------------------------------- | ---------------------- | | Is this node healthy? | The **Monitoring** tab | | How many nodes are ready? | The **Monitoring** tab | | Which pods are restarting? | Prometheus | | What is my app's 99th percentile latency? | Prometheus | | Does a Deployment have all its replicas? | Prometheus | | Which namespace uses the most memory? | Prometheus | ### Install kube-prometheus-stack The community chart kube-prometheus-stack bundles Prometheus, Alertmanager, Grafana, node-exporter, and kube-state-metrics, already wired together. It needs about 2 vCPU and 4 GB of memory across its parts, and Prometheus and Grafana each claim a volume from the cluster's default StorageClass. The default StorageClass offers ReadWriteOnce volumes only, and a chart that asks for ReadWriteMany leaves its claims `Pending`. The flags below set ReadWriteOnce for both volumes: ```bash helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo update helm install monitoring prometheus-community/kube-prometheus-stack \ --namespace monitoring \ --create-namespace \ --set prometheus.prometheusSpec.retention=15d \ --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.accessModes[0]=ReadWriteOnce \ --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage=50Gi \ --set grafana.persistence.enabled=true \ --set grafana.persistence.size=10Gi \ --set grafana.persistence.accessModes[0]=ReadWriteOnce ``` Watch the pods start: ```bash kubectl get pods -n monitoring -w ``` For how volumes work on the cluster, see [Use persistent volumes](/tir/tir-kubernetes/how-to/use-persistent-volumes/), which keep their data when a pod restarts. ### Open Grafana For a quick look, forward Grafana's port to your computer: ```bash kubectl port-forward -n monitoring svc/monitoring-grafana 3000:80 ``` Open `http://localhost:3000` and sign in as `admin`. To read the generated password, run: ```bash kubectl get secret -n monitoring monitoring-grafana \ -o jsonpath='{.data.admin-password}' | base64 -d; echo ``` Change the password when you first sign in. To reach Grafana through an address you already have, give it its own hostname on your ingress controller, which routes incoming web traffic by hostname. Grafana can read everything the stack collects, so expose it only with TLS and a changed admin password. Where you can, keep it internal and reach it through a port forward or a VPN. ```bash helm upgrade monitoring prometheus-community/kube-prometheus-stack \ --namespace monitoring --reuse-values \ --set grafana.ingress.enabled=true \ --set grafana.ingress.ingressClassName=nginx \ --set grafana.ingress.hosts[0]=grafana.example.com ``` For the ingress controller and HTTPS, see [Set up an ingress controller with HTTPS](/tir/tir-kubernetes/how-to/set-up-an-ingress-controller/). ### What the stack shows The chart comes with dashboards for: - CPU and memory by cluster, namespace, and pod - The state of Deployments, StatefulSets, and DaemonSets - Pod restarts and container status - Node resources, from node-exporter - The API server, scheduler, and controller manager - Volume use Its default alerts cover pods that keep crashing, nodes that are not ready, volume claims that are filling up, and failing jobs. ### Collect your own app's metrics Expose Prometheus metrics from your app, then tell Prometheus to collect them with a ServiceMonitor: ```yaml apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: my-app namespace: monitoring labels: release: monitoring spec: namespaceSelector: matchNames: - default selector: matchLabels: app: my-app endpoints: - port: metrics interval: 30s ``` The `release: monitoring` label is what makes this chart's Prometheus pick the ServiceMonitor up. The app's Service needs a port named `metrics` to match: ```yaml apiVersion: v1 kind: Service metadata: name: my-app labels: app: my-app spec: selector: app: my-app ports: - name: metrics port: 9090 targetPort: 9090 ``` To check that Prometheus collects it, forward Prometheus's port and look under **Status** > **Targets** at `http://localhost:9090`: ```bash kubectl port-forward -n monitoring svc/monitoring-kube-prometheus-prometheus 9090:9090 ``` ### Keep the stack's storage in check How long Prometheus keeps data drives how much storage it uses. Fifteen days in 50 GiB suits a small cluster; a large or busy cluster needs more. To use less, keep data for fewer days, collect less often, or keep only the series you need with relabel rules. The create form shows cluster storage as a usage-based estimate per GB per month. For the rates, see [E2E's pricing page](https://www.e2enetworks.com/pricing). ### Remove the stack ```bash helm uninstall monitoring -n monitoring kubectl delete namespace monitoring ``` Helm leaves the stack's custom resource definitions behind. Remove them only when the stack was their only user. To list them, run: ```bash kubectl get crd | grep monitoring.coreos.com ``` You now see the cluster's nodes in the portal and your workloads in Grafana. When a figure points at a problem, such as a node that is not ready, or the stack itself fails to start, see [Troubleshooting TIR Kubernetes](/tir/tir-kubernetes/troubleshooting/tir-kubernetes/).