Monitor a Kubernetes cluster

View as Markdown

Keep an eye on a cluster from its details page. The Overview tab says what the cluster is, how much capacity it has, and which network it is on. The Monitoring tab, built into every cluster, has the health and resource use of every node. For what runs inside the cluster, such as pods and your own apps, run Prometheus and Grafana in it.

Before you begin

Permission to read Kubernetes in the project. Both tabs need only that.
For Prometheus and Grafana:
- A cluster in the Ready state and its kubeconfig set up for kubectl. See Connect to a Kubernetes cluster.
- Helm installed on your computer. See the Helm install guide.

Open the cluster’s details page

In Kubernetes, select the cluster’s name. To look at another cluster, choose it in Select Kubernetes Cluster at the top of the page.

Read the Overview tab

Overview gives the cluster’s name, status, Kubernetes version, and who created it and when; its capacity, as control plane nodes and as worker pools and nodes; and its VPC, with the VPC’s CIDR.

Workers counts the nodes your pools are set to have, which can differ from the nodes that are ready. To see how many nodes are running, use Nodes Ready on the Monitoring tab, or the node states on the Node Pool tab.

Watch the nodes on the Monitoring tab

The Monitoring tab reads its figures from the virtual machines that run the nodes. It works on every cluster, alongside any monitoring you run yourself.

Choose a time range and a scope

At the top of the tab, choose a time range: 5m, 15m, 1h, 6h, 1d, 7d, or 30d. It starts on 5m.

Then choose what to look at:

Cluster Overview covers health and totals across every node.
Control Plane covers the control plane nodes.
Workers covers the worker nodes, and lets you narrow it to one Worker pool. It is available once the cluster has worker nodes.

With Control Plane or Workers, you can also choose one node to look at.

A dash means the figure was missing from the report, which is different from zero.

Read the cluster’s health

Cluster Overview gives:

Nodes Ready: the nodes that are ready out of those reporting, such as 4/5.
CPU (cores), Memory, and Ephemeral Storage: each a gauge of what is in use against what the nodes have. Ephemeral storage is the nodes’ local disk, whose contents are lost when a node is removed.
Network In and Network Out: the traffic coming into and going out of the nodes.

Four panels below break these down:

CPU Breakdown: User, System, and vCPU Wait. vCPU Wait is the time a virtual CPU was ready to run but waited for the host.
Memory Detail: Page Cache in GB, which is memory the system can reclaim, and Major Faults, which are memory reads that had to go to disk.
Disk I/O: Read and Write, Read IOPS and Write IOPS, and Read Latency and Write Latency in ms.
Network Detail: Packets In and Packets Out in pkt/s, Errors, and Dropped.

The tab flags vCPU Wait, Major Faults, Errors, and Dropped whenever they are above zero. When things feel slow, look at these four first:

A high vCPU Wait points at the host rather than your workload.
A rising Major Faults count means real memory pressure.
Errors or Dropped above zero mean the node’s network is failing or losing packets.

Read the nodes table

The nodes table has a row for each node:

Column	Shows
Node	The node’s hostname and plan
Role	Control plane or Worker
Health	Up, Down, or Unknown when the node is silent
CPU	CPU use
Memory	Memory in use against what the node has
Storage	Disk in use against what the node has
Load	The node’s load average, read against its number of cores
Uptime	How long the node has been up

Select a row to open that node.

Look at one node

A node’s view has its Health, CPU Utilisation, Load Average against the node’s number of cores, and Uptime. Below them are the same gauges and detail panels as for the cluster, scoped to this node. Then come charts of CPU Utilisation, Memory Utilisation, Disk Utilisation, Load Average, and Network Throughput. Select a chart to open it larger.

Until a new cluster’s nodes start reporting, the tab shows “No nodes are reporting metrics for this cluster yet.”

See what runs inside the cluster

The Monitoring tab covers the nodes only. For pods, namespaces, and workloads, such as your Deployments and jobs, use kubectl, or run Prometheus and Grafana in the cluster:

Question	Where to look
Is this node healthy?	The Monitoring tab
How many nodes are ready?	The Monitoring tab
Which pods are restarting?	Prometheus
What is my app’s 99th percentile latency?	Prometheus
Does a Deployment have all its replicas?	Prometheus
Which namespace uses the most memory?	Prometheus

Install kube-prometheus-stack

The community chart kube-prometheus-stack bundles Prometheus, Alertmanager, Grafana, node-exporter, and kube-state-metrics, already wired together. It needs about 2 vCPU and 4 GB of memory across its parts, and Prometheus and Grafana each claim a volume from the cluster’s default StorageClass.

The default StorageClass offers ReadWriteOnce volumes only, and a chart that asks for ReadWriteMany leaves its claims Pending. The flags below set ReadWriteOnce for both volumes:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install monitoring prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace \
  --set prometheus.prometheusSpec.retention=15d \
  --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.accessModes[0]=ReadWriteOnce \
  --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage=50Gi \
  --set grafana.persistence.enabled=true \
  --set grafana.persistence.size=10Gi \
  --set grafana.persistence.accessModes[0]=ReadWriteOnce

Watch the pods start:

kubectl get pods -n monitoring -w

For how volumes work on the cluster, see Use persistent volumes, which keep their data when a pod restarts.

Open Grafana

For a quick look, forward Grafana’s port to your computer:

kubectl port-forward -n monitoring svc/monitoring-grafana 3000:80

Open http://localhost:3000 and sign in as admin. To read the generated password, run:

kubectl get secret -n monitoring monitoring-grafana \
  -o jsonpath='{.data.admin-password}' | base64 -d; echo

Change the password when you first sign in.

To reach Grafana through an address you already have, give it its own hostname on your ingress controller, which routes incoming web traffic by hostname. Grafana can read everything the stack collects, so expose it only with TLS and a changed admin password. Where you can, keep it internal and reach it through a port forward or a VPN.

helm upgrade monitoring prometheus-community/kube-prometheus-stack \
  --namespace monitoring --reuse-values \
  --set grafana.ingress.enabled=true \
  --set grafana.ingress.ingressClassName=nginx \
  --set grafana.ingress.hosts[0]=grafana.example.com

For the ingress controller and HTTPS, see Set up an ingress controller with HTTPS.

What the stack shows

The chart comes with dashboards for:

CPU and memory by cluster, namespace, and pod
The state of Deployments, StatefulSets, and DaemonSets
Pod restarts and container status
Node resources, from node-exporter
The API server, scheduler, and controller manager
Volume use

Its default alerts cover pods that keep crashing, nodes that are not ready, volume claims that are filling up, and failing jobs.

Collect your own app’s metrics

Expose Prometheus metrics from your app, then tell Prometheus to collect them with a ServiceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-app
  namespace: monitoring
  labels:
    release: monitoring
spec:
  namespaceSelector:
    matchNames:
      - default
  selector:
    matchLabels:
      app: my-app
  endpoints:
    - port: metrics
      interval: 30s

The release: monitoring label is what makes this chart’s Prometheus pick the ServiceMonitor up. The app’s Service needs a port named metrics to match:

apiVersion: v1
kind: Service
metadata:
  name: my-app
  labels:
    app: my-app
spec:
  selector:
    app: my-app
  ports:
    - name: metrics
      port: 9090
      targetPort: 9090

To check that Prometheus collects it, forward Prometheus’s port and look under Status > Targets at http://localhost:9090:

kubectl port-forward -n monitoring svc/monitoring-kube-prometheus-prometheus 9090:9090

Keep the stack’s storage in check

How long Prometheus keeps data drives how much storage it uses. Fifteen days in 50 GiB suits a small cluster; a large or busy cluster needs more. To use less, keep data for fewer days, collect less often, or keep only the series you need with relabel rules. The create form shows cluster storage as a usage-based estimate per GB per month. For the rates, see E2E’s pricing page.

Remove the stack

helm uninstall monitoring -n monitoring
kubectl delete namespace monitoring

Helm leaves the stack’s custom resource definitions behind. Remove them only when the stack was their only user. To list them, run:

kubectl get crd | grep monitoring.coreos.com

You now see the cluster’s nodes in the portal and your workloads in Grafana. When a figure points at a problem, such as a node that is not ready, or the stack itself fails to start, see Troubleshooting TIR Kubernetes.