Set up an ingress controller with HTTPS

View as Markdown

Serve many sites from one public address with an NGINX ingress controller, then give each site a free HTTPS certificate with cert-manager. An ingress controller is a single entry point for incoming web traffic: one LoadBalancer Service that holds one address and routes each request to the right backend by its hostname and path. Otherwise, every public site needs a LoadBalancer Service and a reserved IP of its own, and each reserved IP is billed. Behind the controller, your backend Services stay ClusterIP, and a new site shares the address the controller already has.

Diagram as text

flowchart TD
    IP[One reserved IP] --> C[ingress-nginx controller, type LoadBalancer]
    C -->|app.example.com| W[web Service, ClusterIP]
    C -->|api.example.com| A[api Service, ClusterIP]
    C -->|docs.example.com| D[docs Service, ClusterIP]

Before you begin

A cluster in the Ready state and its kubeconfig set up for kubectl. See Connect to a Kubernetes cluster.
A free address in the cluster’s LB IP pool. The controller’s own Service takes one, so attach a reserved IP first; see Expose services with a load balancer.
Helm on your computer. See the Helm install guide.
Access to the DNS zone of the hostnames you plan to serve.

Install ingress-nginx

Install the controller with its Service set to LoadBalancer, and make it the default ingress class so that an Ingress that leaves out the class still uses it:

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

helm install ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx \
  --create-namespace \
  --set controller.service.type=LoadBalancer \
  --set controller.ingressClassResource.default=true

Watch the controller’s Service take an address from the pool:

kubectl get svc -n ingress-nginx -w

NAME                       TYPE           EXTERNAL-IP      PORT(S)
ingress-nginx-controller   LoadBalancer   203.0.113.10     80:31234/TCP,443:31235/TCP

The EXTERNAL-IP is one of the reserved IPs you attached, and the examples that follow use 203.0.113.10 for it. Traffic flows only once an address replaces <pending>; see Troubleshooting TIR Kubernetes.

(Optional) To keep each visitor’s IP address in the requests your apps receive, or to run more than one controller replica, add these flags to the helm install command:

# Keep the client's IP address in requests to your apps
--set controller.service.externalTrafficPolicy=Local
# Run two controller replicas
--set controller.replicaCount=2

Point DNS at the address

At your DNS provider, create an A record for each hostname, all pointing at the same address:

app.example.com.   A   203.0.113.10
api.example.com.   A   203.0.113.10

The controller tells the sites apart by the Host header of each request. Check that a record resolves before you go further:

dig +short app.example.com

Deploy the backends

This example runs two small apps, each behind a ClusterIP Service. Save it as backends.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: web
          image: nginx
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  type: ClusterIP
  selector:
    app: web
  ports:
    - port: 80
      targetPort: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 2
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
        - name: api
          image: hashicorp/http-echo
          args: ["-text=hello from api"]
          ports:
            - containerPort: 5678
---
apiVersion: v1
kind: Service
metadata:
  name: api
spec:
  type: ClusterIP
  selector:
    app: api
  ports:
    - port: 80
      targetPort: 5678

kubectl apply -f backends.yaml

Route each hostname to its Service

An Ingress holds the routing rules. This one sends each hostname to its own Service. Save it as ingress.yaml:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: sites
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
  ingressClassName: nginx
  rules:
    - host: app.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: web
                port:
                  number: 80
    - host: api.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: api
                port:
                  number: 80

Apply it and test both sites:

kubectl apply -f ingress.yaml
kubectl get ingress sites
curl http://app.example.com
curl http://api.example.com

The first curl returns the nginx welcome page and the second hello from api. A third site now needs only a ClusterIP Service, a DNS record, and another rule.

Route by path instead

To serve several apps under one hostname, route by path:

spec:
  ingressClassName: nginx
  rules:
    - host: example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: web
                port: { number: 80 }
          - path: /api
            pathType: Prefix
            backend:
              service:
                name: api
                port: { number: 80 }

If the backend serves its routes from / rather than /api, strip the prefix with a rewrite:

metadata:
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  rules:
    - host: example.com
      http:
        paths:
          - path: /api(/|$)(.*)
            pathType: ImplementationSpecific
            backend:
              service:
                name: api
                port: { number: 80 }

(Optional) Raise request limits

These annotations go in an Ingress’s metadata when a site needs larger uploads or a slow backend needs more time:

metadata:
  annotations:
    # Accept request bodies up to 50 MB
    nginx.ingress.kubernetes.io/proxy-body-size: "50m"
    # Wait up to 300 seconds for the backend
    nginx.ingress.kubernetes.io/proxy-read-timeout: "300"

Add HTTPS with cert-manager

cert-manager requests TLS certificates from Let’s Encrypt and renews them before they expire. The HTTP-01 check used here works by Let’s Encrypt fetching a file from your hostname over port 80, so every hostname must resolve publicly to the ingress address and answer on port 80. Check both first:

dig +short app.example.com
curl -I http://app.example.com

Install cert-manager

helm repo add jetstack https://charts.jetstack.io
helm repo update

helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --set crds.enabled=true

kubectl get pods -n cert-manager

Three pods reach Running: the controller, the webhook, and the CA injector.

Test with the staging issuer

Start with Let’s Encrypt’s staging issuer. Its rate limits are generous and browsers flag its certificates as untrusted, which is what you want while you get the setup right. In email, replace user@example.com with an address you read; Let’s Encrypt sends expiry and problem notices there. Save it as issuer-staging.yaml:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: user@example.com
    privateKeySecretRef:
      name: letsencrypt-staging-account-key
    solvers:
      - http01:
          ingress:
            class: nginx

kubectl apply -f issuer-staging.yaml
kubectl get clusterissuer

READY reads True once the issuer has registered. Then add the issuer annotation and a tls block to the Ingress, and cert-manager requests the certificate. Save it as ingress-tls.yaml:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: sites
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-staging
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - app.example.com
      secretName: app-example-com-tls
  rules:
    - host: app.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: web
                port:
                  number: 80

kubectl apply -f ingress-tls.yaml
kubectl get certificate -w

READY usually changes to True within a minute or two. If it stays False, follow the request:

kubectl describe certificate app-example-com-tls
kubectl get certificaterequest
kubectl get challenge

Switch to production

Production limits how many certificates a domain can get each week, which is why you test on staging first. Once staging issues a certificate, create the production issuer as issuer-prod.yaml, again with your own address in email:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: user@example.com
    privateKeySecretRef:
      name: letsencrypt-prod-account-key
    solvers:
      - http01:
          ingress:
            class: nginx

Apply the issuer, point the Ingress at it, and delete the staging certificate’s secret so that a new certificate is issued:

kubectl apply -f issuer-prod.yaml
kubectl annotate ingress sites \
  cert-manager.io/cluster-issuer=letsencrypt-prod --overwrite
kubectl delete secret app-example-com-tls
kubectl get certificate -w

Check the certificate the site now serves:

curl -I https://app.example.com
echo | openssl s_client -connect app.example.com:443 -servername app.example.com 2>/dev/null \
  | openssl x509 -noout -issuer -dates

The issuer now reads plain Let’s Encrypt, and (STAGING) is gone.

Redirect HTTP to HTTPS

Turn the redirect on only after the first certificate exists. The HTTP-01 check needs port 80, and a redirect turned on too early blocks your own first certificate. Then set these annotations on the Ingress:

metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/ssl-redirect: "true"

Cover several hostnames

One certificate can cover several hostnames. Each one still needs its own DNS record pointing at the ingress address, and its own rule:

spec:
  tls:
    - hosts:
        - app.example.com
        - api.example.com
      secretName: example-com-tls

cert-manager renews each certificate by itself, normally 30 days before it expires. To see every certificate and when one expires, run:

kubectl get certificate -A
kubectl get certificate app-example-com-tls -o jsonpath='{.status.notAfter}'

Your sites now share one reserved IP and answer over HTTPS, and each new site costs you a DNS record and a rule rather than another address. To give a monitoring dashboard such as Grafana its own hostname on this controller, see Monitor a Kubernetes cluster. When a site returns 404 or 503, or a certificate stays False, see Troubleshooting TIR Kubernetes.