Use persistent volumes

View as Markdown

Keep the data your pods write on a persistent volume, which is storage that stays when a pod is restarted or moved. A pod’s own filesystem is lost when the pod is rescheduled. A node’s local disk goes with the node when its pool is scaled down. Every TIR Kubernetes cluster has a default StorageClass, so you can create a PersistentVolumeClaim right away.

Before you begin

You need a cluster in the Ready state and its kubeconfig set up for kubectl. See Connect to a Kubernetes cluster.

The default StorageClass

Every cluster you create in the portal comes with Ceph RBD block storage behind a default StorageClass, which supports ReadWriteOnce volumes. The create form shows it under Cluster storage for information only.

Each claim creates its own volume, so a claim is all you write.
The access mode is ReadWriteOnce.
The create form shows cluster storage as a usage-based estimate per GB per month. See E2E’s pricing page.
The volumes are deleted with the cluster.

Check that the class is there, and which one is the default:

kubectl get storageclass

The default class is marked (default). A claim that skips the StorageClass field uses it.

What ReadWriteOnce allows

ReadWriteOnce means that one node at a time mounts the volume for reading and writing. Several pods can share it only while they run on the same node.

You want	Works?	What to do
One pod writing to one volume	Yes	Create one claim and mount it in the pod
Several replicas, each with its own volume	Yes	Use a StatefulSet with volume claim templates
Several pods on one node sharing a volume	Yes	Schedule them onto the same node with pod affinity
Several pods on different nodes sharing one volume	No	Have one pod write and the others read through an API, or keep the files in object storage

The built-in storage offers ReadWriteOnce volumes only. A claim that asks for ReadWriteMany stays Pending, and a second pod on another node that mounts the same ReadWriteOnce volume stays in ContainerCreating. TIR Shared File System and Parallel File System are separate products, and they are not mounted into Kubernetes clusters. For files that many pods read, use TIR Object Storage.

Claim a volume

A PersistentVolumeClaim asks for storage, and the StorageClass creates a volume for it. This claim asks for 20 GiB:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: app-data
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi

kubectl apply -f pvc.yaml
kubectl get pvc app-data

The claim reads Bound once its volume exists. A class whose binding mode is WaitForFirstConsumer waits for a pod to use the claim, so the claim reads Pending until then. The VOLUMEBINDINGMODE column of kubectl get storageclass shows which mode the default class uses.

Mount the volume in a pod

Name the claim under volumes and mount it in the container:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app
  template:
    metadata:
      labels:
        app: app
    spec:
      containers:
        - name: app
          image: nginx
          volumeMounts:
            - name: data
              mountPath: /var/lib/app
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: app-data

Check that the data stays after a restart:

kubectl exec deploy/app -- sh -c 'echo hello > /var/lib/app/test.txt'
kubectl rollout restart deploy/app
kubectl exec deploy/app -- cat /var/lib/app/test.txt

The last command prints hello.

Keep a Deployment that shares one ReadWriteOnce claim at one replica. With several replicas, it works only while every replica lands on the same node. When each replica needs storage of its own, use a StatefulSet.

Give each replica its own volume with a StatefulSet

For databases, and anything else where each replica needs its own disk, let a StatefulSet create one claim per replica. The Secret db-secret holds the database password under the key password. Create it first, with your own password in place of <password>:

kubectl create secret generic db-secret --from-literal=password=<password>

Then apply the StatefulSet:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: db
spec:
  serviceName: db
  replicas: 3
  selector:
    matchLabels:
      app: db
  template:
    metadata:
      labels:
        app: db
    spec:
      containers:
        - name: db
          image: postgres:16
          env:
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: db-secret
                  key: password
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 50Gi

This creates the claims data-db-0, data-db-1, and data-db-2, each bound to its own pod. Each is ReadWriteOnce, which suits replicas that each keep a disk of their own. To list them, run:

kubectl get pvc -l app=db

Make a volume larger

A volume can grow if its StorageClass allows expansion. Check the class first:

kubectl get storageclass -o custom-columns=NAME:.metadata.name,EXPANSION:.allowVolumeExpansion

If EXPANSION reads true, raise the claim’s requested size and watch it change:

kubectl patch pvc app-data -p '{"spec":{"resources":{"requests":{"storage":"40Gi"}}}}'
kubectl get pvc app-data -w

Volumes can only grow. To make one smaller, create a smaller volume and copy the data across.

Delete a volume

Whether deleting a claim also deletes its volume depends on the StorageClass’s reclaim policy. Check it before you delete:

kubectl get storageclass -o custom-columns=NAME:.metadata.name,RECLAIM:.reclaimPolicy

Delete removes the volume and its data with the claim. Retain keeps the volume, and you clean it up yourself.

kubectl delete pvc app-data

Keep data safe when nodes change

Before you scale a worker pool down, make sure that your workloads have replicas on other nodes and keep their data on persistent volumes. The nodes removed from the pool take their local disks with them. See Resize or autoscale a worker pool.

Deleting the cluster permanently deletes its storage and every volume on it. Copy anything you need out of the cluster first. See Delete a Kubernetes cluster.

Your pods now keep their data across restarts. For a claim that stays Pending or a pod that cannot mount its volume, see Troubleshooting TIR Kubernetes.