Use an SFS from Kubernetes

View as Markdown

An SFS gives the pods on a Kubernetes cluster shared storage. The files stay in place when a pod restarts or moves to another node, and pods on different nodes all see the same files. Kubernetes calls this a persistent volume, which means it is kept when pods restart.

There are two ways to do this. Set up the storage yourself when one or two apps share a single SFS. Let Kubernetes create it for you when many apps each need their own space on the same SFS.

Before you begin

The Kubernetes cluster and the SFS are in the same region and the same VPC. You pick the cluster’s VPC under Network Settings when you create the cluster.
The SFS gives VPC-wide access, so the cluster’s nodes can reach it; see Grant a VPC access to an SFS.
You have the SFS’s Endpoint, from its Info tab.
kubectl is installed on your computer; see Install Tools. You also need the cluster’s kubeconfig file. To get it, open the cluster’s Cluster Details, find the Kubeconfig row, and click Download. The file is named <cluster name>-kubeconfig.yaml.
For the second way only, Helm is installed on your computer; see Installing Helm.

Point kubectl at the cluster, replacing <path> with where you saved the kubeconfig file:

export KUBECONFIG=<path>
kubectl get nodes

Each node reads Ready when the cluster is ready to use.

Set it up yourself

Save this as sfs-pv.yaml, replacing <endpoint> with the SFS’s Endpoint. It describes the SFS to Kubernetes as a persistent volume that many pods can read and write at once.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: sfs-pv
spec:
  capacity:
    storage: 200Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: ""
  mountOptions:
    - soft
    - timeo=600
    - retrans=2
  nfs:
    server: <endpoint>
    path: /data

Save this as sfs-pvc.yaml. It is the claim your apps use to ask for that volume. The pods can use the whole SFS; the 200Gi in both files matches the claim to the volume.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: sfs-pvc
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: ""
  volumeName: sfs-pv
  resources:
    requests:
      storage: 200Gi

Apply both files, and check that the claim reads Bound:

kubectl apply -f sfs-pv.yaml -f sfs-pvc.yaml
kubectl get pvc sfs-pvc

In your app, mount the claim where the app keeps its files. This example runs two copies of nginx that serve the same web pages from the SFS. Save it as web.yaml, apply it with kubectl apply -f web.yaml, and check that kubectl get pods lists two web pods reading Running:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: nginx
          image: nginx
          volumeMounts:
            - name: sfs
              mountPath: /usr/share/nginx/html
      volumes:
        - name: sfs
          persistentVolumeClaim:
            claimName: sfs-pvc

Deleting this claim and volume later leaves the files on the SFS in place, because the volume keeps its data with the Retain setting.

Let Kubernetes create the storage

The NFS CSI driver lets Kubernetes create a volume on the SFS each time an app asks for one.

Install the driver with Helm:

helm repo add csi-driver-nfs https://raw.githubusercontent.com/kubernetes-csi/csi-driver-nfs/master/charts
helm install csi-driver-nfs csi-driver-nfs/csi-driver-nfs --namespace kube-system --version 4.13.4
kubectl get csidrivers

The driver is ready when kubectl get csidrivers lists nfs.csi.k8s.io.

Save this as sfs-storageclass.yaml, replacing <endpoint> with the SFS’s Endpoint, and apply it with kubectl apply -f sfs-storageclass.yaml. The reclaim policy Delete removes a claim’s files when the claim is deleted; to keep them, set it to Retain before you apply the file, because a StorageClass keeps the settings it was created with:
```
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: sfs
provisioner: nfs.csi.k8s.io
parameters:
  server: <endpoint>
  share: /data
reclaimPolicy: Delete
volumeBindingMode: Immediate
mountOptions:
  - soft
  - timeo=600
  - retrans=2
```

Ask for storage with a claim that names the class. Save this as app-data-pvc.yaml, apply it with kubectl apply -f app-data-pvc.yaml, and check that kubectl get pvc app-data reads Bound. Kubernetes creates the volume and binds the claim to it:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: app-data
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: sfs
  resources:
    requests:
      storage: 10Gi

To mount app-data in your app, add the mount to your container and the volume to the pod’s spec, as in the Deployment in the first method, then apply your app:

containers:
  - name: app
    image: nginx
    volumeMounts:
      - name: sfs
        mountPath: /usr/share/nginx/html
volumes:
  - name: sfs
    persistentVolumeClaim:
      claimName: app-data

Each volume the driver creates is its own folder under /data on the SFS, named after the volume. That keeps each app’s files apart. Every claim can use the whole SFS’s space, whatever size it asks for, so size the SFS for all of them together.

Write a file from one pod and read it from another. Replace <pod-1> and <pod-2> with two pod names from kubectl get pods, and <mount path> with where your app mounts the claim:

kubectl exec <pod-1> -- sh -c "echo hello > <mount path>/test.txt"
kubectl exec <pod-2> -- cat <mount path>/test.txt
kubectl exec <pod-1> -- df -h <mount path>

The second command prints hello, and df shows the SFS’s endpoint in the first column. Your pods now share lasting storage on the SFS. If a pod’s mount reports access denied by server, see Troubleshooting SFS. If it seems to hang, check that the cluster is in the SFS’s VPC; see Troubleshooting SFS.

Use an SFS from Kubernetes

Before you begin

Set it up yourself

Let Kubernetes create the storage

Check that the pods share the SFS