Kubernetes Backup & Restore Guide
Using Velero and E2E Object Storage
A step-by-step implementation guide for E2E Kubernetes customers to configure automated, policy-driven backup and disaster recovery for stateful workloads running on E2E Cloud.
1. Introduction
E2E Networks Ltd. is one of India's leading cloud infrastructure providers, offering compute, networking, managed Kubernetes (E2E Kubernetes), and S3-compatible object storage powered by MinIO. As organizations migrate and grow stateful workloads on E2E Cloud, protecting persistent application data and Kubernetes configuration becomes a critical operational requirement.
This guide documents how to implement a production-grade backup and restore solution for E2E Kubernetes clusters using Velero — the industry-standard open-source Kubernetes backup tool — integrated with E2E Object Storage as the backup store. The procedures in this document have been validated through a Proof of Concept (POC) conducted on E2E Cloud infrastructure.
Key Terms
| Term | Definition |
|---|---|
| Velero | Open-source tool for backing up and restoring Kubernetes cluster resources and persistent volumes. |
| E2E Object Storage | S3-compatible object storage provided by E2E Networks, powered by MinIO. |
| PVC (PersistentVolumeClaim) | A request for storage by a Kubernetes workload. |
| PV (PersistentVolume) | The actual storage resource provisioned for a PVC. |
1.1 What This Guide Covers
- Backing up all Kubernetes resources: Namespaces, Deployments, Services, ConfigMaps, Secrets, CRDs, PVCs, and more.
- Copying persistent volume (PV) data to E2E Object Storage during backup.
- Restoring workloads in the same cluster (rollback / recovery scenarios).
- Performing cross-cluster restores for disaster recovery and cluster migration.
- Configuring backup retention policies and automated cleanup.
PV Backup Behavior: Unlike snapshot-based approaches, Velero with the restic/kopia integration copies actual PV data directly to E2E Object Storage. On restore, a new PV is provisioned and the backed-up data is written to it. No volume snapshots are involved in this workflow.
1.2 Why Velero + E2E Object Storage?
| Benefit | Details |
|---|---|
| Kubernetes-native | Velero is purpose-built for Kubernetes, backed by the CNCF ecosystem. |
| S3-compatible | E2E Object Storage implements the S3 API; Velero integrates natively with no extra plugins. |
| Full PV backup | Persistent volume data is copied to object storage — no dependency on volume snapshots. |
| Automated scheduling | Built-in cron scheduling for daily/weekly backups with configurable TTL-based retention. |
| Disaster recovery | Supports cross-cluster restore, enabling recovery to a completely new cluster. |
| Cost-efficient | Pay only for object storage consumed; no snapshot infrastructure required. |
2. Architecture Overview
The backup solution consists of three integrated layers. Understanding data flow between these layers is essential before installation.
2.1 How It Works
When a backup is triggered, Velero performs two parallel operations:
- Resource Backup — All Kubernetes API objects (Deployments, Services, ConfigMaps, Secrets, PVCs, etc.) are serialized into compressed JSON archives and uploaded to your E2E Object Storage bucket.
- PV Data Backup — Using its built-in file-based backup agent (restic/kopia), Velero reads data directly from each PVC's mounted filesystem and uploads it to E2E Object Storage.
On restore, Velero reconstructs all Kubernetes resources and provisions new PVs, then writes the backed-up data back to the new volumes — making applications fully operational.
2.2 What Gets Stored Where
| Data Type | Storage Location | Notes |
|---|---|---|
| Kubernetes manifests (Deployments, Services, etc.) | E2E Object Storage | Compressed JSON archives |
| PVC data (application files, databases, etc.) | E2E Object Storage | Uploaded by restic/kopia agent |
| Backup metadata & index | E2E Object Storage | Used by Velero for restore operations |
3. Prerequisites
Ensure all of the following are in place before beginning installation. Skipping prerequisites is the most common cause of backup failures.
3.1 E2E Cloud Infrastructure
- An active E2E Kubernetes cluster with at least one node group and workloads using PVCs.
- An E2E Object Storage bucket dedicated to Velero backups. Note the endpoint URL, bucket name, access key, and secret key from the E2E Cloud Console.
- Network connectivity from the E2E Kubernetes cluster to the E2E Object Storage endpoint.
3.2 Tooling on Your Local Machine
veleroCLI installed (v1.9 or later recommended). Download from https://github.com/vmware-tanzu/velero/releaseskubectlconfigured to communicate with your E2E Kubernetes cluster.- Sufficient RBAC permissions — ClusterAdmin role or equivalent — to install CRDs and create namespaces.
3.3 Software Version Reference
| Component | Recommended Version |
|---|---|
| Velero CLI & Server | v1.9 or later |
| Kubernetes | v1.20 or later |
| E2E Object Storage | MinIO-compatible S3 API |
4. Installation and Configuration
Follow the steps in this section in the order presented. Each step builds on the previous one.
Step 1 — Create E2E Object Storage Credentials File
Velero needs credentials to authenticate with E2E Object Storage. Create a plain-text credentials file as shown below. You can find your access key and secret key in the E2E Cloud Console under Object Storage > Access Keys.
# File: credentials-velero
# Obtain these values from the E2E Cloud Console
[default]
aws_access_key_id = <YOUR_E2E_OBJECT_STORAGE_ACCESS_KEY>
aws_secret_access_key = <YOUR_E2E_OBJECT_STORAGE_SECRET_KEY>
Keep this file secure. It will be converted into a Kubernetes Secret during Velero installation. Do not commit it to version control.
Step 2 — Install Velero with E2E Object Storage Backend
Run the following command from your local machine. Replace placeholder values with your actual E2E Object Storage endpoint and bucket name.
velero install \
--provider aws \
--bucket <e2e-object-storage-bucket-name> \
--secret-file ./credentials-velero \
--use-node-agent \
--backup-location-config \
region=e2e-default,\
s3ForcePathStyle="true",\
s3Url=https://objectstore.e2enetworks.net
Parameter explanation:
| Parameter | Description |
|---|---|
--provider aws | Velero uses the S3-compatible API. E2E Object Storage implements this same API. |
--bucket | Name of the bucket you created in E2E Object Storage for Velero. |
--secret-file | Path to the credentials file created in Step 1. |
--use-node-agent | Enables the restic/kopia node agent for PV data backup. Required for backing up PVC data to object storage. |
--backup-location-config | Passes the E2E Object Storage endpoint URL and forces path-style access (required for MinIO-compatible stores). |
Velero will be installed in the velero namespace. Verify its pods are running before proceeding:
kubectl get pods -n velero
Step 3 — Annotate PVCs for Backup
Because PV backup uses the node agent, each PVC that should have its data backed up must be annotated. Without this annotation, Velero will back up the PVC manifest but not the actual data inside the volume.
Annotate a specific PVC:
kubectl annotate pvc <pvc-name> \
backup.velero.io/backup-volumes=<volume-name> \
-n <namespace>
Annotate all PVCs on a Pod (run for each pod that mounts PVCs):
kubectl annotate pod/<pod-name> \
backup.velero.io/backup-volumes=<comma-separated-volume-names> \
-n <namespace>
The volume name in the annotation refers to the volume name defined in the Pod spec (under spec.volumes[].name), not the PVC name.
5. Backing Up Kubernetes Workloads
With Velero configured, you can create on-demand backups or set up automated schedules.
5.1 On-Demand Backup
To immediately back up a specific namespace including all annotated PVCs:
velero backup create my-backup \
--include-namespaces my-namespace \
--default-volumes-to-fs-backup
For a full cluster-wide backup (all namespaces), omit the --include-namespaces flag:
velero backup create full-cluster-backup \
--default-volumes-to-fs-backup
--default-volumes-to-fs-backup instructs Velero to back up all volumes using the file-system backup agent, even if not individually annotated. Use this flag or individual PVC annotations — not both.
5.2 Backing Up Statically Provisioned PVCs (without Storage Class)
Static PVCs without a storage class cannot currently be backed up by Velero's file-system backup agent, as a storage class is required to provision the new PV during restore. Assign a storage class to the PVC before taking a backup. Contact E2E Networks support for assistance.
5.3 Scheduled Backups (Recommended for Production)
Automate backups using a cron schedule. The example below creates a daily backup at 2:00 AM with a 7-day retention period:
velero schedule create daily-backup \
--schedule="0 2 * * *" \
--include-namespaces production,staging \
--default-volumes-to-fs-backup \
--ttl 168h0m0s
The --ttl (time-to-live) flag controls how long each backup is retained. When a backup exceeds its TTL, Velero automatically deletes it from E2E Object Storage.
Common TTL values:
| TTL Value | Retention Period |
|---|---|
168h0m0s | 7 days |
720h0m0s | 30 days |
2160h0m0s | 90 days |
5.4 Verifying Backup Status
# List all backups
velero backup get
# Detailed report — shows resources, PV data, warnings
velero backup describe my-backup --details
# View full backup log
velero backup logs my-backup
6. Restoring from Backup
Velero supports two restore scenarios. The correct procedure depends on whether you are restoring within the same cluster or migrating to a different cluster.
6.1 Restore Scenario Comparison
| Scenario | Use Case | PV Restore Method |
|---|---|---|
| Same-Cluster Restore | Rollback after accidental deletion or data corruption | New PV provisioned; data copied from E2E Object Storage |
| Cross-Cluster Restore | Disaster recovery; migration to a new cluster | New PV provisioned in target cluster; data copied from E2E Object Storage |
6.2 Same-Cluster Restore
Use this procedure when restoring to the same E2E Kubernetes cluster where the backup was originally created.
velero restore create --from-backup my-backup
To restore to a different namespace than the original:
velero restore create --from-backup my-backup \
--namespace-mappings original-namespace:restored-namespace
During a same-cluster restore, Velero:
- Reads all Kubernetes manifests from E2E Object Storage and recreates them in the cluster.
- Recreates Namespaces, Deployments, Services, ConfigMaps, Secrets, and PVCs.
- Provisions a new PV for each PVC using the storage class specified in the PVC manifest.
- Copies backed-up PV data from E2E Object Storage into the new PVs.
- Binds the restored PVCs to the new PVs and allows application pods to start normally.
6.3 Cross-Cluster Restore (Disaster Recovery)
Use this procedure when restoring to a different E2E Kubernetes cluster — for example, recovering from a total cluster failure or migrating workloads to a new cluster.
Before running the restore, ensure:
- The target cluster has Velero installed with the same E2E Object Storage bucket configured as the backup location.
- The target cluster has the same storage class names as the source cluster, or you perform namespace/storage class mapping during restore.
Point your kubectl context to the target cluster, then run:
velero restore create --from-backup my-backup
Velero will recreate all Kubernetes resources in the target cluster, provision new PVs using the configured storage class, and restore PV data from E2E Object Storage.
6.4 Verifying Restore Status
# List all restores
velero restore get
# Detailed restore report
velero restore describe <restore-name> --details
# View restore log
velero restore logs <restore-name>
7. Backup Retention and Cleanup
Without a retention policy, backups accumulate indefinitely, consuming E2E Object Storage capacity. Velero provides TTL-based automatic retention and supports manual deletion.
7.1 TTL-Based Automatic Retention
Set the --ttl flag when creating backups or schedules. Once a backup exceeds its TTL, Velero's garbage collection automatically deletes it from E2E Object Storage.
# Backup with 30-day retention
velero backup create monthly-snapshot \
--include-namespaces production \
--default-volumes-to-fs-backup \
--ttl 720h0m0s
# Update TTL on an existing backup
kubectl patch backup monthly-snapshot \
-n velero \
--type merge \
-p '{"spec":{"ttl":"720h0m0s"}}'
7.2 Manual Deletion
# Delete a named backup
velero backup delete my-backup
# Delete all backups matching a label
velero backup delete --selector environment=staging
# Confirm deletion
velero backup get
Ensure you have at least one valid restore point before deleting any backup.
8. Quick Command Reference
| Command | Description |
|---|---|
velero backup get | List all backups and their status |
velero backup describe <name> --details | Detailed information about a backup |
velero backup logs <name> | Full log output for a backup operation |
velero backup delete <name> | Delete a backup and its object storage data |
velero restore get | List all restores and their status |
velero restore describe <name> --details | Detailed information about a restore |
velero restore logs <name> | Full log output for a restore operation |
velero schedule get | List all configured backup schedules |
velero schedule delete <name> | Delete a backup schedule |
9. Important Operational Notes
9.1 Storage Behaviour
- All restores occur within an existing target E2E Kubernetes cluster. Velero does not create a new cluster.
- PV data is fully copied to E2E Object Storage — restores provision entirely new PVs and populate them from object storage.
- E2E Object Storage usage reflects the total size of Kubernetes manifests plus all backed-up PV data.
- Multiple restore points should be maintained according to your organisation's recovery point objective (RPO).
9.2 Security Best Practices
- Store E2E Object Storage credentials (
credentials-velero) securely. Rotate access keys periodically via the E2E Cloud Console. - Restrict access to the
veleroKubernetes namespace using RBAC. Only authorised platform engineers should be able to create or delete backups. - Consider enabling object versioning and object locking on your E2E Object Storage bucket for immutable backup storage in compliance-sensitive environments.
9.3 Performance Considerations
- Schedule backups during off-peak hours to minimise the impact of PV data transfers on application I/O performance.
- For large volumes, the initial backup may take significant time. Subsequent backups are incremental — only changed blocks are transferred.
- Restore time depends on the total size of PV data to be transferred from E2E Object Storage.
10. Troubleshooting
| Symptom | Likely Cause | Resolution |
|---|---|---|
| Backup status: Failed | Incorrect E2E Object Storage credentials or endpoint | Re-verify the credentials-velero file and the s3Url in the backup location config. |
| PVC data not included in backup | PVC annotation missing or --default-volumes-to-fs-backup not set | Annotate the PVC/Pod or use --default-volumes-to-fs-backup flag. |
| Restore shows Partially Failed | Target storage class missing in destination cluster | Ensure the target cluster has the same storage class names, or use --storage-class-mappings. |
| Velero pods not starting | Missing RBAC or namespace permissions | Verify ClusterAdmin role is assigned and the velero namespace was created successfully. |
| Backup of static PVC fails | No storage class assigned to static PVC | Assign a valid storage class to the PVC before taking a backup. |
| Restore PVC remains Pending | Target cluster has no available storage provisioner | Confirm the Ceph CSI or other CSI driver is active in the target cluster. |
11. Summary
This guide has covered the complete end-to-end workflow for Kubernetes backup and restore on E2E Cloud — from architecture and prerequisites, through installation and scheduled automation, to same-cluster recovery and cross-cluster disaster recovery.
By combining Velero's Kubernetes-native orchestration with E2E Object Storage as both the metadata and PV data backend, E2E Cloud customers can achieve:
- Recovery Point Objectives (RPOs) measured in hours with automated scheduled backups.
- Recovery Time Objectives (RTOs) proportional to PV data size — new volumes are provisioned and populated from object storage on each restore.
- True cross-cluster disaster recovery without dependency on any external infrastructure.
- Full backup coverage — both Kubernetes resource definitions and persistent volume data are protected.