Skip to main content

Run a restore drill

Inspect 1.36

Restore a backup today, into a namespace nobody is using, and watch the pods come up. That is the whole drill. A backup you have never restored is a guess, and the moment you need it is the worst moment to find out it was wrong. Rehearsing turns the guess into a fact.

Restore into a scratch namespace #

Velero can rewrite the target namespace as it restores, so you bring a backup back beside the live workload without touching it. Map the source namespace to a scratch one with --namespace-mappings:

		$ velero restore create app-drill \
    --from-backup app-backup \
    --namespace-mappings my-app:my-app-drill
$ velero restore describe app-drill --details

This recreates every resource from app-backup inside my-app-drill. The live my-app namespace keeps running, untouched.

Note

Velero skips resources that already exist and fills the gaps. It never overwrites a running resource. In a scratch namespace nothing exists yet, so the restore recreates everything from the backup.

Verify pods and data #

A restore that reports success has only copied objects. You still have to confirm the workload runs and the data came with it:

		$ kubectl -n my-app-drill get pods
$ kubectl -n my-app-drill exec deploy/<app> -- ls /data

Check that pods reach Running, that PVCs are Bound, and that the files or records you expect are actually inside the volume. A green restore with an empty volume is a failed drill.

Tear down #

Once the drill passes, delete the scratch namespace so it does not linger and drift. The live workload was never touched, so nothing else needs cleanup:

		$ kubectl delete namespace my-app-drill

Databases restore differently#

Velero brings back a database's Kubernetes objects and its volume files, but that volume copy is . To rehearse a real database recovery, bootstrap a fresh instance in the scratch namespace and load it from the database's own archive, the dump or operator backup you stored in object storage. The exact steps belong to your database and its operator.

Restore to a different cluster#

The strongest drill restores to a second cluster, because a real disaster can take the whole original cluster with it. Point a velero client at another cluster that reads the same bucket, then run the restore there:

		$ velero restore create app-dr --from-backup app-backup
$ velero restore describe app-dr --details

If the backup lives in object storage outside the cluster, any cluster with credentials to that bucket can restore it. That is what makes recovery from a full loss possible.

Make it a habit#

A drill proves recovery only for the setup you tested. Run it again whenever that setup could have drifted:

  • Every quarter, as a standing rehearsal.
  • After any change to the backup configuration, the storage class, or the cluster definition.
  • After a Kubernetes version upgrade.

Write down what you ran and what you checked. Turn the working steps into a runbook so the person doing the real restore at 3 a.m. is following a tested script, not improvising.

The restore commands here are the same ones a full rebuild uses. See for the backup side, and grow this drill into the full rehearsal in .

Volume snapshots and their limits

Why Syself Autopilot backs up PVC data by file copy instead of CSI volume snapshots, and what that means for your data.

Offsite and immutable backups

Keep a copy of your Syself Autopilot backups in a second region and lock it with object-lock so ransomware cannot delete it.