Disaster recovery for stateful data

Disaster recovery on Syself Autopilot leans on one fact: the cluster is declarative, so most of it rebuilds from Git. What is left to recover by hand is the runtime data, and the order you recover it in decides how fast you are back. This is the plan for a full loss. Syself Consulting can help you plan disaster recovery, or run the whole recovery for you as a service.

The order of operations#

During a real outage, work top to bottom and do not skip ahead. The order is not a preference, it is a dependency chain: workloads need a cluster, PVC data needs the workloads that mount it, and a database cannot replay until its volume is back. Treat each numbered item as a checklist entry, and confirm its checkpoint before you start the next.

1. Rebuild the cluster from the Cluster object. The manifest that defines the control plane and node pools lives in Git. Reapply it to a management cluster and Syself Autopilot provisions a fresh cluster. Checkpoint: the new nodes report Ready and the API server answers.
2. Re-sync workloads from Git. Point Argo CD or Flux at the same repository. The Kubernetes resources come back as they were declared. Checkpoint: the applications reconcile and their pods schedule, even while they still wait on data.
3. Reconnect apps to their object-storage data. Uploads, artifacts, and datasets already live in a bucket that survived the cluster, so there is nothing to restore: re-point each app at the same bucket and its data is there. The one exception is a filesystem app with no native backup, whose PVC files you bring back from Velero as a fallback. See Back up cluster state with Velero . Checkpoint: apps read their objects from the bucket, and any Velero-restored PVCs are Bound with their files present.
4. Recover databases from their own backups. Restore each database from its native backup in object storage, then roll forward to the latest point in time the archive supports. Checkpoint: the database accepts connections and its own consistency check passes.
5. Verify, then send traffic. Confirm pods are healthy and data is present, then point users at the cluster. This is the only step that puts the cluster under real load, so it comes last.

flowchart TD
    A[Rebuild cluster from the Cluster object] --> B[Re-sync workloads from Git]
    B --> C[Reconnect apps to object storage]
    C --> D[Database point-in-time recovery]
    D --> E["Verify, then send traffic"]

Restore PVC data from Velero (fallback)#

This step applies only to a filesystem app with no native backup; data already in object storage needs no restore, you re-point the app at the bucket. Velero restores into the cluster and skips resources that already exist, so it fills the gaps after a re-sync rather than fighting it. Run a restore drill has the velero restore create commands and shows how to rehearse them in a scratch namespace first.

Point-in-time recovery for databases#

A database restore is not a file copy. Bootstrap a new instance from the last full backup, then replay the write-ahead log or transaction archive stored alongside it to reach a chosen moment. This is how you recover to just before a bad migration instead of losing everything since the last full backup. The exact steps belong to your database and its operator.

Know your RPO and RTO per data class#

Two numbers set your targets. RPO (recovery point objective) is how much data you can afford to lose, which equals how often you back up. RTO (recovery time objective) is how long recovery can take. They differ by data class:

Set the backup frequency of each class to meet its RPO, and size recovery so the slowest class still fits your RTO.

Test the whole chain#

A recovery plan you have never run is a guess. The restore drill owns the cadence and how to rehearse a restore. The DR-specific test is to grow it into the whole chain end to end, from cluster rebuild through database replay, not just a single restore.