Backup layers and what needs protecting

Your cluster holds two kinds of data, and treating them the same is how backups fail. Declared state is everything written down: the manifests in Git and the Cluster object that defines the cluster. Runtime state is what the running system produced: database rows, user uploads, and anything a pod wrote to a volume. Each needs a different backup.

flowchart TD
    C[Cluster data] --> D[Declared state]
    C --> R[Runtime state]
    D --> G["Manifests and Cluster object in Git"]
    G --> RS[Recover by re-sync]
    R --> BK["Database rows, uploads, volume data"]
    BK --> BN[Needs its own backup]

Declared state: Git already holds it#

When you manage workloads with GitOps, the Kubernetes manifests already live in version control. Recreating them is a re-sync, not a restore. Argo CD or Flux reapply the manifests from the repository and the resources come back.

The cluster itself is declarative in the same way. The Cluster object that defines your control plane and node pools is a manifest. Kept in Git, it rebuilds the cluster from scratch. So declared state rarely needs its own backup. Recovery is: recreate the cluster from the Cluster object, reconnect infrastructure, re-sync workloads from Git.

Runtime state: this is what you back up#

Manifests are reproducible. The data a workload wrote is not. The backups that matter are:

• Databases
• User uploads
• AI and machine learning datasets
• Secrets generated in the cluster that are not stored in Git

Everything in that list lives in a PVC or in the cluster's own state, not in a repository, so a re-sync will not bring it back.

How to capture runtime state#

Start from the workload, not the volume. Most runtime data is already covered by something that backs it up natively, and only a narrow slice needs a generic volume backup.

• A database backs itself up. Run it under an operator such as CloudNativePG, and it archives its write-ahead log to object storage and supports point-in-time recovery. For anything transactional, that is the backup: application-consistent, restored by the database's own tooling. Most of your real data lives here.
• Files belong in object storage. Uploads, artifacts, and datasets written to an S3 bucket are durable at that layer, with no volume to back up. Have the app write to object storage directly where you can.
• Volume file-copy is the fallback. When an application keeps data in a filesystem volume and has no backup of its own, Velero copies the files in the volume out to object storage. Few workloads need this: with databases backing themselves up and files in object storage, it covers a narrow case. Velero also captures the surrounding Kubernetes resources and any non-Git Secrets.

See Volume snapshots and their limits for why a snapshot is not one of these options here.

Every backup leaves the cluster#

A backup stored on the cluster it protects is not a backup. Send every copy to object storage outside the cluster, and for anything you cannot lose, to a second region with immutability .

What to back up#

Once the layers are backed up, rehearse getting them back.

The Syself Consulting team can design a backup and recovery strategy with you, or run the whole thing end to end if you would rather have it handled.

Related: Run a restore drill and Disaster recovery for stateful data .