Syself heals your infrastructure but does not back up your data. When a node dies, a controller replaces it and the cluster returns to the shape you declared in Git. You do nothing. But the disk that node was writing to is not part of the cluster's declared shape, so no controller brings it back. That data is yours to protect.

> [!NOTE]
> For step-by-step setup guides on Velero backups and disaster recovery procedures, see [Data protection and DR](/docs/hetzner/apalla/storage/backup/back-up-with-velero).

People often learn this only after a node dies and the data is already gone. It was on that disk, no one wrote down that it was there, and now it cannot be recovered. To avoid this you need to know your state: what exists, where each piece is stored, and what brings it back. Extra backup tools do not give you that. On Syself Autopilot the list of state to track is short.

## The state map

Everything in a cluster is one of two things. You wrote it down, or your application wrote it while running. The first kind is the `Cluster` object, your manifests, your Helm values. This is state you wrote, so you can keep it in version control. If you keep it in Git, which Syself recommends, Git is its backup and you need nothing more; if you do not, it is yours to back up like anything else. The second kind is database rows, uploaded files, and whatever sits in a PersistentVolume. (A PersistentVolume is the disk your app asks for through a PersistentVolumeClaim, or PVC.) It is stored on disks and comes back only from a copy you made.

```mermaid
flowchart LR
    subgraph DECL["Declared state (you wrote it)"]
        CO["Cluster object:<br/>machine types, node count,<br/>region, version"]
        AM["App manifests,<br/>Helm values"]
    end
    subgraph RUN["Runtime state (your apps wrote it)"]
        PG["PostgreSQL data"]
        PVC["Other PVC data"]
        SEC["Secrets only<br/>in Kubernetes"]
    end
    OSD["Node OS disk"]
    CO --> GIT["Git"]
    AM --> GIT
    GIT -->|"re-sync (Argo CD / Flux)"| NEW["Rebuilt cluster"]
    PG -->|"barman-cloud:<br/>WAL stream + full backups"| OBJ["Object storage<br/>(outside the cluster)"]
    PVC -->|"Velero file<br/>system backup"| OBJ
    SEC -->|"Velero, or move<br/>to Sealed Secrets"| OBJ
    OBJ -->|"restore"| NEW
    OSD -->|"rewritten on every reprovision"| GONE["Disposable.<br/>Never backed up."]
```

| Data                                       | Where it lives                   | Recovery path                                                                                                                           |
| ------------------------------------------ | -------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| `Cluster` object, cluster topology         | Git                              | Re-apply; the management cluster provisions a new cluster from it                                                                       |
| Application manifests, Helm values         | Git                              | Re-sync with Argo CD or Flux                                                                                                            |
| Secrets and objects never committed to Git | Only inside the cluster, in etcd | Velero backup, or move them into Git via Sealed Secrets                                                                                 |
| PostgreSQL data                            | PersistentVolume                 | CloudNativePG `barman-cloud` archives in object storage                                                                                 |
| Other PVC data                             | PersistentVolume                 | Velero file system backup, or an application-level export                                                                               |
| Node OS disk                               | The server                       | None. Rewritten on every reprovision from the immutable OS image Syself builds, which is fixed at build time and never patched in place |

Every one of these backup paths lands in object storage outside the cluster, on purpose. Lose the cluster and you must not lose the backup with it.

> [!WARNING]
> Syself operates the management cluster and the node lifecycle. Syself does not back up your PersistentVolume contents, your Kubernetes-only Secrets, or anything you applied by hand outside of Git. Those are yours to protect.

## Declared state: back it up by keeping it in Git

Syself recommends keeping your declared state in Git and driving it with GitOps. This is optional, but if you do it, the backup is already handled: Git holds every version, so you recover by re-applying from Git. If you do not use Git, this state is yours to back up like anything else.

The `Cluster` object describes the whole cluster: machine types, node count, region, Kubernetes version. Lose the workload cluster and you re-apply that object. The management cluster builds a new one to match, the same way it built the first. Argo CD or Flux re-syncs your manifests on top. The cluster comes back exactly as your last commit described it. (The management cluster is ours to run, not yours to back up.)

The same holds for etcd, the database where the control plane keeps every Kubernetes object. Syself does not snapshot etcd for workload clusters, and you do not need it to. You recover the control plane by recreating it from the `Cluster` object in Git and re-syncing, not by restoring a snapshot. That works because everything in etcd that matters should trace back to a file in Git.

This works only for what is actually in Git. A re-sync recovers what is in Git and nothing else. A Deployment someone applied with `kubectl` during an incident, a Secret created live and never committed, a ConfigMap edited in place: the re-sync does not know these exist, so a rebuild does not bring them back. So when you change something outside Git, put it back into Git the same day.

## Runtime state on disks

Storage comes as cloud block volumes and local disks, and they behave differently when a node goes away. A cloud block volume (the Hetzner CSI `standard` class) is network storage: it detaches when the node drains and reattaches wherever the pod lands next, so the data follows the workload. A local disk on bare metal (TopoLVM) is a physical disk on one server: it survives a reprovision of that same server, but it does not follow a pod to another machine, so a database on local storage stays safe by replicating across servers rather than trusting one disk. [The storage model](/docs/hetzner/apalla/concepts/internals/storage) covers this survival behavior in full.

`reclaimPolicy: Retain` is not a backup. Both storage classes use it, so deleting a PVC leaves the underlying volume in place, which saves you from an accidental `kubectl delete pvc`. It does nothing when a pod corrupts its own data or when the volume itself is gone. For anything you cannot recreate, you still need an independent copy in object storage.

## Velero: for Kubernetes-only state and plain files

For Secrets and resources that never made it into Git, and for files an application keeps in a volume and does not back up itself, [Velero](https://velero.io) copies them to your own object storage bucket. It does not ship with the cluster; you install it and point it at your bucket. It copies files while the application keeps writing, which is fine for plain files but can catch a half-written page in a database, so databases get their own path below. [Back up with Velero](/docs/hetzner/apalla/storage/backup/back-up-with-velero) has the steps and how a restore behaves.

## Databases: back up the change log

For a database, back up the change log rather than copy files. CloudNativePG streams the write-ahead log to object storage with `barman-cloud`, with scheduled full backups on top, giving point-in-time recovery to any moment in the retention window. Do not confuse this with replication: replication across instances covers a dead node or disk, while a backup covers a bad `DELETE` or a broken migration that replication copies to every instance in the same instant. You need both. [Run databases](/docs/hetzner/apalla/workloads/stateful/run-databases) sets this up end to end.

## Secrets and etcd

Secrets are the usual hole in a Git-based plan, because you should not commit them in the clear. Close it by encrypting them into Git with [Sealed Secrets](https://github.com/bitnami-labs/sealed-secrets) or pointing at an external store with [External Secrets Operator](https://external-secrets.io), so the secret becomes declared state and comes back on a re-sync; for Secrets you keep Kubernetes-only, Velero is the fallback.

etcd holds Secrets unencrypted by default. Encryption at rest is available but opt-in, set through your `Cluster` object and rolled out across the control plane one machine at a time; the [encrypt etcd](/docs/hetzner/apalla/security/encrypt-etcd) guide has the steps and the ordering to get right.

## Recovering a cluster

You already know what state exists and where each piece is stored, so you do not have to diagnose a destroyed cluster under pressure. You re-apply the `Cluster` object, re-sync with Argo CD or Flux, and run the restores you rehearsed. Declared state comes back from Git, database state from its own archive, and plain files from Velero. The OS disk needs no recovery, because the platform rewrites it on every reprovision. Anything that fits no row in the table above is your gap, and you can find it now instead of during an outage.

Rehearse the restores before you need them. A restore you have never run might not work. Restore a Velero backup into a scratch namespace, kept apart from the real one. Recover a CloudNativePG cluster from its archive into a throwaway cluster. Do this quarterly, and once more after any change to the backup configuration.

For the exact commands, see [back up and restore](/docs/hetzner/apalla/storage/backup/back-up-with-velero) for Velero and the GitOps recovery path, and [run databases](/docs/hetzner/apalla/workloads/stateful/run-databases) for CloudNativePG end to end. For why the OS disk needs no backup, see [self-healing and node replacement](/docs/hetzner/apalla/concepts/operations/self-healing-and-node-replacement); for how cloud volumes and local disks each survive a node loss, [the storage model](/docs/hetzner/apalla/concepts/internals/storage); and for why the `Cluster` object in Git is a recovery source and not just config, [declarative cluster management](/docs/hetzner/apalla/concepts/foundations/declarative-cluster-management).