Syself Autopilot replaces nodes on every upgrade and repair, and the effect on a stateful pod depends on where its data lives. An upgrade reinstalls the OS disk and leaves the data disk untouched, so a local volume survives it. A hardware repair that replaces the machine or a failed disk does not preserve local data. Place replicas so that losing one node's local data never takes down the workload.

## What replacement does to each storage kind

Only local NVMe (TopoLVM) makes node replacement a design problem. The volume is pinned to a physical node's data disk. An upgrade reinstalls only the OS disk, so the data disk and its volume survive the replacement. A hardware repair that replaces the machine or a failed disk takes the data disk with it, and the replica comes back on a fresh, empty volume. Hetzner Cloud volumes detach and reattach; JuiceFS lives on object storage and simply follows the pod, so both move their data automatically on any replacement. [Run a StatefulSet with storage](/docs/hetzner/apalla/workloads/stateful/run-a-statefulset-with-storage) walks through all three in full.

The design problem is therefore scoped to local volumes on a hardware repair or replacement, where a replica comes back on an empty disk and must rebuild. The objective is to ensure this never affects a majority of replicas simultaneously.

## Place replicas so a rebuild never breaks quorum

For a quorum-based application such as a three-member database, an etcd cluster, or a raft group, two settings protect your workloads from node-replacement:

- **One replica per node**, with required pod anti-affinity on `kubernetes.io/hostname`, so no two members share a machine. See [Affinity and anti-affinity](/docs/hetzner/apalla/workloads/placement/affinity-and-anti-affinity).
- **A PodDisruptionBudget sized for quorum**, `maxUnavailable: 1` for three members, so a drain evicts one member and blocks until it rejoins before touching the next. See [Add a PodDisruptionBudget](/docs/hetzner/apalla/workloads/production/pod-disruption-budgets).

Put together, a single replacement can only ever take one member down. When a repair loses that member's local data, it rebuilds from the surviving two, and quorum holds the entire time:

```mermaid
flowchart LR
  A["Node marked for<br/>replacement"]:::platform --> B["PDB admits<br/>one eviction"]:::platform
  B --> C["Pod reschedules<br/>to new node"]:::app
  C --> D["Fresh empty<br/>local volume"]:::data
  D --> E["Resync from<br/>two live peers"]:::data
  E --> F["Member rejoins<br/>quorum"]:::app
  F --> G["Next node<br/>now eligible"]:::platform
```

The guarantee is the ordering: no second node becomes eligible until the first member is back.

> [!WARNING]
> Without one-per-node placement, two members can land on the same machine. A single replacement then takes two of three at once, which loses quorum and, on a hardware repair with local storage, loses their data. On a local-storage workload this is the failure to design against.

## Draining a stateful node on purpose

When you take a stateful node out yourself for hardware maintenance or a manual move, drain it rather than deleting the pod: `kubectl drain <node>` respects the PodDisruptionBudget, evicts one member, and waits. Let it rebuild and rejoin before you drain the next.

The platform's upgrade drain waits `nodeDrainTimeoutSeconds` for a node to empty. That timeout is set per worker pool in the Cluster object. Once it elapses, the drain proceeds. If a member rebuilds slower than that window, the drain can move on before it has rejoined, so a large local dataset is a reason to raise `nodeDrainTimeoutSeconds` for that pool. See [Run a production-ready workload](/docs/hetzner/apalla/workloads/production/run-a-production-ready-workload) for how to set it in the `Cluster` topology.

## Restore from a replica, or from backup

A rebuilt member resyncs from a healthy peer, which the database's own replication handles as long as quorum survived. If you lose more than quorum, say two of three replaced at once, the fallback is a restore from [backup](/docs/hetzner/apalla/storage/backup/disaster-recovery-stateful). Replication protects you against one node; backups protect you against the mistake that takes several.