Syself Autopilot replaces nodes on every upgrade and repair, and a stateful pod feels that differently depending on where its data lives. The rule to internalize: a local volume does not survive its node. Place replicas so that losing one node, and its local data, never costs you the workload.

## What replacement does to each storage kind

Only one storage kind makes node replacement a design problem. Local NVMe (TopoLVM) is pinned to a physical node, so when the node goes, the volume goes with it and the pod comes back on the replacement with a fresh, empty disk. Hetzner Cloud volumes detach and reattach; JuiceFS lives on object storage and simply follows the pod. Both of those move their data for free. [Run a StatefulSet with storage](/docs/hetzner/apalla/workloads/stateful/run-a-statefulset-with-storage) walks through all three in full.

So the whole design question is about local volumes: replacement means one replica rebuilds from empty, and your job is to guarantee that never happens to a majority at once.

## Place replicas so a rebuild never breaks quorum

For a quorum-based application such as a three-member database, an etcd cluster, or a raft group, two settings turn a node replacement into a non-event:

- **One replica per node**, with required pod anti-affinity on `kubernetes.io/hostname`, so no two members share a machine. See [Affinity and anti-affinity](/docs/hetzner/apalla/workloads/placement/affinity-and-anti-affinity).
- **A PodDisruptionBudget sized for quorum**, `maxUnavailable: 1` for three members, so a drain evicts one member and blocks until it rejoins before touching the next. See [Add a PodDisruptionBudget](/docs/hetzner/apalla/workloads/production/pod-disruption-budgets).

Put together, a single replacement can only ever take one member down. It rebuilds from the surviving two, and quorum holds the entire time:

```mermaid
flowchart LR
  A[Node marked for<br/>replacement] --> B[PDB admits<br/>one eviction]
  B --> C[Pod reschedules<br/>to new node]
  C --> D[Fresh empty<br/>local volume]
  D --> E[Resync from<br/>two live peers]
  E --> F[Member rejoins<br/>quorum]
  F --> G[Next node<br/>now eligible]
```

The guarantee is the ordering: no second node becomes eligible until the first member is back.

> [!WARNING]
> Without one-per-node placement, two members can land on the same machine. A single replacement then takes two of three at once, which loses quorum and, with local storage, loses their data. On a local-storage workload this is the failure to design against.

## Draining a stateful node on purpose

When you take a stateful node out yourself for hardware maintenance or a manual move, drain it rather than deleting the pod: `kubectl drain <node>` respects the PodDisruptionBudget, evicts one member, and waits. Let it rebuild and rejoin before you drain the next.

One timing edge is worth knowing. The platform's own upgrade drain waits `nodeDrainTimeoutSeconds` for a node to empty, 180 seconds per pool by default, overridable per pool under `spec.topology.workers.machineDeployments[].deletion.nodeDrainTimeoutSeconds`. If a member rebuilds slower than that window, the drain can move on before it has rejoined, so a large local dataset is a reason to raise the timeout on that pool.

## Restore from a replica, or from backup

A rebuilt member resyncs from a healthy peer, which the database's own replication handles as long as quorum survived. If you lose more than quorum, say two of three replaced at once, the fallback is a restore from [backup](/docs/hetzner/apalla/storage/backup/disaster-recovery-stateful). Replication protects you against one node; backups protect you against the mistake that takes several.