Keep stateful pods safe across node replacement

Syself Autopilot replaces nodes on every upgrade and repair, and a stateful pod feels that differently depending on where its data lives. The rule to internalize: a local volume does not survive its node. Place replicas so that losing one node, and its local data, never costs you the workload.

What replacement does to each storage kind#

Only one storage kind makes node replacement a design problem. Local NVMe (TopoLVM) is pinned to a physical node, so when the node goes, the volume goes with it and the pod comes back on the replacement with a fresh, empty disk. Hetzner Cloud volumes detach and reattach; JuiceFS lives on object storage and simply follows the pod. Both of those move their data for free. Run a StatefulSet with storage walks through all three in full.

So the whole design question is about local volumes: replacement means one replica rebuilds from empty, and your job is to guarantee that never happens to a majority at once.

Place replicas so a rebuild never breaks quorum#

For a quorum-based application such as a three-member database, an etcd cluster, or a raft group, two settings turn a node replacement into a non-event:

• One replica per node, with required pod anti-affinity on kubernetes.io/hostname, so no two members share a machine. See Affinity and anti-affinity .
• A PodDisruptionBudget sized for quorum, maxUnavailable: 1 for three members, so a drain evicts one member and blocks until it rejoins before touching the next. See Add a PodDisruptionBudget .

Put together, a single replacement can only ever take one member down. It rebuilds from the surviving two, and quorum holds the entire time:

flowchart LR
  A[Node marked for<br/>replacement] --> B[PDB admits<br/>one eviction]
  B --> C[Pod reschedules<br/>to new node]
  C --> D[Fresh empty<br/>local volume]
  D --> E[Resync from<br/>two live peers]
  E --> F[Member rejoins<br/>quorum]
  F --> G[Next node<br/>now eligible]

The guarantee is the ordering: no second node becomes eligible until the first member is back.

Draining a stateful node on purpose#

When you take a stateful node out yourself for hardware maintenance or a manual move, drain it rather than deleting the pod: kubectl drain <node> respects the PodDisruptionBudget, evicts one member, and waits. Let it rebuild and rejoin before you drain the next.

One timing edge is worth knowing. The platform's own upgrade drain waits nodeDrainTimeoutSeconds for a node to empty, 180 seconds per pool by default, overridable per pool under spec.topology.workers.machineDeployments[].deletion.nodeDrainTimeoutSeconds. If a member rebuilds slower than that window, the drain can move on before it has rejoined, so a large local dataset is a reason to raise the timeout on that pool.

Restore from a replica, or from backup#

A rebuilt member resyncs from a healthy peer, which the database's own replication handles as long as quorum survived. If you lose more than quorum, say two of three replaced at once, the fallback is a restore from backup . Replication protects you against one node; backups protect you against the mistake that takes several.