Keep stateful pods safe across node replacement

Syself Autopilot replaces nodes on every upgrade and repair, and the effect on a stateful pod depends on where its data lives. An upgrade reinstalls the OS disk and leaves the data disk untouched, so a local volume survives it. A hardware repair that replaces the machine or a failed disk does not preserve local data. Place replicas so that losing one node's local data never takes down the workload.

What replacement does to each storage kind

Only local NVMe (TopoLVM) makes node replacement a design problem. The volume is pinned to a physical node's data disk. An upgrade reinstalls only the OS disk, so the data disk and its volume survive the replacement. A hardware repair that replaces the machine or a failed disk takes the data disk with it, and the replica comes back on a fresh, empty volume. Hetzner Cloud volumes detach and reattach; JuiceFS lives on object storage and simply follows the pod, so both move their data automatically on any replacement. Run a StatefulSet with storage walks through all three in full.

The design problem is therefore scoped to local volumes on a hardware repair or replacement, where a replica comes back on an empty disk and must rebuild. The objective is to ensure this never affects a majority of replicas simultaneously.

Place replicas so a rebuild never breaks quorum

For a quorum-based application such as a three-member database, an etcd cluster, or a raft group, two settings protect your workloads from node-replacement:

• One replica per node, with required pod anti-affinity on kubernetes.io/hostname, so no two members share a machine. See Affinity and anti-affinity .
• A PodDisruptionBudget sized for quorum, maxUnavailable: 1 for three members, so a drain evicts one member and blocks until it rejoins before touching the next. See Add a PodDisruptionBudget .

Put together, a single replacement can only ever take one member down. When a repair loses that member's local data, it rebuilds from the surviving two, and quorum holds the entire time:

flowchart LR
  A[Node marked for<br/>replacement] --> B[PDB admits<br/>one eviction]
  B --> C[Pod reschedules<br/>to new node]
  C --> D[Fresh empty<br/>local volume]
  D --> E[Resync from<br/>two live peers]
  E --> F[Member rejoins<br/>quorum]
  F --> G[Next node<br/>now eligible]

The guarantee is the ordering: no second node becomes eligible until the first member is back.

Draining a stateful node on purpose

When you take a stateful node out yourself for hardware maintenance or a manual move, drain it rather than deleting the pod: kubectl drain <node> respects the PodDisruptionBudget, evicts one member, and waits. Let it rebuild and rejoin before you drain the next.

The platform's upgrade drain waits nodeDrainTimeoutSeconds for a node to empty. That timeout is set per worker pool in the Cluster object. Once it elapses, the drain proceeds. If a member rebuilds slower than that window, the drain can move on before it has rejoined, so a large local dataset is a reason to raise nodeDrainTimeoutSeconds for that pool. See Run a production-ready workload for how to set it in the Cluster topology.

Restore from a replica, or from backup

A rebuilt member resyncs from a healthy peer, which the database's own replication handles as long as quorum survived. If you lose more than quorum, say two of three replaced at once, the fallback is a restore from backup . Replication protects you against one node; backups protect you against the mistake that takes several.