Data survival across reprovision

Syself Autopilot reprovisions nodes as a matter of routine. Upgrades, scaling, and self-healing all rebuild a node from scratch rather than patching it in place. If you keep a database on local disk, the obvious worry is that a rebuild takes your data with it. It does not. To see why, look at what a reprovision actually replaces and what it leaves alone.

Two disks with two different fates#

A bare metal server has an OS disk and one or more data disks, and a reprovision treats them in opposite ways.

The OS disk holds a sealed, immutable image. Syself Autopilot never edits it: on every reprovision it throws the old image away and writes a fresh one in its place. That is what keeps nodes identical, with no accumulated drift. The tradeoff is that nothing you write to the OS disk survives.

The data disks are where your local volumes live, in the LVM volume groups you prepared. A reprovision does not touch them. The thin pools, the logical volumes, and the bytes inside them stay exactly as they were. When the rebuilt node comes back, TopoLVM rescans the same volume groups and the same volumes reattach.

flowchart TB
  subgraph before["Same server, before reprovision"]
    os1["OS disk<br/>sealed image"]
    data1["Data disks<br/>your local volumes"]
  end
  subgraph after["Same server, after reprovision"]
    os2["OS disk<br/>fresh image, rewritten"]
    data2["Data disks<br/>untouched, volumes intact"]
  end
  os1 -->|"replaced"| os2
  data1 -->|"left alone"| data2

The diagram shows the split: the OS disk is swapped for a new image, the data disks pass through the reprovision unchanged.

What survival depends on#

Local data survives because it stays on the same physical server, so the one thing that breaks it is the workload moving to a different server. Local storage does not follow a pod. The volume is a set of blocks on the disks of one machine, and it stays there. That is why a bound local volume pins its pod: Kubernetes gives the pod the volume's node affinity, so the pod can only schedule back to the server that holds the data. A pod that lands anywhere else, or a fresh pod after a move, starts with an empty volume group, because the data never left the old server.

The boundary comes down to what happens to the server:

• Same server reprovisions. The workload lands back on the same machine, the data disks were never touched, and the volume reattaches with its data. This is the normal upgrade and self-healing path for a healthy server.
• Workload moves to another server. The scheduler places the pod on a different machine, perhaps because the original is busy or cordoned. The new server has its own empty volume group, so the pod starts with empty storage. The old data still exists on the old server, but the workload is no longer there to read it.
• Hardware is replaced. A failed server is swapped for different hardware. The old disks are gone, and with them the data. The replacement starts empty.

That last case has a second edge: a decommissioned or returned disk still holds your data, readable by whoever ends up with it, unless you encrypted it. Encrypt data at rest with LUKS closes that gap, so a disk that leaves your control gives up nothing.

You still need a backup, and replication on top#

The single local copy survives a reprovision, but not a failed drive, a decommissioned server, or a workload rescheduled elsewhere. Two things cover that, in this order.

First, keep an off-server backup. This is the floor, and nothing substitutes for it: it is the copy that survives losing the whole server, the data disk, or the site, and it is what you rebuild from when every live copy is gone. Disaster recovery for stateful data covers where the backup lives and how to restore it.

Then replicate, so a lost server costs seconds of availability instead of a restore. Running a production database as replicas across servers is standard practice, and a Kubernetes operator does it for you: CloudNativePG keeps PostgreSQL copies on different servers and fails over on its own, so losing a machine just means serving from another copy.

How many servers you need depends on how the database picks a new leader.

• Primary and replica, such as CloudNativePG. The copies never vote. The operator promotes the standby through the Kubernetes control plane, so two of them never both claim the role, and two servers are enough to fail over.
• Majority based, such as MySQL group replication or Galera. A majority of members must stay alive through a failure, so the count has to be odd: three or more.

We recommend three bare metal servers either way. That satisfies the majority case and leaves room to run both kinds on the same hardware.

Related: local PV lifecycle for how a claim becomes a thin volume and how to reclaim its space, bare metal and cloud for the immutable-image design that makes reprovisions clean, and bare metal for why to run Kubernetes on it.