Skip to main content

Keep databases available

Inspect 1.36

A database has state, so an upgrade has to move both the pod and its data when the node it runs on is replaced. Whether the database stays reachable during that move depends on two choices: where the data lives, and whether you run one instance or a replicated set. For setting a database up in the first place, see .

The data has to survive the node move#

The demo pattern of a single pod with an emptyDir volume loses everything the moment the pod moves. A database needs a persistent volume so its data outlives the pod and the node:

  • Hetzner Cloud volumes are network storage. When the pod moves to a new node, the volume detaches from the old node and reattaches on the new one. The data survives; the detach-and-attach adds to the gap. See .
  • Local storage on bare metal (TopoLVM) keeps the data disk when Syself reprovisions the server, so the data is still there when the server comes back. It is node-bound, so the pod waits for that same server to finish reprovisioning rather than moving elsewhere. See .

Either way, use a StatefulSet with a volumeClaimTemplate, not an emptyDir, so the claim follows the database.

A single instance has a short gap#

One database pod, however well its storage is set up, is unavailable while its node is replaced. The pod stops, the volume detaches and reattaches (or the bare-metal server reprovisions), the pod starts, and the database runs its recovery. That is seconds to tens of seconds on a cloud volume, longer on a bare-metal reprovision.

You cannot remove that gap with a single instance, but you can keep it clean:

  • Give the database a PodDisruptionBudget so the drain waits for a graceful eviction rather than killing it mid-write.
  • Set a terminationGracePeriodSeconds long enough for the database to flush and shut down cleanly, so recovery on the new node is fast.
  • Make the application reconnect. A client with a connection pool and retry rides out the gap; one that fails on the first error surfaces the outage to users.

A single instance is the right choice when a short, well-defined gap during an upgrade is acceptable. When it is not, replicate.

A replicated database stays available#

To keep a database serving through an upgrade, run more than one instance with automatic failover, and let the platform move them one at a time. An operator handles the replication and the failover for you. For PostgreSQL, runs a primary and one or more replicas and promotes a replica when the primary goes away.

The same rules from the apply to the database instances:

  • Two or more instances (a primary and at least one replica).
  • Spread across nodes, so a single drain never takes the primary and a replica together. Operators usually expose anti-affinity as a setting.
  • A PodDisruptionBudget, so the drain moves one instance at a time and waits for the cluster to be healthy again before the next.

When the primary's node is drained, the operator promotes a replica, and the application reconnects to the new primary. The write path has a brief failover blip rather than a full outage, and reads from replicas continue. The application still needs reconnect and retry, because a failover always drops the current connections.

Which to choose#

Single instance Replicated with failover
Availability during an upgrade Short gap while the node is replaced Brief failover blip, no full outage
Data on a persistent volume Required, or the data is lost Required
Setup A StatefulSet and a volume claim An operator (for example CloudNativePG)
Resource cost One instance A primary and one or more replicas
Application change Reconnect and retry Reconnect and retry

Start with a single instance and a persistent volume if a short gap is acceptable. Move to a replicated set when the database must stay writable through every upgrade.