Keep databases available
A database has state, so an upgrade has to move both the pod and its data when the node it runs on is replaced. Whether the database stays reachable during that move depends on two choices: where the data lives, and whether you run one instance or a replicated set. For setting a database up in the first place, see Run databases .
The data has to survive the node move#
The demo pattern of a single pod with an emptyDir volume loses everything the moment the pod moves. A database needs a persistent volume so its data outlives the pod and the node:
- Hetzner Cloud volumes are network storage. When the pod moves to a new node, the volume detaches from the old node and reattaches on the new one. The data survives; the detach-and-attach adds to the gap. See Use Hetzner Cloud volumes .
- Local storage on bare metal (TopoLVM) keeps the data disk when Syself reprovisions the server, so the data is still there when the server comes back. It is node-bound, so the pod waits for that same server to finish reprovisioning rather than moving elsewhere. See Set up local NVMe with TopoLVM .
Either way, use a StatefulSet with a volumeClaimTemplate, not an emptyDir, so the claim follows the database.
A single instance has a short gap#
One database pod, however well its storage is set up, is unavailable while its node is replaced. The pod stops, the volume detaches and reattaches (or the bare-metal server reprovisions), the pod starts, and the database runs its recovery. That is seconds to tens of seconds on a cloud volume, longer on a bare-metal reprovision.
You cannot remove that gap with a single instance, but you can keep it clean:
- Give the database a PodDisruptionBudget so the drain waits for a graceful eviction rather than killing it mid-write.
- Set a
terminationGracePeriodSecondslong enough for the database to flush and shut down cleanly, so recovery on the new node is fast. - Make the application reconnect. A client with a connection pool and retry rides out the gap; one that fails on the first error surfaces the outage to users.
A single instance is the right choice when a short, well-defined gap during an upgrade is acceptable. When it is not, replicate.
A replicated database stays available#
To keep a database serving through an upgrade, run more than one instance with automatic failover, and let the platform move them one at a time. An operator handles the replication and the failover for you. For PostgreSQL, CloudNativePG runs a primary and one or more replicas and promotes a replica when the primary goes away.
The same rules from the stateless guide apply to the database instances:
- Two or more instances (a primary and at least one replica).
- Spread across nodes, so a single drain never takes the primary and a replica together. Operators usually expose anti-affinity as a setting.
- A PodDisruptionBudget, so the drain moves one instance at a time and waits for the cluster to be healthy again before the next.
When the primary's node is drained, the operator promotes a replica, and the application reconnects to the new primary. The write path has a brief failover blip rather than a full outage, and reads from replicas continue. The application still needs reconnect and retry, because a failover always drops the current connections.
Which to choose#
| Single instance | Replicated with failover | |
|---|---|---|
| Availability during an upgrade | Short gap while the node is replaced | Brief failover blip, no full outage |
| Data on a persistent volume | Required, or the data is lost | Required |
| Setup | A StatefulSet and a volume claim | An operator (for example CloudNativePG) |
| Resource cost | One instance | A primary and one or more replicas |
| Application change | Reconnect and retry | Reconnect and retry |
Start with a single instance and a persistent volume if a short gap is acceptable. Move to a replicated set when the database must stay writable through every upgrade.
Related#
- Run databases : set up a database and a database operator.
- Back up and restore : the safety net underneath any database.
- Keep stateless workloads available : the rules the database replicas also need.
- Prepare workloads for upgrades : the four rules every workload needs to ride out a node drain.
Keep ingress and load balancers available
Keep your ingress controller and Hetzner load balancer serving through a Syself Autopilot upgrade with spread ingress replicas, a PodDisruptionBudget, and the right externalTrafficPolicy.
Run dev, staging, and prod
Stand up separate clusters per environment and promote the same declaration through them.