Run a StatefulSet with storage
On this platform a node leaving the cluster is routine, not an incident. Upgrades roll nodes, autoscaling retires them, hardware gets recycled underneath you. So when you put data on disk, the question is never "what happens if a node goes away", because it goes away, but "does the data come back when the node does." That answer lives almost entirely in one field: the storage class your PersistentVolumeClaims request.
Everything else about a StatefulSet is machinery around that one decision.
A StatefulSet gives each replica a stable identity and its own volume that follows that identity. A Deployment's pods are interchangeable and share nothing durable.
| Deployment | StatefulSet | |
|---|---|---|
| Pod identity | random, interchangeable | stable and ordered (db-0, db-1, db-2) |
| Storage | shared or none | one persistent volume per pod, via volumeClaimTemplates |
| Rollout and scale | all at once | ordered, one at a time |
| Network name | via the Service | a stable per-pod DNS name from a headless Service |
Pick the storage class#
Each option behaves differently the moment a node is replaced, and each pins you to a particular access mode. Choose by how your workload already handles durability, not by raw speed.
- Local NVMe via TopoLVM (
local-nvme,ReadWriteOnce) is the fastest and cheapest, but the volume is physically on one node. If that node is replaced, the data is gone with it, so this is for databases and brokers that already replicate at the application layer, where every replica holds its own fast local copy and losing one is a rebuild, not a data loss. See Set up local NVMe with TopoLVM . - Hetzner Cloud volumes (
ReadWriteOnce) are network-attached. When a node dies, the volume detaches and reattaches to the replacement, so the data moves with the pod. Slower than local NVMe, but survives node loss on its own. See Use Hetzner Cloud volumes . - JuiceFS (
ReadWriteMany) backs a shared filesystem with object storage, for the case where several pods must read and write the same volume at once. See ReadWriteMany with JuiceFS .
For how pods get placed and rescheduled around these choices, see Keep stateful pods safe across node replacement .
volumeClaimTemplates gives each pod a volume#
volumeClaimTemplates is the field that makes a StatefulSet stateful: it mints one PVC per replica and keeps that claim bound to the pod's identity for life. db-0 always gets data-db-0, on whichever storage class you named above.
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: db
spec:
serviceName: db
replicas: 3
selector:
matchLabels:
app: db
template:
metadata:
labels:
app: db
spec:
containers:
- name: db
image: your/db:tag
volumeMounts:
- name: data
mountPath: /var/lib/db
volumeClaimTemplates:
- metadata:
name: data
spec:
storageClassName: local-nvme
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 100Gi
Stable names from a headless Service#
Replicas find each other by DNS, and that only works if each pod's name never changes. A headless Service (clusterIP: None) gives every pod one: db-0.db.namespace.svc.cluster.local. Most database and broker topologies require it.
apiVersion: v1
kind: Service
metadata:
name: db
spec:
clusterIP: None
selector:
app: db
ports:
- port: 5432
The serviceName in the StatefulSet must point at this Service, or the names never resolve.
The PVC outlives the pod#
Warning
Scaling a StatefulSet down does not delete the PVCs it leaves behind, and neither does deleting the StatefulSet itself. This is deliberate, since a rescheduled pod finds its old volume waiting, but it means scaling down never frees storage, and you keep paying for it. When you truly retire a replica, delete its leftover PVC by hand.
With each pod bound to its own claim on a storage class that matches how you replicate, a node leaving stops being a data-loss event and becomes a reschedule. For a real workload on top of this shape, see Run databases on the cluster .
Scale a workload to zero
On your own hardware, scaling to zero is the difference between paying for a node all night and letting the cluster drop it.
Keep stateful pods safe across node replacement
How to place stateful replicas so a Syself Autopilot node replacement rebuilds one member at a time and never costs you quorum or local data.