Run a StatefulSet with storage
On Syself Autopilot a node leaving the cluster is routine. Upgrades roll nodes, autoscaling retires them, hardware gets recycled. So when you put data on disk, the question is never "what happens if a node goes away", because it goes away, but "does the data come back when the node does." That answer depends almost entirely on the storage class your PersistentVolumeClaims request.
Pick the storage class
Each option behaves differently the moment a node is replaced, and each pins you to a particular access mode. Choose by how your workload already handles durability, not by speed.
- Local NVMe via TopoLVM (
local-nvme,ReadWriteOnce) is the fastest and cheapest, but the volume is physically on one node. If that node is replaced, the data is gone with it, so this is for databases and brokers that already replicate at the application layer, where every replica holds its own fast local copy and losing one is a rebuild, not a data loss. See Set up local NVMe with TopoLVM . - Hetzner Cloud volumes (
ReadWriteOnce) are network-attached. When a node dies, the volume detaches and reattaches to the replacement, so the data moves with the pod. Slower than local NVMe, but survives node loss on its own. See Use Hetzner Cloud volumes . - JuiceFS (
ReadWriteMany) backs a shared filesystem with object storage, for the case where several pods must read and write the same volume at once. See ReadWriteMany with JuiceFS .
For how pods get placed and rescheduled around these choices, see Keep stateful pods safe across node replacement .
volumeClaimTemplates gives each pod a volume
volumeClaimTemplates is the field that makes a StatefulSet stateful: it creates one PVC per replica and keeps that claim bound to the pod's identity for life. db-0 always gets data-db-0, on whichever storage class you named above.
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: db
spec:
serviceName: db
replicas: 3
selector:
matchLabels:
app: db
template:
metadata:
labels:
app: db
spec:
containers:
- name: db
image: your/db:tag
volumeMounts:
- name: data
mountPath: /var/lib/db
volumeClaimTemplates:
- metadata:
name: data
spec:
storageClassName: local-nvme
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 100Gi
Stable names from a headless Service
Replicas find each other by DNS, and that only works if each pod's name never changes. A headless Service (clusterIP: None) gives every pod one: db-0.db.namespace.svc.cluster.local. Most database and broker topologies require it.
apiVersion: v1
kind: Service
metadata:
name: db
spec:
clusterIP: None
selector:
app: db
ports:
- port: 5432
The serviceName in the StatefulSet must point at this Service, or the names never resolve.
The PVC outlives the pod
Warning
Scaling a StatefulSet down does not delete the PVCs it leaves behind, and neither does deleting the StatefulSet itself. This is deliberate, since a rescheduled pod finds its old volume waiting, but it means scaling down never frees storage, and you keep paying for it. When you truly retire a replica, delete its leftover PVC by hand.
With each pod bound to its own claim on a storage class that matches how you replicate, a node leaving stops being a data-loss event and becomes a reschedule. For a real workload on top of this, see Run databases on the cluster .
Scale a workload to zero
On your own hardware, scaling to zero is the difference between paying for a node all night and letting the cluster drop it.
Keep stateful pods safe across node replacement
How to place stateful replicas so a Syself Autopilot node replacement rebuilds one member at a time and never costs you quorum or local data.