Skip to main content

Run a StatefulSet with storage

Inspect 1.36

On this platform a node leaving the cluster is routine, not an incident. Upgrades roll nodes, autoscaling retires them, hardware gets recycled underneath you. So when you put data on disk, the question is never "what happens if a node goes away", because it goes away, but "does the data come back when the node does." That answer lives almost entirely in one field: the storage class your PersistentVolumeClaims request.

Everything else about a StatefulSet is machinery around that one decision.

Pick the storage class#

Each option behaves differently the moment a node is replaced, and each pins you to a particular access mode. Choose by how your workload already handles durability, not by raw speed.

  • Local NVMe via TopoLVM (local-nvme, ReadWriteOnce) is the fastest and cheapest, but the volume is physically on one node. If that node is replaced, the data is gone with it, so this is for databases and brokers that already replicate at the application layer, where every replica holds its own fast local copy and losing one is a rebuild, not a data loss. See .
  • Hetzner Cloud volumes (ReadWriteOnce) are network-attached. When a node dies, the volume detaches and reattaches to the replacement, so the data moves with the pod. Slower than local NVMe, but survives node loss on its own. See .
  • JuiceFS (ReadWriteMany) backs a shared filesystem with object storage, for the case where several pods must read and write the same volume at once. See .

For how pods get placed and rescheduled around these choices, see .

volumeClaimTemplates gives each pod a volume#

volumeClaimTemplates is the field that makes a StatefulSet stateful: it mints one PVC per replica and keeps that claim bound to the pod's identity for life. db-0 always gets data-db-0, on whichever storage class you named above.

statefulset.yamlyaml
		apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: db
spec:
  serviceName: db
  replicas: 3
  selector:
    matchLabels:
      app: db
  template:
    metadata:
      labels:
        app: db
    spec:
      containers:
        - name: db
          image: your/db:tag
          volumeMounts:
            - name: data
              mountPath: /var/lib/db
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        storageClassName: local-nvme
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 100Gi
	

Stable names from a headless Service#

Replicas find each other by DNS, and that only works if each pod's name never changes. A headless Service (clusterIP: None) gives every pod one: db-0.db.namespace.svc.cluster.local. Most database and broker topologies require it.

yaml
		apiVersion: v1
kind: Service
metadata:
  name: db
spec:
  clusterIP: None
  selector:
    app: db
  ports:
    - port: 5432
	

The serviceName in the StatefulSet must point at this Service, or the names never resolve.

The PVC outlives the pod#

Warning

Scaling a StatefulSet down does not delete the PVCs it leaves behind, and neither does deleting the StatefulSet itself. This is deliberate, since a rescheduled pod finds its old volume waiting, but it means scaling down never frees storage, and you keep paying for it. When you truly retire a replica, delete its leftover PVC by hand.

With each pod bound to its own claim on a storage class that matches how you replicate, a node leaving stops being a data-loss event and becomes a reschedule. For a real workload on top of this shape, see .