Set up local NVMe with TopoLVM

Local storage is the fastest disk on the platform, and it is where a production database belongs. Reads and writes go straight to the NVMe with no network in between. Local storage never replicates for you, so anything that must survive the loss of a server has to arrange that itself: run a database under an operator such as CloudNativePG⁠, which replicates across servers and backs itself up, or keep the data in object storage , where the S3 backend carries the redundancy. Choose storage for a workload covers when and why to run a database this way, and Performance of local vs network storage covers why it is fast. This page sets up the storage underneath.

How a database uses local storage#

To the storage layer, a database is nothing special. The operator asks for a PersistentVolumeClaim on the local-nvme class, the same way any workload claims storage, and Kubernetes binds it to a logical volume on the server. Local just means the disk sits in the machine instead of across the network.

CloudNativePG does not build on a StatefulSet. It manages the Postgres pods and their claims through its own controller, because a database needs decisions a StatefulSet cannot make from a pod's number alone: which instance is the primary, what order to update in, and on failover promoting the replica that is furthest ahead rather than the next one in order. Some operators do run on StatefulSets with Patroni. Either way, from the disk's point of view it is an ordinary PVC on local-nvme.

You run TopoLVM yourself. It is not part of the cluster stack, so you install and operate it, which gives you full control over your local storage. Syself keeps the chart compatible with Syself Autopilot and documents each step here, so stick to these steps and your setup keeps working across releases. If you would rather not operate it yourself, the Syself Consulting team can share best practices or run TopoLVM's setup and maintenance for you as a full service.

Setup happens in two one-time steps. Once per cluster you install cert-manager and the TopoLVM driver. Once per bare metal server you turn its data disks into LVM volume groups. After the cluster is ready, each new server needs only the disk step, as part of bringing it online.

Install the driver, once per cluster#

		$ helm repo add jetstack https://charts.jetstack.io
$ helm repo update
$ helm template --namespace=kube-system cert-manager jetstack/cert-manager --set installCRDs=true | kubectl apply -n kube-system -f -
	

		$ helm repo add syself https://charts.syself.com
$ helm repo update
$ helm template --namespace=kube-system csi-local syself/topolvm | kubectl apply -n kube-system -f -
	

		$ kubectl get storageclasses
NAME                 PROVISIONER         RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
local-hdd            topolvm.io          Retain          WaitForFirstConsumer   true                   10d
local-nvme           topolvm.io          Retain          WaitForFirstConsumer   true                   10d
local-ssd            topolvm.io          Retain          WaitForFirstConsumer   true                   10d
standard (default)   csi.hetzner.cloud   Retain          WaitForFirstConsumer   true                   10d
	

Prepare a server's disks, once per server#

Preparing the disks belongs with bringing a server into the cluster. When you add a bare metal server you already pick its OS disk by WWN, and every other disk becomes a data disk. This step turns those data disks into the LVM volume groups TopoLVM serves. You do it once, when the server joins. Cluster upgrades and reprovisions leave the data disks alone, so you never repeat it for the same server.

Run the commands on the server's host. Two ways get you there without handing out SSH keys. On a server that has already joined, open a root shell with a privileged debug pod:

		$ kubectl debug node/<node-name> -it --image=busybox --profile=sysadmin -- chroot /host
	

If you are preparing the server before it joins, do the same from the Hetzner rescue system , the environment you already use to clear a disk at onboarding. Direct SSH is a break-glass fallback, not the normal path.

		$ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
nvme1n1     259:0    0 476.9G  0 disk
nvme0n1     259:1    0 476.9G  0 disk
|-nvme0n1p1 259:2    0   512M  0 part /boot/efi
|-nvme0n1p2 259:3    0     1G  0 part /boot
`-nvme0n1p3 259:4    0 475.4G  0 part /
	

		$ pvcreate /dev/nvme1n1
	

		$ vgcreate vg-nvme /dev/nvme1n1
	

		$ lvcreate --thinpool pool-nvme --extents 100%FREE vg-nvme
	

		$ lvcreate --thinpool pool-ssd --extents 100%FREE vg-ssd
	

		$ lvcreate --thinpool pool-hdd --extents 100%FREE vg-hdd
	

		$ kubectl get node <node-name> -o jsonpath='{.metadata.annotations}' | tr ',' '\n' | grep capacity.topolvm.io
	

		$ kubectl -n kube-system delete pod -l app.kubernetes.io/name=topolvm,app.kubernetes.io/component=node --field-selector spec.nodeName=<node-name>
	

Use the classes#

To pick which class fits which workload, and so which disks to prepare, see Performance of local vs network storage .

Point a PVC at local-nvme, local-ssd, or local-hdd like any other class. A claim on a type the target server does not have stays Pending, because there is no matching volume group to fill it.

		apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pv-claim
spec:
  storageClassName: local-nvme
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
	

A local volume is ReadWriteOnce and stays on the server it was created on. Because the volume cannot move, the pod that uses it must schedule to that same server, and if the server is full or gone the pod stays Pending. The volume carries no redundancy of its own, so give it an off-server backup first, and application-level replication on top. To keep a lost or decommissioned disk from giving up its data, encrypt it at rest: Encrypt data at rest with LUKS layers LUKS under these same volume groups.

Related: Encrypt data at rest with LUKS , Performance of local vs network storage , and Local PV lifecycle .