Local PV lifecycle: provision, reclaim, wipe
A local PVC turns into a thin logical volume on one server. When you claim local-nvme, TopoLVM picks a prepared server, carves a logical volume out of that server's thin pool, and binds the PVC to it. The volume lives inside the volume group for its disk type (vg-nvme, vg-ssd, or vg-hdd) and draws from the matching thin pool (pool-nvme, pool-ssd, or pool-hdd). It stays on that server for its whole life.
The classes use reclaim policy Retain. That keeps your data safe when a claim is deleted, but the freed space is not returned to the pool on its own. You return it by hand, after reading what a volume is using.
Inspect what a volume is using#
Space accounting lives on the server, not in Kubernetes. A thin logical volume only occupies pool space for the blocks it has actually written, so a 100Gi PVC that holds 4Gi of data uses 4Gi of the pool.
You can read how full each volume group is straight from the node, without a shell on it. The health daemon publishes it in the autopilot.syself.com/storage annotation, and raises a DiskUsageHigh condition when a group or thin pool runs near full, which is your cue to reclaim space or add a disk:
$ kubectl get node <node-name> -o jsonpath='{.metadata.annotations.autopilot\.syself\.com/storage}' | jq .
For the per-volume detail, which logical volume belongs to which PVC, open a shell on the server with a privileged debug pod (kubectl debug node/<node-name> -it --image=busybox --profile=sysadmin -- chroot /host), then read the LVM state.
List the volume groups and how full each one is:
$ vgs
VG #PV #LV #SN Attr VSize VFree
vg-nvme 1 4 0 wz--n- 476.44g 12.00g
List the logical volumes, including the thin pool and its data usage:
$ lvs
LV VG Attr LSize Pool Data% Meta%
pool-nvme vg-nvme twi-aotz-- 464.00g 41.87 2.31
a488ec17-8d5f-4c31-ad67-847e2a2af8dc vg-nvme Vwi-aotz-- 100.00g pool-nvme 38.90
The Data% column on the pool-nvme row is the number that matters. When it climbs toward 100%, every volume drawing from that pool is at risk, because a full thin pool cannot serve new writes. Each other row is one PVC's logical volume, named after the volume's ID.
Note
A logical volume's name is a TopoLVM volume ID. It is not derived from the PV name or the PVC name, so you cannot match them directly. TopoLVM records the mapping on a LogicalVolume resource named after the PV, which the reclaim steps below use to find the right volume.
Reclaim the space a deleted PVC leaves behind#
After kubectl delete pvc, the PersistentVolume moves to Released and its logical volume survives on the server, holding the blocks it wrote. Freeing that space is a deliberate, manual step you take once you are sure the data is no longer needed.
Find the released PV #
$ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE
pvc-1a2b3c4d 100Gi RWO Retain Released default/pv-claim local-nvme <unset> 10d
Look up its logical volume and node #
TopoLVM creates a LogicalVolume resource named after the PV. It holds both the server that
carries the volume and the volume ID that names it in lvs:
$ kubectl get logicalvolume pvc-1a2b3c4d -o jsonpath='{.spec.nodeName}{"\n"}{.status.volumeID}{"\n"}'
bm-1
a488ec17-8d5f-4c31-ad67-847e2a2af8dc
Remove the logical volume with lvremove #
On that server, reached with a debug pod (kubectl debug node/<node-name> -it --image=busybox --profile=sysadmin -- chroot /host), confirm the logical volume, then remove it:
$ lvs vg-nvme
$ lvremove vg-nvme/a488ec17-8d5f-4c31-ad67-847e2a2af8dc
Caution
lvremove erases the data on that volume for good. There is no undo, and the blocks return to the pool for reuse. Take the volume ID from the LogicalVolume of the Released PV you mean to reclaim, never by guessing from the name, and never remove a volume whose PV is still Bound.
Then delete the Released PV object so it stops lingering in the cluster:
$ kubectl delete pv pvc-1a2b3c4d
After lvremove, the pool's Data% in lvs drops by whatever the volume was holding.
Sometimes you want the whole disk back, not one volume. You might repurpose a server, or a disk carries LVM state from an earlier setup that you need to clear before preparing it again .
Remove a volume group once its logical volumes are gone:
$ vgremove vg-nvme
Then clear the LVM signature from each physical volume so the disk is blank:
$ pvremove /dev/nvme1n1
$ wipefs --all /dev/nvme1n1
Caution
vgremove, pvremove, and wipefs destroy every volume on the disk at once, not one PVC. Run them only on a data disk you have confirmed is empty of anything you need, and never on the OS disk (the one mounted at / and /boot). Wiping the OS disk breaks the running server.
A disk that still holds old LVM state from a previous preparation refuses a fresh pvcreate until you clear it. Wipe it with the steps above first, then create the physical volume and group as normal.
Grow a group instead of reclaiming#
When a pool is filling up and you have a spare disk in the server, add capacity rather than deleting volumes. Create a physical volume on the new disk and extend the existing group into it:
$ pvcreate /dev/nvme2n1
$ vgextend vg-nvme /dev/nvme2n1
The thin pool does not grow automatically. Extend the pool into the new free space so volumes can use it:
$ lvextend --extents +100%FREE vg-nvme/pool-nvme
TopoLVM picks up the larger group and reports the higher free capacity on that node.
Next, watch pool usage before it becomes urgent with monitor storage capacity , or read data survival across reprovision to see why a reprovision leaves these volumes intact.
Set up local NVMe with TopoLVM
Install cert-manager and TopoLVM, prepare each bare metal server's data disks with LVM, and get the local-nvme, local-ssd, and local-hdd storage classes.
Data survival across reprovision
Syself Autopilot rewrites the OS disk on every reprovision and never touches your data disks, so local data survives in place unless the workload moves to another server.