Monitor storage capacity

A full thin pool does not return an error to the application. It freezes writes. Every volume carved from that pool stops accepting data at the same moment, so one workload filling the disk takes down every other workload sharing it. By the time a database reports a write failure, the damage is already spread. Monitoring is how you see the pool approaching full while there is still room to act.

Start with the signal Syself already computes, then layer your own detail on top.

Start with the built-in condition#

Every node runs a health daemon that watches disk and pool fill and raises a condition when it crosses a threshold, so your first line of monitoring needs no setup. DiskUsageHigh turns True when a filesystem, an LVM volume group, or a thin pool is near full, ahead of kubelet's own DiskPressure eviction. Alert on it and a filling pool surfaces on its own.

		$ kubectl get node <node-name> -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}{"\n"}{end}'
	

Watch DiskUsageHigh for capacity, and DisksFailure and DiskWearHigh for a disk that is failing or wearing out. The daemon also publishes the detail behind the condition as a node annotation, autopilot.syself.com/storage. It lists each volume group with how full it is and whether it is thin-pool backed, so you can read pool fill straight from the node without a shell on it:

		$ kubectl get node <node-name> -o jsonpath='{.metadata.annotations.autopilot\.syself\.com/storage}' | jq .
	

Beyond the built-in signal, three levels fill in the rest: how full each volume is, how full each thin pool is, and how much free capacity each node still advertises.

Per-volume usage from kubelet#

Kubelet exposes usage for every mounted PVC to Prometheus. The kubelet_volume_stats_used_bytes and kubelet_volume_stats_capacity_bytes series give you the used and total size of each volume, labeled by namespace and PVC name. Divide one by the other for a fill percentage, and alert when a volume crosses a threshold you can still grow before it hits the wall.

These metrics tell you which workload is growing and how fast. They do not tell you whether the disk underneath has room to grow it into. That is what the thin pool answers.

Per-pool fill with vgs and lvs#

A local class is backed by an LVM thin pool, and a thin pool is overcommittable: the volumes on it can claim more space in total than the disk physically holds. So a volume can look half empty while the pool behind it is nearly full. Two figures matter, both from lvs on the node:

• Data% is how much of the pool's real disk space is used. At 100% the pool is out of room and writes freeze.
• Meta% is how full the pool's metadata area is. Metadata can fill before data does, and exhausting it freezes the pool just the same. Watch it as closely as Data%.

Read them on the node itself. Open a shell with a privileged debug pod, kubectl debug node/<node-name> -it --image=busybox --profile=sysadmin -- chroot /host, and run:

		$ lvs -o lv_name,vg_name,data_percent,metadata_percent vg-nvme/pool-nvme
  LV        VG      Data%  Meta%
  pool-nvme vg-nvme 63.20  11.40
$ vgs vg-nvme
  VG      #PV #LV #SN Attr   VSize   VFree
  vg-nvme   1   1   0 wz--n- 476.90g 0
	

Per-node capacity from TopoLVM#

TopoLVM publishes each server's remaining free capacity per disk type as a node annotation. This is the number the scheduler reads when it decides whether a new local PVC can fit on a node, so watching it tells you when a node can no longer accept new volumes of a given class.

		$ kubectl get node <node-name> -o jsonpath='{.metadata.annotations}' | tr ',' '\n' | grep capacity.topolvm.io
	

A node whose capacity for a disk type has dropped to zero will not schedule any more volumes of that class, and PVCs targeting it stay Pending. Track this alongside the pool Data% so a node running low is visible before a claim gets stuck.

Object storage fills differently#

The juicefs class does not live on a local pool, so Data% does not apply. Its data sits in an object storage bucket, and the limit you watch there is the bucket's own quota and the size of the metadata engine. Check bucket usage against its quota on the same schedule you check pool fill, so a shared filesystem does not run out from under the pods using it.

Dashboards and thresholds#

Build the alerts in tiers so you get warning before emergency. A first alert around two-thirds full gives you time to plan more disk or clean up. A second, louder alert near the point of no return means act now. Set the same tiers on Meta%, since it can fill independently of Data%. Put the kubelet volume series on a dashboard grouped by namespace so you can see which workload is driving growth.

When a dashboard shows a pool climbing, head to Tune storage performance to reclaim and size the headroom that brings it back down.