Troubleshoot a stuck PVC

A stuck PVC is almost always one of a handful of causes, and the same three commands surface every one of them. Before matching symptoms, gather the evidence: kubectl describe pvc <name> shows the claim's events and whether a class or pod is missing, kubectl describe pod <name> shows why a pod will not schedule or mount, and the CSI driver logs show what the provisioner actually tried.

		$ kubectl describe pvc <pvc-name>
$ kubectl describe pod <pod-name>
$ kubectl -n kube-system logs -l app.kubernetes.io/name=topolvm,app.kubernetes.io/component=node --tail=100
	

		$ kubectl describe pvc data-local
Name:          data-local
Namespace:     default
StorageClass:  local-nvme
Status:        Pending
Volume:
Events:
  Type    Reason                Age                From                         Message
  ----    ------                ----               ----                         -------
  Normal  WaitForFirstConsumer  58s                persistentvolume-controller  waiting for first consumer to be created before binding
  Normal  WaitForPodScheduled   10s (x4 over 55s)  persistentvolume-controller  waiting for pod app to be scheduled 
	

		$ kubectl describe pod app
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  59s   default-scheduler  0/4 nodes are available: 1 node(s) did not have enough free storage, 1 node(s) had untolerated taint(s), 2 node(s) didn't match Pod's node affinity/selector.
	

Match your symptom to the sections below.

Symptom to fix#

Pending because WaitForFirstConsumer#

A PVC on standard or a local-* class shows Pending the moment you create it, and stays there until a pod that mounts it is scheduled. This is normal. These classes use WaitForFirstConsumer binding, which delays creating the volume until it knows which node the pod landed on. Nothing is wrong, and there is nothing to fix. Access modes and volume binding explains why the delay is deliberate. If the PVC is still Pending after its pod is scheduled, it is one of the cases below.

Pod on the wrong node type for the class#

This symptom comes from the scheduling direction: the scheduler places the pod first, and the volume has to be able to exist where the pod landed. A class only serves volumes on the node type that backs it. The standard class needs a cloud node, because it provisions Hetzner Cloud volumes. The local-* classes need a bare metal server, because they come from disks inside one. A pod that requests a local-nvme volume but is scheduled onto a cloud node leaves the PVC Pending forever, since the class cannot create a volume there. If the volume already exists, its node affinity pins the pod to the server that holds it, so a pod pushed elsewhere by other scheduling rules strands the claim the same way. Check where the pod is scheduled with kubectl describe pod, and fix it with node affinity or a node selector that keeps the pod on the node type its class can serve.

No volume group for the requested local class#

Even on the right server, a local class needs the matching volume group to exist. A local-ssd claim on a server that has only NVMe disks prepared stays Pending, because there is no vg-ssd to carve from. The describe pod events show the scheduler finding no node with free storage of that type. Either prepare that disk type on the server, following Set up local NVMe with TopoLVM , or schedule the pod onto a server that already has that disk type. Confirm what a node advertises with its TopoLVM capacity annotations:

		$ kubectl get node <node-name> -o jsonpath='{.metadata.annotations}' | tr ',' '\n' | grep capacity.topolvm.io
	

A disk type with no entry means no volume group of that type is ready on that node.

Multi-attach error when a pod moves#

A ReadWriteOnce volume attaches to one node at a time, and its node affinity normally keeps every pod that uses it on that one node. The error window opens when a pod moves: the volume has to detach from the old node before it can attach to the new one where the pod was scheduled. If the old pod has not fully terminated, the volume is still held, and the new pod sits in ContainerCreating with a Multi-Attach error for volume event. Usually this clears itself once the old pod finishes terminating. If it hangs, find the old pod and let it terminate, or if its node is gone, force the volume to detach by deleting the stuck old pod.

Finalizers blocking deletion#

A PVC that will not leave Terminating is almost always held by a finalizer. On a cloud volume the kubernetes.io/pvc-protection finalizer stops the PVC being deleted while a pod still uses it, which is a safeguard, not a bug. kubectl describe pvc lists the finalizers and the pod still referencing the claim. Delete or reschedule that pod, and the finalizer clears on its own so the PVC finishes deleting. Removing the finalizer by hand is a last resort and only safe once you have confirmed nothing is still mounting the volume.

A Released PV holding space#

When you delete a PVC, the underlying PV does not disappear, because every class here uses the Retain reclaim policy. The PV moves to Released and keeps its data and its space. A pile of Released PVs on a local pool holds disk that new claims cannot use, so a fresh PVC goes Pending even though the pool looks like it should have room. List them with kubectl get pv and look for Released in the status. Reclaim them to free the space; Local PV lifecycle covers doing that safely for local volumes.

When the cause is none of these#

If the symptom does not match any case above, go back to the CSI driver logs. For local volumes that is the TopoLVM node and controller pods in kube-system; for cloud volumes it is the Hetzner CSI controller. The provisioner logs the real reason it could not create or attach the volume, which is often more specific than the PVC event. Pair that with kubectl get events --sort-by=.lastTimestamp in the namespace to see the sequence of what happened.

Still stuck after these checks? The Syself Consulting team can dig into it with you.

Related: Access modes and volume binding for the attachment rules behind these symptoms, Monitor storage capacity for seeing a pool fill before it strands the next claim, and Local PV lifecycle for reclaiming the volumes a deleted PVC leaves behind.