A failed or stuck upgrade, and rollback
An upgrade that stops partway through is almost always blocked on one node, not broken across the cluster. Syself Autopilot rolls an upgrade by replacing nodes one step at a time, so a single node that cannot drain or cannot join holds the whole rollout. Find the blocked node, clear it, and the rollout continues.
How the rolling replace works#
An upgrade replaces nodes with fresh sealed nodes; nothing is patched in place. The control plane rolls one node at a time, so etcd never loses more than one member and keeps quorum. Worker pools roll in parallel with each other, but within a pool the platform respects your PodDisruptionBudgets. Knowing this tells you where to look: a stalled control-plane upgrade is one control-plane node; a stalled worker upgrade is one pool, usually one drain.
The common blocks#
- A drain blocked by a PodDisruptionBudget. If a workload's PDB will not allow the eviction, the drain waits, up to
nodeDrainTimeoutSeconds: 180on the pool, then proceeds and the pods go with the node. A PDB that can never be satisfied (a single replica withmaxUnavailable: 0) makes every drain wait the full 180 seconds. Fix the workload's replicas and PDB; see Prepare workloads for upgrades . - Capacity blocking a new node. The replacement node cannot be created because Hetzner has no capacity in the region, so the rollout waits for a machine that never comes. This is No servers available ; move the pool to bare metal or wait for capacity.
- A new node failing to join. The replacement provisions but does not become Ready. Read its
Machineevents and, if you can reach it, debug the node .
The one-minor-at-a-time rule#
You upgrade one Kubernetes minor at a time, on the latest patch first. A skipped minor is rejected the moment you apply it rather than stalling part way through: the ClusterClass pins one Kubernetes version, so spec.topology.version has to match the release that classRef.name points at. To move two minors, do it as two steps: upgrade to the intermediate minor, let it finish and settle, then upgrade again. See Upgrade to a new Kubernetes version .
Hold or step back by pinning#
There is no in-place patch and no partial rollback of a single node, because nodes are replaced, not modified. What you have instead is the release pin: the cluster runs the Cluster Stack release you point it at. To hold a rollout, or step back after a within-minor update went wrong, point classRef.name at the previous known-good release and let the platform roll back to it the same way it rolled forward. This works only while spec.topology.version stays the same. Kubernetes versions cannot be decreased, so if the release you are stepping back to carries an older patch, the change is rejected. A Kubernetes minor upgrade is forward-only (you cannot downgrade a Kubernetes minor), so for a failed minor upgrade the recovery is to fix the block and finish forward, or in the worst case rebuild from your Git declaration.
When to escalate#
If the control plane itself is unhealthy during the upgrade, an etcd member is down or the API is flapping, stop and treat it as Control-plane and etcd recovery , and open a support case rather than forcing more changes onto a control plane that is already short of quorum.
No servers available
Resolve cluster degradation from Hetzner Cloud capacity limits. Move control planes and workers to bare-metal servers to avoid provisioning failures.
Control-plane and etcd recovery
The API server is flapping or an etcd member is unhealthy, so restore quorum without making it worse.