Skip to main content

A failed or stuck upgrade, and rollback

Inspect 1.36

An upgrade that stops partway through is almost always blocked on one node, not broken across the cluster. Syself Autopilot rolls an upgrade by replacing nodes one step at a time, so a single node that cannot drain or cannot join holds the whole rollout. Find the blocked node, clear it, and the rollout continues.

How the rolling replace works#

An upgrade replaces nodes with fresh sealed nodes; nothing is patched in place. The control plane rolls one node at a time, so etcd never loses more than one member and keeps quorum. Worker pools roll in parallel with each other, but within a pool the platform respects your PodDisruptionBudgets. Knowing this tells you where to look: a stalled control-plane upgrade is one control-plane node; a stalled worker upgrade is one pool, usually one drain.

The common blocks#

  • A drain blocked by a PodDisruptionBudget. If a workload's PDB will not allow the eviction, the drain waits, up to nodeDrainTimeoutSeconds: 180 on the pool, then proceeds and the pods go with the node. A PDB that can never be satisfied (a single replica with maxUnavailable: 0) makes every drain wait the full 180 seconds. Fix the workload's replicas and PDB; see .
  • Capacity blocking a new node. The replacement node cannot be created because Hetzner has no capacity in the region, so the rollout waits for a machine that never comes. This is ; move the pool to bare metal or wait for capacity.
  • A new node failing to join. The replacement provisions but does not become Ready. Read its Machine events and, if you can reach it, .

The one-minor-at-a-time rule#

You upgrade one Kubernetes minor at a time, on the latest patch first. A skipped minor is rejected the moment you apply it rather than stalling part way through: the ClusterClass pins one Kubernetes version, so spec.topology.version has to match the release that classRef.name points at. To move two minors, do it as two steps: upgrade to the intermediate minor, let it finish and settle, then upgrade again. See .

Hold or step back by pinning#

There is no in-place patch and no partial rollback of a single node, because nodes are replaced, not modified. What you have instead is the release pin: the cluster runs the Cluster Stack release you point it at. To hold a rollout, or step back after a within-minor update went wrong, point classRef.name at the previous known-good release and let the platform roll back to it the same way it rolled forward. This works only while spec.topology.version stays the same. Kubernetes versions cannot be decreased, so if the release you are stepping back to carries an older patch, the change is rejected. A Kubernetes minor upgrade is forward-only (you cannot downgrade a Kubernetes minor), so for a failed minor upgrade the recovery is to fix the block and finish forward, or in the worst case rebuild from your Git declaration.

When to escalate#

If the control plane itself is unhealthy during the upgrade, an etcd member is down or the API is flapping, stop and treat it as , and open a support case rather than forcing more changes onto a control plane that is already short of quorum.