Control-plane and etcd recovery
When the API server is flapping or an etcd member is unhealthy, the danger is not the failure itself but the recovery: a hasty deletion can turn a one-member problem into a lost quorum. The platform has guards that stop unsafe actions, and the right move is to work with them, not around them. This is the runbook for restoring a control plane without making it worse.
Warning
Do not delete a control-plane Machine to "reset" it while quorum is already at risk. A three-member etcd cluster needs two healthy members; removing one when only two are healthy loses quorum, and the API server can no longer accept changes. Diagnose first.
Read the symptoms#
Typical signs: kubectl intermittently fails or times out, one control-plane node is NotReady, or an etcd member is reported down. Check the control-plane Machines and etcd health from the management cluster:
$ kubectl get machines -l cluster.x-k8s.io/control-plane -n my-organization
$ kubectl get kubeadmcontrolplane -n my-organization
Read the KubeadmControlPlane (KCP) status and conditions: it reports how many replicas are ready and whether etcd is healthy. That tells you whether you still have quorum (two of three) or have already lost it (one of three).
The guards that protect you#
The platform will not let you make the situation worse by accident:
- The KCP quorum guard refuses a control-plane Machine deletion that would break etcd quorum. If a delete is rejected or does not proceed, that guard is doing its job: you are trying to remove a member the cluster cannot afford to lose.
- The concurrent-remediation gate stops the platform from replacing more than one control-plane Machine at a time, so self-healing cannot churn the whole control plane at once during an incident.
Let these hold. If the guard is blocking a deletion, the answer is to restore the unhealthy member, not to force the delete.
Do manual work under a pause#
If you need to investigate or act on a specific Machine by hand, pause its remediation first so the platform does not replace it out from under you mid-investigation:
$ kubectl annotate machine <machine> -n my-organization cluster.x-k8s.io/paused=true
Do your work, then remove the annotation to let normal reconciliation and remediation resume. A paused Machine is not remediated, so do not leave it paused.
How a replacement rejoins safely#
When a control-plane member is replaced, it rejoins etcd as a learner first: it catches up on the data without voting, and is only promoted to a full voting member once it is in sync. This is why a fresh control-plane node does not endanger quorum while it is coming up, it is not counted until it is ready. Let that process complete rather than intervening.
Reboot before replace#
For a member that is unhealthy but whose data may be intact, a reboot is less disruptive than a replacement: it keeps the existing etcd data instead of resyncing a fresh member. The platform's remediation reboots before it replaces for this reason. If you are acting by hand, try recovering the existing member before you remove it.
When to open a support case immediately#
Open a case, do not keep trying changes, if: quorum is already lost (only one of three members healthy), the KCP reports etcd unhealthy and it does not recover on its own, or you are unsure whether an action is safe. A control plane short of quorum is exactly the situation where one more wrong change causes data loss, and Syself can recover etcd in ways that are not safe to attempt blind. Preserve the state and get help.
Related#
- A failed or stuck upgrade, and rollback : when a rollout, not the control plane itself, is the problem.