Kubernetes Control-Plane Failover That Keeps Workers Online

Note

This resource is part of the Syself Autopilot 1.36 release. For the whole picture, see what's new in 1.36.

Starting from Syself Autopilot 1.36, a worker no longer depends on the control-plane load balancer to reach the API server. Each worker talks to a small proxy on the node itself, and that proxy connects to the control planes and spreads traffic across them. So when the load balancer has a bad moment, your workers stay connected instead of dropping offline together.

What broke when the load balancer went down?

Every worker reached the API server through a single control-plane load balancer. If that load balancer went down, every worker lost the API server at the same instant. After about forty seconds Kubernetes marked them all NotReady and started moving their pods. One failure, and the whole worker fleet wobbled.

That meant the load balancer was a single point of failure, and no amount of care elsewhere in the cluster made up for it. And it's provider infrastructure that's outside our control.

How does the failover proxy work?

The proxy runs on the worker and is reachable only from that worker. The kubelet points at it instead of at the load balancer. From there the proxy spreads its own connections across the healthy control planes, and when the load balancer is unavailable it reaches them directly. The worker decides its own route to the control plane rather than handing that job to one shared endpoint.

It holds a long-lived, multiplexed connection open, so requests keep flowing without reconnecting for every call. A health check on the node re-proves the whole path about every fifteen seconds and repoints the kubelet if anything has changed.

The proxy is one of the services built into every node. Control-plane nodes don't run it, because their kubelet already talks to the API server right there on the machine.

It's automatic. No switch to flip, nothing to page you about. A load-balancer blip that used to ripple out into rescheduled pods, restarted workloads, and a wall of NotReady alerts now stays where it happened, and the rest of the cluster carries on.

Scenario	Before 1.36 (load balancer)	With the failover proxy
Normal traffic	The load balancer spreads it across control planes	The node spreads it across control planes itself
A control-plane node fails	The load balancer drops it, routes to healthy ones	The node drops it, routes to healthy ones itself
The load balancer itself fails	Every worker `NotReady` in about 40 seconds	Workers keep talking to the control planes directly
Intervention needed	Manual, once you notice	None

How is this different from the big managed platforms?

Every major managed Kubernetes service routes worker-to-API traffic the same way: through a single control-plane load balancer. That load balancer is shared cloud infrastructure, and it goes down more often than you'd think. AWS's Network Load Balancer health checks failed during the October 2025 us-east-1 outage, Microsoft's Azure Front Door load-balancing layer went down for hours that same month, and Google Cloud keeps a running history of Cloud Load Balancing incidents of its own. The usual answer is to make the load balancer more redundant, which still leaves every worker depending on one system sitting in the middle.

We used to route traffic through that same single load balancer. We wanted a cluster that stays up even when shared cloud infrastructure doesn't, so we moved the decision onto the node. Each worker now spreads its own connections across the control planes, and there's no shared box in the middle whose failure takes the fleet with it.

What the proxy doesn't do

The proxy protects the path from a worker to the control plane. It doesn't make the control plane itself more resilient, so you still run multiple control-plane nodes for that. It lives only on workers, since a control-plane node's kubelet already reaches the API server on the same machine. And it carries the kubelet's own traffic to the API server, not your application traffic. It exists to keep node health and self-healing from being tripped by a problem the node could route around on its own.

FAQ

What happened before, exactly?

A load balancer outage cut every worker off from the API server at once. Around forty seconds later Kubernetes declared them all NotReady and rescheduled their pods, a lot of churn from a single failure.

Does this replace running an HA control plane?

No. You still want multiple control-plane nodes for the control plane's own resilience. The failover proxy protects the path from workers to that control plane, which is a separate risk.

Do I need to configure it?

No. It ships on every worker and needs no configuration.

Keeping workers connected to the control plane is one direction of traffic. The opposite direction, the control plane reaching back into a node, is a different piece entirely.