High-availability control plane

For production clusters, we strongly recommend that you run at least three control plane nodes.

A three-node control plane is the standard high-availability baseline. It allows the cluster to continue operating when one control plane node becomes unavailable while preserving etcd quorum and keeping the Kubernetes API available through the control-plane load balancer.

High availability also depends on how those nodes are placed and sized. Syself recommends at least 4 vCPU and 8 GB of RAM per control plane node, with enough additional capacity for the size and activity of the cluster.

On Hetzner Cloud, place control plane VMs in a placement group so that they run on different physical hosts.

Why the control plane needs quorum#

Each Syself Autopilot control plane node runs an etcd member.

etcd stores the Kubernetes cluster state and uses consensus to decide which writes are accepted. A majority of members must be available for etcd to continue processing changes.

The required quorum is:

		quorum = floor(number of members / 2) + 1
	

This is why highly available etcd clusters normally use an odd number of members.

Adding a fourth member to a three-member control plane increases the number of machines participating in consensus but does not increase failure tolerance.

A five-node control plane is required before the cluster can tolerate two unavailable control plane members.

For most clusters, three members provide the best balance between availability, cost, and consensus overhead.

Configure the number of control plane nodes#

Set the number of control plane nodes with spec.topology.controlPlane.replicas in the Cluster object:

		apiVersion: cluster.x-k8s.io/v1beta2
kind: Cluster
metadata:
  name: mycluster
spec:
  topology:
    controlPlane:
      class: hcloud
      replicas: 3
	

Syself Autopilot continuously maintains the requested number of control plane Machines.

If one Machine fails and cannot recover, the platform can replace it while the remaining members continue serving the cluster.

How many control plane nodes should you run?#

One node#

Use one control plane node only for environments where control-plane downtime is acceptable, such as:

• Development clusters
• Temporary test environments
• Disposable CI clusters
• Small non-production environments

A one-node control plane has no redundancy. If the node is unavailable, the Kubernetes API and etcd are unavailable until the node recovers or is replaced.

Existing workloads on healthy workers may continue running, but Kubernetes cannot schedule new pods or reconcile changes while the control plane is unavailable.

Three nodes#

Use three control plane nodes for normal production clusters.

Three members:

• Maintain etcd quorum when one member is unavailable
• Allow one control plane node to be replaced at a time
• Provide multiple API server backends behind the load balancer
• Support rolling upgrades without intentionally taking the control plane offline
• Keep consensus overhead reasonable

This is the standard Syself recommendation for a highly available cluster.

Five nodes#

Use five control plane nodes only when you have a reason to tolerate two simultaneous control plane failures or when the cluster is large or experiences unusually high control-plane churn.

Examples include clusters with:

• Large numbers of nodes
• Large numbers of Kubernetes objects
• High API request volume
• Many controllers or operators
• Frequent object updates
• Stronger control-plane failure requirements

More etcd members are not automatically better. Every write must be replicated through the consensus group, so increasing the number of members also increases coordination work.

For most clusters, three properly sized and properly distributed control plane nodes are preferable to five undersized ones.

Size control plane nodes correctly#

Syself recommends at least 4 vCPU and 8 GB of RAM per control plane node.

On Hetzner Cloud, use CPX32 or larger for production control planes.

Do not use CX instances for production control plane nodes.

Configure the machine type through the Cluster topology:

		spec:
  topology:
    controlPlane:
      class: hcloud
      replicas: 3
 
    variables:
      - name: controlPlaneMachineTypeHcloud
        value: cpx32
	

The minimum is a starting point, not a universal size.

Control-plane resource usage grows with the cluster.

Important factors include:

• Number of worker nodes
• Number of pods
• Number of Kubernetes objects
• Number of namespaces
• API request volume
• Number and activity of operators and controllers
• Admission webhooks
• Watch traffic
• Frequency of workload deployments and updates
• etcd database size and write rate

A cluster with 30 large workers can place more pressure on the control plane than a cluster with three small workers, even when both run the same number of control plane nodes.

As the cluster grows, increase control-plane resources rather than adding control plane replicas purely for compute capacity.

See Server types and sizing for sizing guidance.

Spread Hetzner Cloud control planes across physical hosts#

Three control plane VMs do not provide meaningful host-level redundancy if several of them run on the same physical server.

For Hetzner Cloud control planes, use a placement group so the control plane VMs are distributed across separate physical hosts.

flowchart TB
    subgraph A["Physical host A"]
        cp1["control-plane-1"]
    end

    subgraph B["Physical host B"]
        cp2["control-plane-2"]
    end

    subgraph C["Physical host C"]
        cp3["control-plane-3"]
    end

    cp1 --> quorum["etcd quorum: 2 of 3 required"]
    cp2 --> quorum
    cp3 --> quorum

If one physical host fails, only one control plane member is lost and the remaining two members retain quorum.

Without this separation, one physical host failure could affect multiple VMs and remove the quorum that the three-node design was intended to protect.

See Use placement groups for configuration.

Keep control plane nodes in one region#

All control plane nodes should run within one Hetzner region.

etcd is latency-sensitive because writes require coordination between a majority of members. Spreading etcd members across distant regions increases consensus latency and makes network partitions more likely to affect cluster availability.

Use separate physical hosts within the region to protect against host failure.

Do not use geographic distance between etcd members as a disaster-recovery mechanism.

Protection against loss of an entire region requires a separate recovery strategy.

Size the whole cluster, not only the control plane#

Control plane sizing and worker sizing solve different problems.

The control plane load grows primarily with the total activity of the cluster, not with application CPU or memory consumption.

Workers run your application workloads. Size them according to:

• Pod CPU requests
• Pod memory requests
• Pod density
• DaemonSet overhead
• Storage requirements
• GPU or other specialized hardware requirements
• Capacity required during maintenance and upgrades

Do not compensate for undersized workers by increasing control plane capacity, or compensate for an overloaded control plane by adding more workers.

They are separate capacity domains.

Add or remove control plane replicas carefully#

Change the replica count to the number you want, and let the platform get there.

For example, to move from one control plane node to three:

		spec:
  topology:
    controlPlane:
      replicas: 3
	

Syself Autopilot adds one control plane node at a time. It waits for each new member to join, for etcd to report it healthy, and for the control plane to be stable again before it starts the next one. You do not need to step through the intermediate values yourself.

Always move between odd numbers. Do not set an even replica count and leave it there: a two-member control plane needs both members for quorum, so it tolerates no failures at all while running twice the hardware that can fail.

The same applies when reducing the control plane. Set the target count and allow etcd membership and Cluster API reconciliation to complete.

What a highly available control plane protects against#

A three-node control plane is designed to tolerate the loss of one control plane member.

For example:

		control-plane-1   Ready
control-plane-2   Ready
control-plane-3   Failed
 
etcd members available: 2 / 3
quorum required:        2 / 3
result:                 cluster continues operating
	

The control-plane load balancer stops sending API traffic to the failed backend and routes clients to the remaining healthy API servers.

Meanwhile, Syself Autopilot can remediate or replace the unhealthy Machine.

This protects against failures such as:

• One control plane VM failing
• One physical host failing when placement groups are used
• One control plane node rebooting
• One control plane node being replaced during an upgrade
• One etcd member temporarily becoming unavailable
• The load balancer routing around one unhealthy API server

For more information about machine remediation, see Self-healing and node replacement .

What it does not protect against#

Three control plane nodes do not make the cluster immune to every infrastructure failure.

Loss of two control plane nodes#

A three-member etcd cluster requires two healthy members.

If two are unavailable:

		etcd members available: 1 / 3
quorum required:        2 / 3
result:                 no quorum
	

The Kubernetes API cannot safely process changes until quorum is restored.

Use five control plane members when tolerating two simultaneous control plane failures is an explicit requirement.

Loss of the entire region#

Keeping three control plane nodes on separate physical hosts protects against a host failure, not a regional failure.

A failure affecting the complete Hetzner region can make every control plane member unavailable at once.

Regional disaster recovery requires backups and a separate recovery strategy rather than spreading one etcd cluster across distant regions.

Application failures#

Control plane HA does not make your applications highly available.

A Deployment with one replica can still become unavailable when its worker fails. Use PodDisruptionBudgets, topology spread and replication so your workload can also tolerate failures.

Worker node failures#

Workers are separate from the control plane.

Losing a worker does not remove etcd quorum or an API server member, but it can affect applications running on that worker.

Syself Autopilot manages worker replacement independently from the control plane.

Control plane and worker availability are separate#

A useful way to think about the cluster is as two availability problems:

flowchart TB
    cluster["Kubernetes cluster"]

    cluster --> cp
    cluster --> workers

    subgraph cp["Control plane"]
        api["Kubernetes API"]
        etcd["etcd"]
        controllers["controllers"]
        scheduler["scheduler"]
    end

    subgraph workers["Workers"]
        apps["Applications"]
        data["Stateful data"]
        capacity["Pod capacity"]
    end

Control plane redundancy keeps Kubernetes itself operational.

Worker redundancy keeps applications operational.

A three-node control plane with one worker is a highly available control plane, but it is not a highly available application platform if every workload depends on that single worker.

Size and distribute both independently according to what must remain available during a failure.

Recommended production baseline#

For a typical production cluster:

• Run three control plane nodes.
• Use at least 4 vCPU and 8 GB RAM per control plane node.
• On Hetzner Cloud, use CPX32 or larger.
• Do not use CX server types for production control planes.
• Put HCloud control plane VMs in a placement group.
• Keep all control plane nodes in the same region.
• Run enough workers to tolerate one being unavailable during maintenance.
• Increase control-plane resources as node, pod, object, and API traffic grows.
• Use five control plane members only when the additional failure tolerance or cluster scale justifies it.

Related#

• Self-healing
• Run bare-metal control planes and clusters
• Use placement groups
• Autoscale a cluster
• Worker-less cluster
• Plan maintenance and prechecks
• Server types and sizing
• High availability