Skip to main content

Tune health checks for node replacement

Inspect 1.36

Load balancer health checks matter more on Syself Autopilot than on a platform that patches nodes in place, because here nodes are drained and replaced during every upgrade and every self-healing event. The health check is what tells the Hetzner load balancer to stop sending traffic to a node that is going away. Tune it so a leaving node drops out of the target pool before its pods stop answering.

What happens on a node swap#

When the platform replaces a node, the load balancer must notice the old node is gone and register the new one. That registration is not instant. For a few seconds the load balancer may still send traffic to the node that is leaving, or not yet send it to the replacement. During that window, requests that land on the leaving node fail unless another node can serve them.

Two measures close that window: a health check that drops the leaving node quickly, and enough healthy replicas elsewhere to take the traffic.

The health-check annotations#

Set these on the type: LoadBalancer Service (or on the ingress controller's Service). Pick the check protocol first, then the timing.

An http (or https) check calls a path and expects a success status, so it follows the application's readiness. A node whose pod is failing leaves the pool even while its port is still open. Use this whenever the application has a health endpoint.

yaml
		metadata:
  annotations:
    load-balancer.hetzner.cloud/health-check-protocol: http
    load-balancer.hetzner.cloud/health-check-http-path: /healthz
    load-balancer.hetzner.cloud/health-check-interval: "3s"
    load-balancer.hetzner.cloud/health-check-timeout: "2s"
    load-balancer.hetzner.cloud/health-check-retries: "2"
  • Write interval and timeout with a time unit. Use "15s" for 15 seconds or "1m30s" for a minute and a half. A plain number such as 15 is rejected, and the Service fails to reconcile.
  • A shorter interval and fewer retries drop a failing node from the pool sooner, which shrinks the swap window. Do not set them so aggressively that a single slow response ejects a healthy node.

The full annotation set, and the rest of what a Service load balancer exposes, is in .

Keep enough replicas to absorb the traffic#

A tight health check alone is not enough. The traffic a leaving node drops must go somewhere:

  • Run at least two replicas of the exposed workload.
  • Spread them across nodes with a topology spread constraint, so one node drain never removes them all.
  • Add a PodDisruptionBudget so the drain cannot evict the last ready replica.

For the full sequence during an upgrade, both hops that external traffic takes, see . For why nodes are replaced rather than patched, see .

Note

With externalTrafficPolicy: Local, the load balancer already health-checks each node for a local ready pod and stops sending to a node that has none. That interacts with the settings here; see .

Configure a Hetzner load balancer

Tune the control-plane load balancer with topology variables and per-Service load balancers with annotations, including the full annotation reference.

Enable PROXY protocol

Turn on PROXY protocol so the load balancer forwards the real client IP, and configure the backend to read it.