Skip to main content

Tune health checks for node replacement

Inspect 1.36

Syself Autopilot provisions fresh nodes for every upgrade, patch, and self-healing event. During a node replacement, the Hetzner Load Balancer relies on active health checks to detect when a node is being drained and remove it from the backend pool. Configuring proper health check intervals and timeouts ensures leaving nodes drop out of the target pool before their workloads stop answering requests.

What happens on a node swap

When Syself Autopilot replaces a node, target deregistration is not instantaneous. If health checks are too slow, the Hetzner Load Balancer may continue routing traffic to a terminating node after its pods have already begun shutting down.

To eliminate dropped connections, the load balancer must detect the draining state and drop the node from rotation before local pods terminate, while remaining cluster nodes absorb ongoing traffic.

The health-check annotations

Set these on the type: LoadBalancer Service (or on the ingress controller's Service). Pick the check protocol first, then the timing.

An http (or https) check performs a GET request against the specified path and expects a 2xx or 3xx HTTP status code, reflecting the application's actual readiness. If the application begins failing or enters shutdown, the node is dropped from the load balancer pool even while its NodePort socket remains open. Use this whenever your workload provides a /healthz or readiness endpoint.

yaml
		metadata:
  annotations:
    load-balancer.hetzner.cloud/health-check-protocol: http
    load-balancer.hetzner.cloud/health-check-http-path: /healthz
    load-balancer.hetzner.cloud/health-check-interval: "3s"
    load-balancer.hetzner.cloud/health-check-timeout: "2s"
    load-balancer.hetzner.cloud/health-check-retries: "2"
  • Write interval and timeout with a time unit. Use "15s" for 15 seconds or "1m30s" for a minute and a half. A plain number such as 15 is rejected, and the Service fails to reconcile.
  • A shorter interval and fewer retries drop a failing node from the pool sooner, which shrinks the swap window. Do not set them so aggressively that a single slow response ejects a healthy node.

For the complete list of supported Service annotations see .

Workload redundancy during node drains

Tuning load balancer health checks ensures fast target removal, but workloads must also be designed to absorb traffic smoothly during node evictions:

  • Multiple replicas: Run at least two replicas of any externally exposed service.
  • Topology spread: Spread pods across nodes using topologySpreadConstraints⁠ so a single node replacement never evicts all replicas simultaneously.
  • PodDisruptionBudgets: Configure a to prevent the drain controller from evicting the last remaining replica before a replacement is ready.

To learn how traffic routes through ingress controllers and load balancers during cluster upgrades, see . For details on the underlying node lifecycle, see .

Note

When externalTrafficPolicy: Local is enabled, the Hetzner Load Balancer checks whether a healthy pod exists on the node itself rather than just checking node reachability. This prevents traffic from landing on nodes that have no local endpoints. See .

Configure a Hetzner load balancer

Tune the control-plane load balancer with topology variables and per-Service load balancers with annotations, including the full annotation reference.

Enable PROXY protocol

Turn on PROXY protocol so the load balancer forwards the real client IP, and configure the backend to read it.