Troubleshoot load balancers and service traffic
Two things go wrong with an exposed Service: it never receives an external address, or it receives one but traffic does not reach the pod. Work them in that order, because there is no value in chasing dropped traffic on a Service that has no load balancer yet. The load balancer itself comes from the Hetzner cloud-controller-manager (CCM), a pod in kube-system that watches Service objects and calls the Hetzner API to create and update the load balancer.
flowchart TD
A["Service exposed"] --> B{"EXTERNAL-IP assigned?"}
B -->|No, stuck pending| C["The CCM has not made the LB.<br/>Read the Service events."]
B -->|Yes, but traffic drops| D{"Hubble shows a DROP<br/>with a policy reason?"}
D -->|Yes| E["A NetworkPolicy is denying it.<br/>Fix the policy."]
D -->|No| F{"App sees only the LB address?"}
F -->|Yes| G["Turn on PROXY protocol<br/>on both ends."]
F -->|No| H["Check the NodePort datapath,<br/>then the node."]The Service is stuck on <pending>#
An EXTERNAL-IP that stays <pending> means the CCM has not created the load balancer. The reason is almost always on the Service's own events:
$ kubectl describe svc <name>
Read the events at the bottom and match them:
| Event on the Service | Cause | Fix |
|---|---|---|
| Annotation rejected, reconcile fails | a value in the wrong format, like a bare 15 where a Go duration "15s" is required | fix the annotation value; see Configure a Hetzner load balancer |
| Hetzner API auth or permission error | the CCM's Hetzner token is missing, wrong, or lacks permission | check the CCM pod logs in kube-system and the token it uses |
| Load balancer made, but no targets | no pod matches the Service selector and is ready | fix the selector or the pods' readiness |
The Service has an IP but traffic is dropped#
Once the address exists, trace the path from the edge inward.
Check for a policy drop #
A NetworkPolicy or CiliumNetworkPolicy denying the traffic is the most common cause. Watch dropped flows:
$ hubble observe --verdict DROPPED --to-pod <pod> --last 100
A DROPPED verdict with a policy reason means a policy is blocking it. See Segment with network policies .
Check the source address the pod sees #
If the application rejects requests because they all appear to come from the load balancer, the client address is being lost. By default a pod sees the load balancer's address, not the client's. Enable PROXY protocol on the Service and the backend together.
Warning
PROXY protocol must match on both ends. If the load balancer sends the PROXY header but the backend does not expect it, or the reverse, the connection is dropped or the backend reads a corrupted request. A KubeGate host policy that admits clients by their real source address also drops them until PROXY protocol is on, because until then the only source it sees is the load balancer. Set it on the Service annotation and the backend in the same change.
Confirm the datapath port answers #
The CCM reaches pods through a NodePort, a port opened on every node that forwards to the Service's pods. Confirm the Service has a NodePort and that a backend responds on it from another pod in the cluster. If pod-to-pod itself is broken, the problem is below the Service; run a Cilium connectivity test .
When it is a node-level problem#
If several Services on the same node misbehave, or an entire node's traffic is affected, the Service is not the problem. Stop tracing the Service and debug the node.