Data-plane and cluster component metrics

The cluster components Syself Autopilot manages for you are where a whole class of problems becomes visible first: a Hetzner API rate limit that stalls load-balancer reconciles, a volume that will not attach, DNS latency that looks like an application bug, packets dropped by a policy nobody remembers writing. Cilium, Hubble, the Hetzner cloud controller manager, the CSI driver, and CoreDNS all report on themselves, and the System Alloy collects them along with everything else on the node.

Where each one lives#

Two groups, split by which network namespace the endpoint is in.

The Cilium agent, the Cilium operator, and the CSI node plugin run with hostNetwork: true, so their metrics are on the node's loopback. The rest run on the pod network, and the agent scrapes the pod IP of the pods on its own node, which spreads the work without any endpoint being scraped twice.

What each one tells you#

Cilium agent. IP address pool consumption per node, which is what you check when pods stop getting addresses; eBPF map pressure; policy enforcement state; and the identity count, which grows with the number of distinct label sets in the cluster.

Hubble. Every dropped packet with its reason, which is the main network detection signal on this platform, plus connection-level flows and DNS and HTTP counters. The reason label is what separates a policy denial from a genuine failure. See flows with Hubble covers the groups the cluster enables, and Alert on dropped packets turns them into alerts.

Cilium operator. Its own health and reconcile activity. It runs as a Deployment, so it exists on some nodes and not others, and a collector that expects it everywhere reports failures on the rest.

Hetzner CCM. Cloud API call counts and errors, rate-limit headroom, and load-balancer reconcile results. This is where a Hetzner API rate limit surfaces, and it surfaces here before it surfaces as a stuck Service.

Hetzner CSI. Volume provision, attach, detach, and resize results from the controller, and node-level plugin activity from the workers. Watch it when a PVC is slow to bind. Both halves also report the Hetzner API calls behind those operations, so an API problem looks the same here as it does in the CCM.

CoreDNS. Query and error rates per zone, plus latency. Rising SERVFAIL or latency here explains a surprising number of application timeouts, and Diagnose pod DNS resolution is the next step when it does.

Confirm the data is arriving#

Run these against your Prometheus or in Grafana. Each one should return a value per node or per pod rather than nothing:

		# IP address capacity per node, from the Cilium agent
sum(cilium_ipam_capacity) by (instance)
 
# Dropped packets by reason and protocol, from Hubble
sum(rate(hubble_drop_total[5m])) by (reason, protocol)
 
# Hetzner API requests in flight, from both halves of the CSI driver
sum(hcloud_api_in_flight_requests{job=~"csi-.*"}) by (job, instance)
 
# CCM reconcile operations by operation
sum(rate(cloud_controller_manager_operations_total{job="ccm"}[5m])) by (op)
 
# CoreDNS requests by zone
sum(rate(coredns_dns_requests_total{job="coredns"}[5m])) by (zone)
	

If a query returns a doubled rate, check the port-name filter above before you trust the number. If it returns nothing at all, Troubleshoot the collectors starts from the target list.

Cilium maintains Grafana dashboards for the agent, the operator, and Hubble that read these series directly; importing them is covered in Dashboards for a Syself Autopilot cluster .