The cluster components Syself Autopilot manages for you are where a whole class of problems becomes visible first: a Hetzner API rate limit that stalls load-balancer reconciles, a volume that will not attach, DNS latency that looks like an application bug, packets dropped by a policy nobody remembers writing. Cilium, Hubble, the Hetzner cloud controller manager, the CSI driver, and CoreDNS all report on themselves, and the [System Alloy](/docs/hetzner/apalla/observability/collection/deploy-the-system-alloy) collects them along with everything else on the node.

## Where each one lives

Two groups, split by which network namespace the endpoint is in.

| Component       | Reached at                         | Nodes                           |
| --------------- | ---------------------------------- | ------------------------------- |
| Cilium agent    | `127.0.0.1:9962`                   | every node                      |
| Hubble          | `127.0.0.1:9965`                   | every node                      |
| Cilium operator | `127.0.0.1:9963`                   | only the nodes running it       |
| CSI node plugin | `127.0.0.1:9189`                   | cloud workers                   |
| Hetzner CCM     | pod IP on `8233`                   | wherever the pods are scheduled |
| CSI controller  | pod IP on `9189`                   | wherever the pods are scheduled |
| hubble-relay    | pod IP on `9966`                   | wherever the pod is scheduled   |
| CoreDNS         | the `kube-dns` endpoints on `9153` | wherever the pods are scheduled |

The Cilium agent, the Cilium operator, and the CSI node plugin run with `hostNetwork: true`, so their metrics are on the node's loopback. The rest run on the pod network, and the agent scrapes the pod IP of the pods on its own node, which spreads the work without any endpoint being scraped twice.

> [!NOTE]
> The CSI node plugin is on the host network because it reads the Hetzner metadata service at its link-local address (`169.254.169.254`), which a pod network namespace cannot reach. Since worker nodes carry public IPs, it binds its metrics strictly to loopback so unauthenticated storage metrics are never published to the internet.

> [!IMPORTANT]
> The Cilium operator, hubble-relay, and the CSI controller each declare more than one container port, and Kubernetes pod discovery emits one target per port. A config that rewrites the address to a fixed metrics port without first selecting the metrics port scrapes each of them twice, which doubles every rate you compute from them. The System Alloy config filters on the port name for this reason.

## What each one tells you

**Cilium agent.** IP address pool consumption per node, which is what you check when pods stop getting addresses; eBPF map pressure; policy enforcement state; and the identity count, which grows with the number of distinct label sets in the cluster.

**Hubble.** Every dropped packet with its reason, which is the main network detection signal on this platform, plus connection-level flows and DNS and HTTP counters. The reason label is what separates a policy denial from a genuine failure. [See flows with Hubble](/docs/hetzner/apalla/observability/network-flows/see-flows-with-hubble) covers the groups the cluster enables, and [Alert on dropped packets](/docs/hetzner/apalla/observability/network-flows/alert-on-drops) turns them into alerts.

**Cilium operator.** Its own health and reconcile activity. It runs as a Deployment, so it exists on some nodes and not others, and a collector that expects it everywhere reports failures on the rest.

**Hetzner CCM.** Cloud API call counts and errors, rate-limit headroom, and load-balancer reconcile results. This is where a Hetzner API rate limit surfaces, and it surfaces here before it surfaces as a stuck Service.

**Hetzner CSI.** Volume provision, attach, detach, and resize results from the controller, and node-level plugin activity from the workers. Watch it when a PVC is slow to bind. Both halves also report the Hetzner API calls behind those operations, so an API problem looks the same here as it does in the CCM.

**CoreDNS.** Query and error rates per zone, plus latency. Rising `SERVFAIL` or latency here explains a surprising number of application timeouts, and [Diagnose pod DNS resolution](/docs/hetzner/apalla/network/debug/diagnose-pod-dns-resolution) is the next step when it does.

## Confirm the data is arriving

Run these against your Prometheus or in Grafana. Each one should return a value per node or per pod rather than nothing:

```text
# IP address capacity per node, from the Cilium agent
sum(cilium_ipam_capacity) by (instance)

# Dropped packets by reason and protocol, from Hubble
sum(rate(hubble_drop_total[5m])) by (reason, protocol)

# Hetzner API requests in flight, from both halves of the CSI driver
sum(hcloud_api_in_flight_requests{job=~"csi-.*"}) by (job, instance)

# CCM reconcile operations by operation
sum(rate(cloud_controller_manager_operations_total{job="ccm"}[5m])) by (op)

# CoreDNS requests by zone
sum(rate(coredns_dns_requests_total{job="coredns"}[5m])) by (zone)
```

If a query returns a doubled rate, check the port-name filter above before you trust the number. If it returns nothing at all, [Troubleshoot the collectors](/docs/hetzner/apalla/observability/collection/troubleshoot-the-collectors) starts from the target list.

Cilium maintains Grafana dashboards for the agent, the operator, and Hubble that read these series directly; importing them is covered in [Dashboards for a Syself Autopilot cluster](/docs/hetzner/apalla/observability/dashboards/dashboards-for-a-syself-cluster).