Node components on a Syself Autopilot cluster expose Prometheus metrics, and where a component binds decides how you scrape it and whether it is ever reachable from the internet. A component on the **pod network** binds a pod IP and sits behind a ClusterIP Service, so you scrape it like any in-cluster endpoint. A component on the **host network** binds `127.0.0.1`, so nothing outside the node can reach it, whatever the firewall says, and you scrape it from a node-local collector. The host-network rule has no exception: every host-network metrics listener binds loopback, except the kubelet, which binds the node IP but requires authentication. Node IPs are public here, so an unauthenticated `/metrics` on `0.0.0.0` would publish on a routable address. ## Health-daemon metrics The health daemon is part of the Syself node agent (`syself-node`). It is the node-problem-detector: it turns node problems into Kubernetes NodeConditions and events, and it exports Prometheus metrics on `127.0.0.1:20257` on every node. The endpoint is plain HTTP with no auth, which is why it stays on loopback. | Metric | Type | What it tells you | | --------------------------------- | --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `problem_counter` | counter | A problem fired. Both a one-shot event (a single disk read error, a corrected memory error) and a NodeCondition turning `True` increment it. A condition that is already `True` does not increment it again. | | `syself_node_unit_restarts{unit}` | gauge, per unit | systemd's restart count since boot for each watched unit: `kubelet`, `containerd`, and the tunnel agent on every node; the failover proxy on a worker; the tunnel server and source-IP helper on a control plane. A climbing series is restart activity, and a fast climb on `kubelet`, `containerd` or the failover proxy is what sets `ServiceNotRecovering`. | | `problem_gauge` | gauge | The current state of each NodeCondition the daemon sets, exported on the same `20257` endpoint. | ## Where each component binds ### Host network (loopback) These share the node's network namespace and bind `127.0.0.1`. Scrape them from a node-local, host-network DaemonSet. | Component | Port | Nodes | Scheme and auth | | ---------------------------- | ------- | --------------- | -------------------------------------------------------------------------------------------------------- | | cilium-agent | `9962` | all | plain HTTP | | Hubble (in cilium-agent) | `9965` | all | plain HTTP | | cilium-operator | `9963` | where scheduled | plain HTTP | | containerd | `1338` | all | plain HTTP, containerd's own metrics (image pull, snapshotter) | | etcd | `2381` | control planes | plain HTTP, no auth | | kube-controller-manager | `10257` | control planes | HTTPS, bearer token | | kube-scheduler | `10259` | control planes | HTTPS, bearer token | | KubeGate | `8080` | control planes | plain HTTP, no auth | | syself-node failover proxy | `9587` | workers | plain HTTP | | syself-node health daemon | `20257` | all | plain HTTP, no auth | | syself-node tunnel server | `8181` | control planes | plain HTTP | | syself-node tunnel agent | `8182` | all | plain HTTP | | csi-node (hcloud-csi driver) | `9189` | hcloud workers | plain HTTP. `hostNetwork` DaemonSet with no Service, so the node-local scrape is the only way in. | | kubelet | `10250` | all | HTTPS with auth. Binds the node IP, not loopback: the one metrics port the host firewall allows inbound. | ### Pod network (Service) These bind a pod IP behind a ClusterIP Service. Scrape them like any in-cluster endpoint, with a Service or a ServiceMonitor. | Component | Port | Exposed as | Notes | | -------------- | -------------------------------- | ------------------------------------------- | ---------------------------------------------------------------------------------- | | CCM | `8233` | `ccm-metrics` Service (headless) | Cloud Controller Manager. | | CoreDNS | `9153` | `kube-dns` Service | | | hubble-relay | `9966` | `hubble-relay-metrics` Service | The one Cilium component on the pod network. | | metrics-server | `10250` container, `443` Service | `metrics-server` Service | HTTPS with auth. Usually read through the `metrics.k8s.io` aggregated API instead. | | csi-controller | `9189` | `csi-controller-metrics` Service (headless) | The controller half of the CSI driver, on the pod network unlike the node half. | ## Why every listener is loopback or authenticated Nodes carry public IPs and there is no private network. Almost every metrics endpoint is unauthenticated (etcd `2381`, KubeGate `8080`, the health daemon `20257`, and all of Cilium serve plain HTTP with no token), so a host-network process that bound `0.0.0.0` would publish that data on a routable, internet-facing address. Binding `127.0.0.1` means the host firewall is not the only thing in front of these endpoints: if a firewall rule is ever mis-edited or fails to apply, a loopback listener is still unreachable. Cilium's Envoy metrics listener is the exception, and it is off for a different reason. These clusters [block L7 network policy](/docs/hetzner/apalla/concepts/internals/networking), so Envoy does no L7 work and its metrics on `9964` would be empty. That listener is also the one the upstream chart cannot bind to loopback, so rather than publish an empty endpoint on the node address, those Envoy metrics are disabled. Nothing useful is lost. > [!WARNING] > Never add a metrics port to a host-firewall policy (a `CiliumClusterwideNetworkPolicy`). If you think you need to, you are scraping from a pod. Scrape from the node-local DaemonSet instead. ## What to scrape and how Scrape the loopback endpoints from a host-network DaemonSet. Set `hostNetwork: true` and `dnsPolicy: ClusterFirstWithHostNet` so the collector shares each node's network namespace and dials `127.0.0.1` directly. That path never leaves the node, so it needs no firewall rule. A scraper pod that is not on the host network cannot reach these ports. Its packet leaves the pod namespace and arrives at the node's host endpoint, where the default-deny host firewall drops anything not explicitly allowed. That is why a ServiceMonitor pointed at `127.0.0.1` on the node cannot work, and why the loopback endpoints have no Service to select. The pod-network endpoints are the opposite: they are ordinary ClusterIP Services, reachable from any namespace, and a normal Service or ServiceMonitor picks them up. Syself does not ship the Prometheus Operator CRDs, so if you use ServiceMonitors you install the operator and write your own. ## Map a firing condition to its metric When a NodeCondition fires and you want the time series behind it, the health daemon on `20257` is the source. The condition itself shows in `problem_gauge` there. A one-shot event that set no condition shows as an increment on `problem_counter`. A repair loop restarting a broken unit shows on `syself_node_unit_restarts{unit}`, and a fast climb there is what escalates to `ServiceNotRecovering`. > [!TIP] > Read the condition first with the health report annotations, then correlate with the metric. See [Node health conditions](/docs/hetzner/apalla/reference/node-health-conditions#how-to-read-them-on-a-node) for the one-liners that print the `autopilot.syself.com/*` report. ## Related - [Ports and listeners](/docs/hetzner/apalla/reference/ports-and-listeners): every port a node opens, not just the metrics ones. - [Node health conditions](/docs/hetzner/apalla/reference/node-health-conditions): the conditions the health-daemon metrics correspond to, and which ones trigger replacement. - [Collect logs with the log collector](/docs/hetzner/apalla/support/collect-logs-with-the-log-collector): the diagnostics bundle to send when a metric points at a problem you cannot resolve.