Skip to main content

Reference: metrics per component

Inspect 1.36

Every Syself Autopilot component reports its own metrics on a small web endpoint. Whether Prometheus can reach that endpoint depends on where the endpoint binds. This page lists each component, its port and scheme, and which of two collection methods reaches it.

The endpoints fall into two groups. Some sit on the pod network, where an ordinary scrape reaches them. The rest bind only to 127.0.0.1 on the node, which keeps their metrics private to that machine.

127.0.0.1 is the loopback address, and binding to it keeps an endpoint off the network. Inside a pod, 127.0.0.1 means the pod itself, not the node, so an ordinary scrape cannot reach a node's loopback endpoint. A host-network collector runs on the node itself, so it can.

Every endpoint on this page is collected by the , which runs on each node's own network namespace and therefore reaches both groups. Your own workloads are the exception: they go to the instead, for the reasons in .

This page is the inventory. For what the series from each component actually tell you, see and .

Reachable on the pod network#

These expose metrics through a Service or their pod IP, so any collector on the pod network reaches them. The System Alloy scrapes the ones running on its own node, which spreads the work across agents without scraping any endpoint twice.

Component Service Port What it tells you
kube-apiserver the kubernetes Service 6443 Request rates, latencies, and error codes. Scrape /metrics with a bearer token that has get on nonResourceURLs: ["/metrics"]. It also listens on each control plane's loopback, which is where the System Alloy scrapes it.
CCM ccm-metrics (headless) 8233 Hetzner API call counts and errors, rate-limit headroom, load-balancer reconcile results.
hubble-relay hubble-relay-metrics 9966 Relay health, connected peers, gRPC traffic.
CSI controller csi-controller-metrics (headless) 9189 Volume provision, attach, detach, and resize results, and the Hetzner API calls behind them.
CoreDNS kube-dns 9153 DNS query and error rates.
metrics-server metrics-server 443 Usually read through the metrics.k8s.io aggregated API, not scraped directly.

Reachable only over the node's loopback#

Most of these bind to 127.0.0.1, so only a collector sharing the node's network namespace reaches them. The kubelet is the exception: it listens on the node IP, not loopback, but the host firewall admits port 10250 only from other nodes and the metrics-server pod, so an ordinary Prometheus pod is dropped and the same agent reaches it over the node's own loopback. The scheme column below is whether the endpoint serves plain HTTP or HTTPS, and whether it needs a token.

The Nodes column is also the gating rule: a job for a control-plane-only or worker-only component has to be restricted to those nodes, or it fails permanently on the rest.

Component Loopback port Scheme Nodes
etcd 2381 HTTP, no auth control planes
kube-controller-manager 10257 HTTPS, bearer token control planes
kube-scheduler 10259 HTTPS, bearer token control planes
kubelet 10250 HTTPS, bearer token every node (SAN is the node IP, so the scraper uses insecure_skip_verify)
cadvisor 10250 HTTPS, bearer token every node; the kubelet's /metrics/cadvisor path, for per-container CPU, memory, and network
KubeGate 8080 HTTP, no auth control planes
Cilium agent 9962 HTTP every node
Cilium operator 9963 HTTP only nodes running it; others refuse the connection
Hubble 9965 HTTP every node
CSI node plugin 9189 HTTP cloud workers only; no Service, no container port. Host network because it reads the Hetzner metadata service at 169.254.169.254, which a pod network namespace cannot reach; it binds metrics to loopback so they stay off the worker's public IP
syself-agent (health daemon) 20257 HTTP every node
syself-tunnel-agent 8182 HTTP every node
syself-tunnel-server 8181 HTTP control planes
syself-proxy (failover) 9587 HTTP workers
containerd 1338 HTTP every node; serves /v1/metrics
Note

Several loopback endpoints carry no authentication (etcd on 2381, KubeGate on 8080, syself-agent on 20257). The host-network agent that reaches them is a privileged collector: keep it in a namespace only your platform team can write to, and review its scrape config like a firewall rule.

Important

The Cilium operator, hubble-relay, and the CSI controller each declare more than one container port, and Kubernetes pod discovery emits one target per port. A config that rewrites the address to a fixed metrics port without first selecting that port scrapes the endpoint twice and doubles every rate computed from it.

Not scrapeable#

  • Cilium Envoy's metrics listener is (its metrics on port 9964 are turned off and it does no L7 proxying here), so it exposes no metrics.
  • GPU nodes ship no GPU metrics on their own. To get GPU utilization and memory, install the NVIDIA DCGM exporter yourself and scrape it like any application metric.

For the full port inventory, including non-metrics listeners, see .