Control-plane metrics
The control plane reports the numbers that tell you whether the cluster's own components are healthy: how long etcd takes to commit a write, whether the scheduler's queue is draining, how many reconciles the controller-manager is behind. All of it is available, and none of it is on the network. These components bind their metrics ports to 127.0.0.1 on purpose, so the numbers stay on the node.
That is why a Prometheus pod cannot collect them: inside a pod, 127.0.0.1 points to the pod itself, not to the node. The System Alloy runs on each node's own network namespace and reaches them there, which is the supported path and the one the config in Collection already covers. This page covers what metrics the control-plane components expose and which of them to watch.
The endpoints
| Component | Port | Scheme | Nodes | Notes |
|---|---|---|---|---|
| etcd | 2381 | HTTP | control planes | No authentication. |
| kube-controller-manager | 10257 | HTTPS | control planes | Bearer token, self-signed serving certificate. |
| kube-scheduler | 10259 | HTTPS | control planes | Bearer token, self-signed serving certificate. |
| kubelet | 10250 | HTTPS | every node | Bearer token; the certificate SAN is the node IP, not 127.0.0.1. |
| cadvisor | 10250 | HTTPS | every node | The kubelet's /metrics/cadvisor path. |
| KubeGate | 8080 | HTTP | control planes | No authentication. |
The API server is the exception in this group: its serving certificate includes 127.0.0.1, so the System Alloy scrapes it on loopback with normal TLS verification. The full catalog, including the data-plane and node daemons, is Reference: metrics per component .
What to watch
Watch etcd first. A slow etcd degrades the whole cluster. Rising fsync latency means the disk cannot keep up with commits. A database size climbing toward its quota ends in a read-only cluster. Frequent leader changes mean the members cannot keep a stable quorum, usually from disk or network latency between control planes.
The controller-manager and scheduler tell you about backlog, not health. Their work-queue depth staying high means reconciles are queuing faster than they finish, so objects stop converging. A scheduler with pending pods and an empty queue is a capacity problem instead, and kube-state-metrics is where you see that.
The kubelet and cadvisor are the per-node and per-container view. The kubelet reports pod lifecycle timings, volume mounts, and its own runtime operations; cadvisor supplies per-container CPU, memory, and network. Together they answer which container on which node is using what.
KubeGate reports the allow and deny decisions it makes in front of the API server. A climbing deny rate means a misconfigured client or something worth investigating, and its audit log has the detail (Log sources on a sealed node ).
Node roles and loopback certificates
The endpoints in the table section come with two collection details, and both are covered in the collection guides. etcd, the controller-manager, the scheduler, and KubeGate exist only on control planes. The System Alloy therefore runs those scrape jobs only on control-plane nodes (Deploy the System Alloy ). The controller-manager, scheduler, and kubelet certificates cannot be verified from loopback, so their scrapes disable certificate verification (Troubleshoot the collectors ).
Once the samples land, alert on them. Rising etcd fsync latency and sustained queue depth are in Platform alert rules , and the control-plane dashboards are in Dashboards for a Syself Autopilot cluster .
Set up Prometheus
Install the Prometheus Operator and one Prometheus that only receives, because the Alloy collectors do all the scraping.
Cluster object metrics with kube-state-metrics
kube-state-metrics turns every Kubernetes object's state into a metric, including the custom NodeConditions the health daemon latches, so you can alert on them.