Cluster object metrics with kube-state-metrics

kube-state-metrics reads the state of every Kubernetes object and exposes it as metrics. It is not about resource usage (that is node-exporter and metrics-server); it is about the objects themselves: how many replicas a Deployment has ready, whether a PVC is bound, whether a Job failed. On a Syself Autopilot cluster it does one extra job that matters: it surfaces the custom node conditions the health daemon sets, so you can alert on node integrity.

It is one Deployment on the pod network, which makes it the one component here that keeps its ServiceMonitor: the Application Alloy discovers and scrapes it like any other cluster component, sharded across its replicas so each endpoint is scraped once.

Install it#

		# The chart does not create the ServiceMonitor by default. Create it, and the
# Application Alloy picks the target up with no further configuration.
prometheus:
  monitor:
    enabled: true
	

		$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
$ helm install kube-state-metrics prometheus-community/kube-state-metrics \
  --namespace monitoring \
  --version <chart-version> \
  --values kube-state-metrics-values.yaml
	

		count(kube_node_status_condition)
	

The node condition metric#

A node health daemon runs on every node and watches for trouble. When it finds a problem it reacts. On cloud pools it reboots the node, and if that does not clear the fault it replaces the node from a fresh image. On bare-metal pools it reboots the server but does not re-image it.

The daemon records what it finds on the Node object in two places. Conditions (status.conditions) are the alertable signal: kube-state-metrics turns them into a metric you can scrape and alert on. A separate health report under autopilot.syself.com/ annotations (/storage, /disks, /certs, /tamper, /services) holds the diagnostic detail; it is for reading, not for alerting. Each annotation is JSON with an updatedAt timestamp, so you can tell how fresh a report is. Read one with kubectl:

		$ kubectl get node <node-name> \
  -o jsonpath='{.metadata.annotations.autopilot\.syself\.com/services}' | jq .
	

The daemon writes conditions like VerityCorruption and NodeTampered onto each Node object. kube-state-metrics turns every node condition, the standard ones and these custom ones, into kube_node_status_condition, with a value of 1 when the condition is set:

		kube_node_status_condition{condition="VerityCorruption", status="true"} == 1
	

This is the bridge from a node condition to an alert, and it matters most for the conditions nothing fixes on its own. SealedOSTampered and NodeTampered are alert-only on every pool, because no reboot or re-image undoes a tampered node. On bare-metal pools only a Ready=False node is fixed automatically (a reboot), so every other condition there, including VerityCorruption and the disk conditions, is yours to alert on. Cloud pools reboot and, if needed, replace a node for more conditions, but the tamper and disk conditions are still alert-only. See Platform alert rules and the full Node health conditions catalog.

Object states worth alerting on#

The same source gives you the raw signals behind most capacity and health alerts:

• Workloads: kube_deployment_status_replicas_available against ..._replicas catches a Deployment that cannot keep its replicas up.
• Storage: kube_persistentvolumeclaim_status_phase catches a PVC stuck Pending.
• Jobs: kube_job_status_failed catches a failed Job or CronJob run.
• Quotas: kube_resourcequota shows a namespace approaching its limit.

Watch the cardinality#

kube-state-metrics emits a series per object, so on a large cluster with many pods, jobs, and PVCs the series count climbs fast. Cardinality is how many distinct time series a metric splits into, and high cardinality is what loads Prometheus. If it becomes a problem, scope it: run kube-state-metrics with a namespace filter, or drop the object kinds and labels you never query on with metric_relabel_configs (relabeling rules that rewrite or discard series before they reach storage). Keep the node condition and workload metrics; drop the ones you do not alert or graph on.