Cluster object metrics with kube-state-metrics

kube-state-metrics reads the state of every Kubernetes object and exposes it as metrics. It is not about resource usage (that is node-exporter and metrics-server); it is about the objects themselves: how many replicas a Deployment has ready, whether a PVC is bound, whether a Job failed. On a Syself Autopilot cluster it does one extra job that matters: it surfaces the custom node conditions the health daemon sets, so you can alert on node integrity.

kube-state-metrics runs as a normal Deployment on the pod network. Unlike the control-plane components, which keep their metrics on each node's loopback, it is reachable from anywhere in the cluster. It ships a ServiceMonitor, and the Application Alloy discovers and scrapes it like any other application.

Install it

		# The chart does not create the ServiceMonitor by default. Create it, and the
# Application Alloy picks the target up with no further configuration.
prometheus:
  monitor:
    enabled: true
	

		$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
$ helm install kube-state-metrics prometheus-community/kube-state-metrics \
  --namespace monitoring \
  --version <chart-version> \
  --values kube-state-metrics-values.yaml
	

		count(kube_node_status_condition)
	

The node condition metric

A node health daemon runs on every node and watches for trouble. It does not fix anything itself: it reports what it finds, and Syself Autopilot reacts to some conditions automatically.

The daemon records what it finds on the Node object in two places. Conditions (status.conditions) are the alertable signal: kube-state-metrics turns them into a metric you can scrape and alert on. A separate health report under autopilot.syself.com/ annotations (/storage, /disks, /certs, /tamper, /services) holds the diagnostic detail; it is for reading, not for alerting. Each annotation is JSON with an updatedAt timestamp, so you can tell how fresh a report is. Read one with kubectl:

		$ kubectl get node <node-name> \
  -o jsonpath='{.metadata.annotations.autopilot\.syself\.com/services}' | jq .
	

The daemon writes conditions like VerityCorruption and NodeTampered onto each Node object. kube-state-metrics turns every node condition, the standard ones and these custom ones, into kube_node_status_condition, with a value of 1 when the condition is set:

		kube_node_status_condition{condition="VerityCorruption", status="true"} == 1
	

This is the bridge from a node condition to an alert, and it matters most for the conditions Syself Autopilot does not fix on its own. Which conditions are remediated automatically and which are alert-only on each pool is covered in Platform alert rules , and the full catalog is in Node health conditions .

Object states worth alerting on

Beyond node conditions, kube-state-metrics gives you the raw signals behind most capacity and health alerts:

• Workloads: kube_deployment_status_replicas_available against ..._replicas catches a Deployment that cannot keep its replicas up.
• Storage: kube_persistentvolumeclaim_status_phase catches a PVC stuck Pending.
• Jobs: kube_job_status_failed catches a failed Job or CronJob run.
• Quotas: kube_resourcequota shows a namespace approaching its limit.

Watch the cardinality

kube-state-metrics emits a series per object, so on a large cluster with many pods, jobs, and PVCs the series count climbs fast. Cardinality is how many distinct time series a metric splits into, and high cardinality is what loads Prometheus. If it becomes a problem, scope it: run kube-state-metrics with a namespace filter, or drop the object kinds and labels you never query on with metric_relabel_configs (relabeling rules that rewrite or discard series before they reach storage). Keep the node condition and workload metrics and drop the ones you do not alert or graph on.