kube-state-metrics reads the state of every Kubernetes object and exposes it as metrics. It is not about resource usage (that is node-exporter and metrics-server); it is about the objects themselves: how many replicas a Deployment has ready, whether a PVC is bound, whether a Job failed. On a Syself Autopilot cluster it does one extra job that matters: it surfaces the custom node conditions the health daemon sets, so you can alert on node integrity.

It is one Deployment on the pod network, which makes it the one component here that keeps its `ServiceMonitor`: the [Application Alloy](/docs/hetzner/apalla/observability/collection/deploy-the-application-alloy) discovers and scrapes it like any other cluster component, sharded across its replicas so each endpoint is scraped once.

## Install it

<Steps>
<Step>Write the values file</Step>

```yaml title="kube-state-metrics-values.yaml"
# The chart does not create the ServiceMonitor by default. Create it, and the
# Application Alloy picks the target up with no further configuration.
prometheus:
  monitor:
    enabled: true
```

<Step>Install the chart</Step>

```console
$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
$ helm install kube-state-metrics prometheus-community/kube-state-metrics \
  --namespace monitoring \
  --version <chart-version> \
  --values kube-state-metrics-values.yaml
```

Pin `--version` to a known chart release so an update does not change behavior under you.

<Step>Confirm the series arrive</Step>

The Application Alloy does the scraping, so check the store rather than the exporter. In Prometheus, query:

```text
count(kube_node_status_condition)
```

A number in the hundreds is normal, because it is one series per node per condition.

> [!NOTE]
> `kube-state-metrics` is scraped via the `ServiceMonitor` by the [Application Alloy](/docs/hetzner/apalla/observability/collection/deploy-the-application-alloy) (installed in chapter 04). If you are following this guide sequentially and have not installed the Application Alloy yet, the query returns zero until that collector is running.

</Steps>

## The node condition metric

A node health daemon runs on every node and watches for trouble. When it finds a problem it reacts. On cloud pools it reboots the node, and if that does not clear the fault it replaces the node from a fresh image. On bare-metal pools it reboots the server but does not re-image it.

The daemon records what it finds on the Node object in two places. Conditions (`status.conditions`) are the alertable signal: kube-state-metrics turns them into a metric you can scrape and alert on. A separate health report under `autopilot.syself.com/` annotations (`/storage`, `/disks`, `/certs`, `/tamper`, `/services`) holds the diagnostic detail; it is for reading, not for alerting. Each annotation is JSON with an `updatedAt` timestamp, so you can tell how fresh a report is. Read one with `kubectl`:

```console
$ kubectl get node <node-name> \
  -o jsonpath='{.metadata.annotations.autopilot\.syself\.com/services}' | jq .
```

The daemon writes conditions like `VerityCorruption` and `NodeTampered` onto each Node object. kube-state-metrics turns every node condition, the standard ones and these custom ones, into `kube_node_status_condition`, with a value of `1` when the condition is set:

```text
kube_node_status_condition{condition="VerityCorruption", status="true"} == 1
```

This is the bridge from a node condition to an alert, and it matters most for the conditions nothing fixes on its own. `SealedOSTampered` and `NodeTampered` are alert-only on every pool, because no reboot or re-image undoes a tampered node. On bare-metal pools only a `Ready=False` node is fixed automatically (a reboot), so every other condition there, including `VerityCorruption` and the disk conditions, is yours to alert on. Cloud pools reboot and, if needed, replace a node for more conditions, but the tamper and disk conditions are still alert-only. See [Platform alert rules](/docs/hetzner/apalla/observability/alerting/platform-alert-rules) and the full [Node health conditions](/docs/hetzner/apalla/reference/node-health-conditions) catalog.

## Object states worth alerting on

The same source gives you the raw signals behind most capacity and health alerts:

- **Workloads:** `kube_deployment_status_replicas_available` against `..._replicas` catches a Deployment that cannot keep its replicas up.
- **Storage:** `kube_persistentvolumeclaim_status_phase` catches a PVC stuck `Pending`.
- **Jobs:** `kube_job_status_failed` catches a failed Job or CronJob run.
- **Quotas:** `kube_resourcequota` shows a namespace approaching its limit.

## Watch the cardinality

kube-state-metrics emits a series per object, so on a large cluster with many pods, jobs, and PVCs the series count climbs fast. Cardinality is how many distinct time series a metric splits into, and high cardinality is what loads Prometheus. If it becomes a problem, scope it: run kube-state-metrics with a namespace filter, or drop the object kinds and labels you never query on with `metric_relabel_configs` (relabeling rules that rewrite or discard series before they reach storage). Keep the node condition and workload metrics; drop the ones you do not alert or graph on.