Platform alert rules

The node health daemon watches every node and fixes what it can on its own. On cloud pools it reboots a broken node and, if the reboot does not help, replaces it from a fresh image; it does this for a Ready=False node and for conditions like a read-only filesystem or a failed integrity check. On bare-metal pools it can only reboot the server, and only when the node reports Ready=False. Every other problem it detects there, a tampered node, a failed integrity check, a failing disk, it reports but does not fix, so on bare metal an alert is the only way you act on it. This page ships those rules, plus the capacity and control-plane rules every cluster should run.

The daemon reports what it finds as node conditions. You alert on them by reading kube_node_status_condition from kube-state-metrics , with node-exporter supplying the capacity metrics. Set those up first.

Node integrity#

VerityCorruption (an integrity check on the sealed OS failed), SealedOSTampered (an unauthorized change in the read-only sealed OS image), and NodeTampered (a protected file changed since boot) are integrity conditions the daemon publishes on every node. SealedOSTampered and NodeTampered are never fixed automatically, on any pool. VerityCorruption is cleared on cloud pools by replacing the node from a fresh image, but on bare metal it is not fixed, so there alerting is the only way to act on it. Alert on all three:

		apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: syself-node-integrity
  namespace: monitoring
  labels:
    prometheus: main
spec:
  groups:
    - name: syself-node-integrity
      rules:
        - alert: NodeVerityCorruption
          expr: kube_node_status_condition{condition="VerityCorruption",status="true"} == 1
          for: 1m
          labels: {severity: critical}
          annotations:
            summary: "sealed OS integrity check failed on {{ $labels.node }}"
            description: "A block on the sealed OS failed its integrity check. On bare-metal pools this does not self-heal, so act on it."
        - alert: NodeSealedOSTampered
          expr: kube_node_status_condition{condition="SealedOSTampered",status="true"} == 1
          for: 1m
          labels: {severity: critical}
          annotations:
            summary: "sealed OS tampering on {{ $labels.node }}"
            description: "An unauthorized change was detected in the read-only sealed OS image. This never self-heals on any pool, so investigate the node."
        - alert: NodeTampered
          expr: kube_node_status_condition{condition="NodeTampered",status="true"} == 1
          for: 1m
          labels: {severity: critical}
          annotations:
            summary: "node tampering detected on {{ $labels.node }}"
            description: "A protected file changed since the boot baseline. This latches; a revert does not clear it. Investigate the node."
	

Ready and the daemon conditions#

The health daemon sets more conditions than the two above. Some of them drive remediation on cloud pools (ReadonlyFilesystem, KernelDeadlock, ServiceNotRecovering); the rest (DisksFailure, CNIUnhealthy, CertRenewalFailing) are alert-only on every pool. On bare metal, all of them alert. Add a rule for each, as a kube_node_status_condition{condition="...",status="true"} == 1. Also alert on Ready itself, so a node that goes NotReady or whose health daemon stops updating surfaces even when nothing else does. The full catalog and which pool remediates each is in Node health conditions .

Capacity#

Catch a cluster running out of room before a workload does:

• CPU and memory: alert when almost all of a node's schedulable CPU or memory is reserved, or when node-exporter shows sustained high memory with little available.
• Disk: alert on a filesystem past ~85% (a /var filling with logs is the classic one), using node-exporter's node_filesystem_avail_bytes.
• PVC full: alert when a volume's used space approaches its capacity, from the kubelet volume metrics.

Control plane#

• etcd: alert on rising fsync latency, a growing database size approaching its quota, and frequent leader changes. These are the early signs of a control plane under strain, from the loopback etcd scrape.
• Saturation: alert on controller-manager and scheduler work-queue depth staying high, which means reconciles are backing up.

The collection tier itself#

Every rule on this page depends on the collectors still running, so alert on them too. A blind spot here is worse than a broken alert, because a silent collector looks exactly like a healthy cluster: no alerts fire, and the graphs simply stop.

• An agent is down. Alert when the System Alloy is missing from a node, not only when it is unhealthy. up on a target that no longer exists disappears rather than going to 0, so compare the count of reporting agents against the node count, or alert on absent() for a series each node should always produce.
• Remote-write is failing. A collector that scrapes correctly but cannot write is losing data on a timer. Alert on remote-write failures and on the pending-samples queue growing, which is the earlier signal.
• The write-ahead log keeps growing. Sustained growth means the ingest rate exceeds what the backend will accept, and the buffer is deferring the loss rather than preventing it.
• Active series jump. A steep climb in series is a cardinality spike in progress, usually from one workload. Alerting on the rate of change catches it before the agent is killed, and Troubleshoot the collectors covers finding the source.
• A target is down on the nodes where it should exist. Scope these per node role. A rule that expects etcd on every node fires forever on workers and teaches everyone to ignore it, which is worse than having no rule.

Certificate expiry#

Alert before a certificate expires, not after. Two sources: the health daemon's CertRenewalFailing node condition for node-level certificates, and, for your ingress certificates, cert-manager's expiry metric. To catch a node certificate before renewal ever fails, read the autopilot.syself.com/certs annotation on the Node, which lists each certificate with its notAfter date. See Rotate and renew certificates .

Keep all of these in Git as PrometheusRule objects so a rebuilt cluster comes back with the same alerts, and route the critical ones to a pager. See Alert routing and receivers .