Detection vs retention

The health daemon detects node problems in real time and fixes them. Your observability stack retains the evidence so you can still investigate after a node is gone. In other words, the platform does the detecting, but the retaining is owned by you.

flowchart LR
  D["Health daemon<br/>detects a problem"] --> R["Reboot; if that fails,<br/>replace the node (cloud)"] --> G["A replaced node's /var<br/>is gone, logs and all"]
  L["You ship logs<br/>and metrics off-node"] -.-> K["Kept in your store,<br/>survives replacement"]

Detection: real time, built in#

The node health daemon runs on every node and watches it in real time. When it finds a problem it sets a NodeCondition, a health flag on the Node object that you can read or alert on, and it drives self-healing. On cloud servers the node is rebooted and, if it does not recover, replaced from a fresh image. On bare-metal servers the machine is rebooted but never replaced with a fresh image, and only a Ready=False node reboots automatically, so every other condition there is alert-only. The daemon also saves a short health report in the node's annotations (storage, disks, certificates, and more), which is diagnostic only and never triggers a fix. The fleet stays healthy with no action from you. See Self-healing and node replacement for the loop, and Node health conditions for the catalog.

Detection tells you something is wrong now and fixes it. It does not tell you why it happened, what the workload was doing, or who touched the API an hour before.

Retention: what replacement erases#

When the platform replaces a node, everything on that node's writable /var goes with it: container logs, journald (the system's own log), the auditd trail (its record of security-relevant events), and on a control plane the API and KubeGate audit streams. The on-node copies rotate and hold only a short recent window, not a full history. So the moment self-healing does its job, the evidence you would investigate with is gone, unless you already shipped it off-node.

That is why retention is yours to run:

• Investigation. To reconstruct a slow request or a crash loop after the node is gone, you need its logs and metrics in a store that outlived it.
• Compliance. An audit trail only counts if it survives. The platform's audit streams are scoped for frameworks like BSI C5, but the framework wants the trail kept off the machine that produced it, precisely so a node loss, or an attacker, cannot erase it.

Keep your own record#

The two are complementary, not redundant: the health daemon keeps the cluster running, and your stores keep the record. Set the retention side up early:

• Ship logs off-node before a node is replaced. See Collect container and journald logs and Ship audit logs off-node .
• Store metrics beyond Prometheus's local window. See Long-term storage and remote-write .
• Alert on the integrity conditions self-healing does not fix, which on bare metal is every one of them. See Platform alert rules .