Detection vs retention

The health daemon detects node problems in real time and fixes them. Your observability stack retains the evidence so you can still investigate after a node is gone. In other words, the platform detects and you retain.

flowchart LR
  D["Health daemon<br/>detects a problem"] --> R["Reboot; if that fails,<br/>replace the node (cloud)"] --> G["A replaced node's /var<br/>is gone, logs and all"]
  L["You ship logs<br/>and metrics off-node"] -.-> K["Kept in your store,<br/>survives replacement"]

Detection: real time, built in

The node health daemon runs on every node and watches it in real time. When it finds a problem it sets a NodeCondition, a health flag on the Node object that you can read or alert on. Syself Autopilot then acts on the conditions that are wired for remediation.

The daemon also saves a short health report in the node's annotations (storage, disks, certificates, and more). That report is diagnostic only and never triggers a fix. See Node health conditions for the catalog, kube-state-metrics for reading them as metrics, and Platform alert rules for alerting on them. Self-healing and node replacement covers what the platform does about them.

Detection tells you something is wrong now and fixes it. It does not tell you why it happened, what the workload was doing, or who touched the API an hour before.

Retention: what replacement erases

When the platform replaces a node, everything on that node's writable /var goes with it: container logs, journald (the system's own log), the auditd trail (its record of security-relevant events), and on a control plane the API and KubeGate audit streams. The on-node copies rotate and hold only a short recent window, not a full history. So the moment self-healing does its job, the evidence you would investigate with is gone, unless you already shipped it off-node.

That is why retention is yours to run:

• Investigation. To reconstruct a slow request or a crash loop after the node is gone, you need its logs and metrics in a store that outlived it.
• Compliance. An audit trail only counts if it survives. The platform's audit streams are scoped for frameworks like BSI C5, but the framework wants the trail kept off the machine that produced it, so neither a node loss nor an attacker can erase it.

Keep your own record

Detection and retention are complementary, not redundant. The health daemon keeps the cluster running, and your stores keep the record. Set the retention side up early:

• Ship logs off-node before a node is replaced. See Collect container and journald logs and Ship audit logs off-node .
• Store metrics beyond Prometheus's local window. See Long-term storage and remote-write .
• Alert on the integrity conditions self-healing does not fix, which on bare metal is every one of them. See Platform alert rules .