The health daemon detects node problems in real time and fixes them. Your observability stack retains the evidence so you can still investigate after a node is gone. In other words, the platform detects and you retain. ```mermaid flowchart LR D["Health daemon
detects a problem"]:::platform --> R["Reboot; if that fails,
replace the node (cloud)"]:::platform --> G["A replaced node's /var
is gone, logs and all"]:::data L(["You ship logs
and metrics off-node"]):::you -.-> K[(Kept in your store,
survives replacement)]:::data ``` ## Detection: real time, built in The node health daemon runs on every node and watches it in real time. When it finds a problem it sets a NodeCondition, a health flag on the Node object that you can read or alert on. Syself Autopilot then acts on the conditions that are wired for remediation. The daemon also saves a short health report in the node's annotations (storage, disks, certificates, and more). That report is diagnostic only and never triggers a fix. See [Node health conditions](/docs/hetzner/apalla/reference/node-health-conditions) for the catalog, [kube-state-metrics](/docs/hetzner/apalla/observability/metrics/kube-state-metrics) for reading them as metrics, and [Platform alert rules](/docs/hetzner/apalla/observability/alerting/platform-alert-rules) for alerting on them. [Self-healing and node replacement](/docs/hetzner/apalla/concepts/operations/self-healing-and-node-replacement) covers what the platform does about them. Detection tells you something is wrong now. For the few conditions wired for remediation, Syself Autopilot acts on it. It does not tell you why it happened, what the workload was doing, or who touched the API an hour before. ## Retention: what replacement erases When the platform replaces a node, everything on that node's writable `/var` goes with it: container logs, journald (the system's own log), the auditd trail (its record of security-relevant events), and on a control plane the API and KubeGate audit streams. The on-node copies rotate and hold only a short recent window, not a full history. So the moment self-healing does its job, the evidence you would investigate with is gone, unless you already shipped it off-node. That is why retention is yours to run: - **Investigation.** To reconstruct a slow request or a crash loop after the node is gone, you need its logs and metrics in a store that outlived it. - **Compliance.** An audit trail only counts if it survives. The platform's audit streams are scoped for frameworks like BSI C5, but the framework wants the trail kept off the machine that produced it, so neither a node loss nor an attacker can erase it. ## Keep your own record Detection and retention are complementary, not redundant. The health daemon keeps the cluster running, and your stores keep the record. Set the retention side up early: - Ship logs off-node before a node is replaced. See [Collect container and journald logs](/docs/hetzner/apalla/observability/logs/collect-container-and-journald-logs) and [Ship audit logs off-node](/docs/hetzner/apalla/observability/logs/ship-audit-logs). - Store metrics beyond Prometheus's local window. See [Long-term storage and remote-write](/docs/hetzner/apalla/observability/metrics/long-term-storage-and-remote-write). - Alert on the integrity conditions self-healing does not fix, which on bare metal is every one of them. See [Platform alert rules](/docs/hetzner/apalla/observability/alerting/platform-alert-rules).