Every node in a Syself Autopilot cluster runs a health daemon. The daemon continuously checks the node and reports problems as Kubernetes NodeConditions (flags on a Node object that describe its health state), events, and metrics. It also writes a JSON health report to the node, as a set of `autopilot.syself.com/*` annotations, one key per domain. What a condition does when it fires depends on the pool. A few conditions trigger automatic machine replacement; the rest are there for visibility and alerting. [Automatic replacement](#automatic-replacement) lists the exact set per pool. ## The two layers: report, then react The daemon and the replacement logic are separate. Keep them distinct. **The daemon reports.** It detects a problem and writes it as a NodeCondition, an event, and a metric. That is all it does for most problems. It does not cordon, drain, or replace nodes. It does not set the `Ready` condition; the kubelet owns that. **Something else reacts.** A few different actors can respond: - **In-place repair** restarts a broken unit (kubelet, containerd) on the node. - **Syself's replacement logic** reboots the node and, if that fails, replaces it. It is the only automatic actor that deletes a node, and it watches `Ready` on bare metal, plus a few daemon conditions on cloud pools. - **You** can force a replacement by hand at any time: delete the `Machine` object, and the controllers drain the node and bring up a fresh one in its place. - **A human** reads an alert and decides what to do. The mapping from condition to actor is fixed. It is called the **disposition** and is defined in the next section. ## Events vs conditions Every monitor rule is one of two types. **Temp (temporary) rules** emit a one-shot Node event and increment the `problem_counter` metric. They do not set a condition. Use these for things that happen once and pass (a single disk read error, a corrected memory error). **Perm (permanent) rules** set a NodeCondition on the Node and hold it. `status: True` means the problem is present. `status: False` means healthy. If the daemon stops, its conditions keep their last value and go stale; they do not clear automatically. An event says "this happened". A condition says "this is the state right now". ## Dispositions: what happens when a condition fires Every condition has exactly one disposition. The disposition decides what happens when the condition turns `True`. > [!NOTE] > The disposition below is what a condition means. Whether it also triggers automatic replacement depends on the pool, and that split lives in [Automatic replacement](#automatic-replacement): bare-metal pools act on `Ready` alone, cloud pools on `Ready` plus a few daemon conditions. | Disposition | What happens | Conditions with this disposition | | --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **repair** | The daemon restarts the unit in place. If restarting stops helping, it escalates via `ServiceNotRecovering` (which has the **alert** disposition). A repair condition never triggers machine replacement directly. | `KubeletUnhealthy`, `ContainerRuntimeUnhealthy` | | **alert** | Meant to reach a human. Out of the box nothing pages anyone: the condition is set, and you build the alert on it (see [How to alert on a condition](#how-to-alert-on-a-condition)). On bare-metal pools no automatic node action follows: either a reboot would not fix the problem, or the same cause can hit many nodes at once, making replacement counterproductive. On cloud pools, `ReadonlyFilesystem`, `KernelDeadlock`, and `ServiceNotRecovering` also trigger reboot, then replace (see [Automatic replacement](#automatic-replacement)). | `ReadonlyFilesystem`, `KernelDeadlock`, `ServiceNotRecovering`, `DisksFailure`, `CperHardwareErrorFatal`, `GpuFallenOffBus`, `CNIUnhealthy`, `ClockNotSynchronized`, `CertRenewalFailing`, `KubeletHeartbeatFailed`, `NodeInfoStale`, `APIServerSourceIPUnavailable` | | **operability** | A Syself service is down, but the node itself is healthy. Surfaced at low priority. No automatic node action. | `NodeServiceDown`, `TunnelDisconnected`, `ProxyNotServing`, `GpuActivationFailed`, `DiskUsageHigh`, `DiskWearHigh`, `DiskTemperatureHigh` | | **security** | A report-only integrity signal; a human investigates. On cloud pools, `VerityCorruption` also triggers reboot, then replace (see [Automatic replacement](#automatic-replacement)); the kernel's dm-verity mode is still the real protection. `SealedOSTampered` and `NodeTampered` are report-only on every pool. | `VerityCorruption`, `SealedOSTampered`, `NodeTampered` | ## In-place repair Before anything heavier runs, a broken critical unit is restarted on the node. **kubelet**: the health daemon probes `/healthz` every 10 seconds. On a failed probe it restarts `kubelet.service` and waits a 1-minute cooldown before trying again. **containerd**: the daemon probes the CRI socket (CRI, the Container Runtime Interface, connects kubelet to the container runtime) every 10 seconds. On failure it restarts `containerd.service` with a 2-minute cooldown. systemd also restarts these units on its own (`Restart=always`). Every restart increments systemd's `NRestarts` counter for that unit. **When repair is not working.** The `service-not-recovering` check runs every minute. For the critical units (kubelet, containerd, and on a worker the failover proxy), it sets `ServiceNotRecovering = True` when a unit is either in the systemd `failed` state, or flapping (its `NRestarts` went up by 3 or more since the last check). `ServiceNotRecovering` has the **alert** disposition, so it surfaces for you to alert on (see [How to alert on a condition](#how-to-alert-on-a-condition)). The daemon also exports `syself_node_unit_restarts{unit}` as a Prometheus metric, so you can see repair activity as a time series. ## Automatic replacement Syself's replacement logic is the only thing that deletes and re-creates a machine. On bare-metal pools it watches only the `Ready` condition. On cloud pools it also watches four of the daemon's own conditions: `ReadonlyFilesystem`, `KernelDeadlock`, `ServiceNotRecovering`, and `VerityCorruption`. Rebooting the node fixes all of these on a fresh VM. ### Cloud pools (HCloud control plane and HCloud workers) | Watched condition | Unhealthy when | Must hold for | Then | | ---------------------- | -------------- | ------------- | -------------------- | | `Ready` | `Unknown` | 600s | reboot, then replace | | `Ready` | `False` | 300s | reboot, then replace | | `ReadonlyFilesystem` | `True` | 120s | reboot, then replace | | `KernelDeadlock` | `True` | 120s | reboot, then replace | | `ServiceNotRecovering` | `True` | 60s | reboot, then replace | | `VerityCorruption` | `True` | 60s | reboot, then replace | Syself reboots the server once, allowing up to 180 seconds for the reboot to complete. If the condition is still true after it has held for the time shown above, Syself replaces the machine with a new VM. ### Bare-metal pools (bare-metal control plane and bare-metal workers) | Watched condition | Unhealthy when | Must hold for | Then | | ----------------- | -------------- | ------------- | ------------------------------------ | | `Ready` | `False` | 300s | reboot (up to 2×), then re-provision | On bare metal, `Ready=Unknown` alerts a human instead of triggering remediation. It is not wired into automatic replacement. > [!NOTE] > Automatic replacement on bare metal is deliberately narrow right now: only `Ready=False` triggers it. Re-provisioning the same physical host will not fix a bad disk or failing hardware, so the daemon's own conditions alert a human instead. Widening the set of conditions that trigger automatic action on bare metal is planned for a future Kubernetes minor track. Syself reboots the physical server up to two times. If the node is still unhealthy after 300 seconds, Syself re-provisions the same physical host (wipe and reinstall). The longer timeout gives hardware more time to stabilise after a reboot before Syself escalates to re-provisioning. ### Safety gates and startup grace These prevent automatic replacement from making an incident worse or from firing on a node that is still booting. | Setting | Control plane | Cloud worker | Bare-metal worker | | -------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------- | ----------------- | | Remediate only while this many are unhealthy | `≤ 1` machine (an absolute count, not a percentage; it caps concurrent remediation. KCP separately refuses any deletion that would cost etcd quorum) | `[0-2]` in the pool | `≤ 1` | | `nodeStartupTimeout` (grace for a booting node) | 1800s | 600s | 1800s | | Drain before delete (`nodeDrainTimeout`) | 180s | 180s | 180s | | Detach volumes before delete (`nodeVolumeDetachTimeout`) | 120s | 120s | 120s | If more nodes are unhealthy than the gate allows, automatic replacement stops. This is intentional: a fleet-wide outage is almost never caused by individual nodes, and replacing them all would make things worse. These values (the startup grace, the drain timeout, the volume-detach timeout, the concurrency gate, and the rollout `maxSurge`/`maxUnavailable`) are upstream Cluster API settings. The cluster stack ships them as tested defaults on the generated `MachineDeployment` and `KubeadmControlPlane`. They are not exposed as topology variables today, and the topology controller reverts a hand edit to a derived object on its next reconcile, so treat them as fixed for now. Making the drain and reschedule behavior tunable per pool is planned for a future release. ## Condition catalog ### Conditions from custom checks Each check runs on an interval and sets one condition, then holds it. | Condition | How it is triggered | How it can happen | Interval | Disposition | | ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------- | -------- | -------------------------------------------- | | `KubeletUnhealthy` | kubelet `/healthz` fails | kubelet hung, out of file descriptors, bad config, cert problem | 10s | repair, escalates via `ServiceNotRecovering` | | `ContainerRuntimeUnhealthy` | containerd CRI probe fails | containerd deadlocked, disk full, corrupt state | 10s | repair, escalates via `ServiceNotRecovering` | | `ServiceNotRecovering` | a critical unit is `failed`, or restarted ≥3× since the last run | repeated repair did not fix kubelet / containerd / proxy | 1m | alert | | `KubeletHeartbeatFailed` | kubelet logged an error meaning it gave up retrying node-status updates to the API server | kubelet lost its API connection and could not re-establish it | 5m | alert | | `DisksFailure` | an NVMe SMART critical-warning bit is set (read directly over an ioctl), or the kernel logged 5 or more disk I/O errors in the last 10 minutes | a physical disk is wearing out or failing | 5m | alert | | `DiskWearHigh` | an NVMe drive has used most of its rated write endurance | a drive is approaching end of life | 30m | operability | | `DiskTemperatureHigh` | an NVMe drive is past its temperature threshold | a cooling or airflow problem | 5m | operability | | `DiskUsageHigh` | statfs or LVM `vgs` over the space or inode threshold | logs, images, or data filled a filesystem | 2m | operability | | `NodeServiceDown` | an operability unit is not `active` | the tunnel agent, server, or source-IP helper crashed and did not restart | 30s | operability | | `NodeInfoStale` | `nodes.json` is older than 5 minutes | the node-info sync loop is stuck, often a CNI problem | 1m | alert | | `CertRenewalFailing` | a node certificate expires within 7 days and has not renewed | the automatic renewer (kubelet rotation, tunnel self-renew) is broken | 1h | alert | | `ProxyNotServing` | the proxy's `ready` gauge on `http://127.0.0.1:9587/metrics` is 0 or missing, or the endpoint is unreachable | the local `syself-proxy` is down or has no valid target set | 30s | operability (worker only) | | `ClockNotSynchronized` | `adjtimex` reports unsynced, or estimated error > 5s | NTP is broken and the clock is drifting | 1m | alert | | `CNIUnhealthy` | local cilium-agent health is not `Ok`, or its socket is unreachable | the cilium-agent crashed or is degraded | 30s | alert | | `TunnelDisconnected` | the tunnel connection count is 0 | the reverse tunnel to the control plane dropped | 30s | operability | | `GpuActivationFailed` | an NVIDIA GPU is present but has no activation marker or CDI spec | `gpu-activate` did not finish, so containerd cannot use the GPU | 1m | operability (GPU nodes only) | | `APIServerSourceIPUnavailable` | the API server's audit records stop carrying the real client IP (KubeGate's source-IP attribution path is broken) | KubeGate can no longer record the real client source IP; contact support | 1m | alert (control plane only) | ### Conditions from the kernel monitor (kmsg) The kmsg monitor watches `/dev/kmsg` (the kernel's message log) and sets these conditions on matching lines, then holds them. | Condition | Kernel log trigger | How it can happen | Disposition | | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ | | `ReadonlyFilesystem` | `Remounting filesystem read-only` | the kernel hit an I/O or filesystem error and remounted read-only to protect data; writes now fail | alert | | `KernelDeadlock` | a critical process stayed blocked for longer than the kernel's hung-task threshold. This covers kubelet, containerd, the container runtime (runc or crun), and a Kata pod's QEMU VM (qemu or virtiofsd) | a critical process is wedged on stuck I/O or a kernel deadlock | alert | | `VerityCorruption` | `device-mapper: verity: … corrupt` | a block on the sealed read-only OS failed its dm-verity check (disk corruption or tampering) | security (report-only on bare metal; on cloud pools it also triggers reboot, then replace) | | `CperHardwareErrorFatal` | firmware CPER event with severity: fatal | a fatal hardware fault in CPU, memory, or PCIe reported by firmware. CPER is the standard format firmware uses to report hardware errors. | alert | | `GpuFallenOffBus` | `NVRM: … GPU has fallen off the bus` | a GPU dropped off the PCI bus due to power, heat, or a hardware fault | alert (GPU nodes only) | ### The tamper conditions | Condition | How it is triggered | How it can happen | Disposition | | ------------------ | ---------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------- | | `SealedOSTampered` | a sealed OS layer's integrity hash no longer matches the hash recorded when the node was provisioned | a sealed layer was swapped for a different image, or its recorded hash was altered on the writable partition | security (report-only; stays set) | | `NodeTampered` | a protected file under `/var` was added, removed, or changed since the boot baseline | a rogue pod, a bad config push, an operator edit, or a userspace rootkit | security (report-only; stays set; a revert does not clear it) | `SealedOSTampered` and `VerityCorruption` both cover the sealed OS layers but catch different attacks. `VerityCorruption` fires when the kernel hits a bad block on a read, meaning a sealed layer was changed in place. `SealedOSTampered` fires when the health daemon compares the root hash the kernel is enforcing against the one recorded at provision time, so it catches a layer swapped for a different image that is internally consistent under its own hash. The daemon reads this state for the sealed OS layers. See [Node and OS security](/docs/hetzner/apalla/security/node-and-os-security) for what is baselined, and [Respond to a tampered node](/docs/hetzner/apalla/security/respond-to-a-tampered-node) for the runbook. ### Events only (no condition set) The kmsg monitor emits one-shot Node events (plus a `problem_counter` increment, but no lasting condition) for: `TaskHung`, `Ext4Error`, `Ext4Warning`, `IOError`, `BlockIOError`, `NvmeError`, `AtaError`, `NvidiaXidError`, `MemoryReadError`, `UnregisterNetDevice`, and CPER corrected/recoverable errors. `OOMKilling` and `KernelOops` are not emitted. A pod OOM is a workload signal the kubelet already reports. A kernel oops is rare and informational, so both were only noise as Node events. ## Reading conditions and the health report The daemon writes two things to the Node object. **NodeConditions** on `Node.status.conditions`: the binary alert state. | Field | Meaning | | -------------------- | --------------------------------------------------------------------- | | `type` | the condition name (for example `DisksFailure`, `ReadonlyFilesystem`) | | `status` | `True` (problem present), `False` (healthy), or `Unknown` | | `reason` | a short machine-readable reason (for example `FilesystemIsReadOnly`) | | `message` | a human-readable detail line | | `lastTransitionTime` | when the status last changed | Conditions carry no severity field. The disposition decides how each one is routed. **The report annotations** under `autopilot.syself.com/`: five keys, one per domain, refreshed every 2 minutes. Each value is a self-contained JSON document carrying `updatedAt` and `role` plus that one domain, so you can read the certs without pulling the rest. ```json // autopilot.syself.com/storage { "updatedAt": "", "role": "controlplane | worker", "filesystems": [ ... ], "volumeGroups": [ ... ] } // autopilot.syself.com/disks { "updatedAt": "", "role": "...", "disks": [ ... ] } // autopilot.syself.com/certs { "updatedAt": "", "role": "...", "certs": [ ... ] } // autopilot.syself.com/tamper { "updatedAt": "", "role": "...", "tamper": { ... } } // autopilot.syself.com/services { "updatedAt": "", "role": "...", "services": [ ... ] } ``` The condition tells you that something is wrong. The report tells you the detail. ## How to read them on a node ```console # Show all conditions and their current status (type=status reason): $ kubectl get node \ -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}{"\n"}{end}' # Show one domain of the health report. There is no single key that returns all of it: $ kubectl get node \ -o jsonpath='{.metadata.annotations.autopilot\.syself\.com/disks}' | jq . # The other four keys work the same way: # autopilot.syself.com/storage, /certs, /tamper, /services ``` A stale condition means the daemon is down. In that case, alert on `Ready` for "node gone" and on the report's `updatedAt` timestamp for "daemon gone". ## How to alert on a condition The daemon sets conditions; it does not send anyone a notification. Alerting is yours to wire up. The report-only conditions (dispositions `alert`, `operability`, and `security`) do nothing on their own out of the box. The path is [`kube-state-metrics`](/docs/hetzner/apalla/observability/metrics/kube-state-metrics). It turns every NodeCondition into a Prometheus metric, `kube_node_status_condition{condition="...",status="true"}`, and you write a Prometheus or Alertmanager rule on it. A rule that fires when `kube_node_status_condition{condition="DisksFailure",status="true"} == 1` gives you the page the `alert` disposition is meant to produce. See [set up Prometheus](/docs/hetzner/apalla/observability/metrics/set-up-prometheus) for installing the metrics pipeline, and [Metrics reference](/docs/hetzner/apalla/reference/metrics) for the daemon's own metrics on port `20257`. ## Related - [Self-healing and node replacement](/docs/hetzner/apalla/concepts/operations/self-healing-and-node-replacement): how Syself's replacement logic fits into the wider recovery model. - [Node and OS security](/docs/hetzner/apalla/security/node-and-os-security): how tamper and verity conditions relate to node integrity. - [Node labels and annotations](/docs/hetzner/apalla/reference/node-labels-and-annotations): the node's labels and the `autopilot.syself.com/*` health annotations. - [Verify node integrity](/docs/hetzner/apalla/security/verify-node-integrity): how to investigate a `NodeTampered` or `VerityCorruption` condition. - [Ports and listeners](/docs/hetzner/apalla/reference/ports-and-listeners): port `20257` where the health daemon exposes its Prometheus metrics.