Syself Autopilot watches every node and replaces the ones that stop being healthy. Most of the time you do nothing. ## What the health check watches Syself Autopilot watches each node's `Ready` state, the kubelet's summary of whether the node can still run pods. When a node reports itself broken, or stops reporting at all, for longer than a short grace period, it counts as unhealthy and remediation begins. On cloud pools, Syself Autopilot also acts on a few of its own health signals that mean the node is unlikely to recover on its own. Bare-metal pools are more conservative: a `Ready` failure triggers remediation, but the deeper health signals alert a person instead of reprovisioning the server, because a reinstall does not fix a hardware or storage fault, and the same fault can strike several servers at once. See [Node problem detection and conditions](/docs/hetzner/apalla/servers-and-nodes/access/node-problem-detection-and-conditions) for what a node reports and how to alert on it. ## What remediation does Remediation tries a reboot first. If the node comes back, it stays. If not, the machine is drained (its pods move to other nodes) and then replaced. ```mermaid flowchart LR A["Node unhealthy past its window"]:::platform --> B["Reboot"]:::platform B --> C{"Node recovers?"}:::decision C -->|Yes| D["Stays in the cluster"]:::platform C -->|No| E["Drain, then replace"]:::platform ``` The last step differs by infrastructure. On cloud, Syself Autopilot reboots the node and, if it does not come back, replaces it with a fresh machine. On bare metal it is more patient, retrying the reboot before falling back to reprovisioning the host, since a dedicated server is worth more effort to recover than a disposable VM. > [!TIP] > The `Ready=False` window is long enough that a normal reboot or a short network blip resolves before Syself Autopilot notices. If your planned work (a firmware update, a manual reboot, hardware swap) takes longer than that, [pause the Machine first](#pause-remediation-during-maintenance), or Syself Autopilot starts remediating a node you are already fixing. See [Node health conditions](/docs/hetzner/apalla/reference/node-health-conditions) for the exact window. ## Self-healing does not repair hardware Remediation swaps a node out. It does not fix the physical machine underneath. On bare metal this matters most when a pool's host selector is narrow. If every host the selector matches is already in use, a machine that fails is released and then re-claims the **same** host, because no other host qualifies. The server is wiped and reinstalled, and a fault in the hardware itself comes straight back. Re-provisioning clears a software or filesystem problem. It does nothing for a failing disk, bad memory, or a dying NIC. So do not lean on self-healing to route around broken hardware. Detect a hardware fault directly instead: - Alert on the hardware-related [node conditions](/docs/hetzner/apalla/servers-and-nodes/access/node-problem-detection-and-conditions). On bare metal these page a human rather than trigger a reboot, precisely because a reboot would not help. - Or run your own observability against the servers (SMART data, memory, temperature, link state) and alert from there. When a server is genuinely broken, take it out of the pool: set `maintenanceMode: true` on its `HetznerBareMetalHost` (or remove it), and open a Hetzner support case for the hardware. Alerting on the node conditions is the reliable way to catch a hardware fault. ## The platform stops before it makes things worse Remediation is bounded on purpose. Syself Autopilot only replaces nodes while the number of unhealthy machines in a pool stays small, and it never removes a control-plane node in a way that would cost etcd its quorum, the majority of control-plane members the cluster needs to stay writable. A node that is still booting is given time to settle before it counts against that limit at all. > [!IMPORTANT] > If more nodes go unhealthy than the limit allows, automatic replacement stops and waits for you. A fleet-wide problem should not trigger fleet-wide reprovisioning. That is your cue to take over: check the [node conditions](/docs/hetzner/apalla/servers-and-nodes/access/node-problem-detection-and-conditions) to see what is happening, [pause remediation](#pause-remediation-during-maintenance) on any node you are working on, and fix the root cause before you let replacement resume. Before a node is removed its pods are drained, and the drain respects PodDisruptionBudgets, so set one on any workload that must not lose too many replicas at once. See [Self-healing and node replacement](/docs/hetzner/apalla/concepts/operations/self-healing-and-node-replacement). ## Pause remediation during maintenance When you plan to disrupt a node yourself (a firmware update, an in-place reboot, hardware work), pause remediation so Syself Autopilot does not mistake your maintenance for a failure. Pausing a `Machine` stops the management cluster from reconciling it at all, not only from remediating it. Annotate the Machine In the management cluster: ```console $ kubectl annotate machine -n cluster.x-k8s.io/paused=true ``` Verify the annotation is set Before you start work, check the value: ```console $ kubectl get machine -n \ -o jsonpath='{.metadata.annotations.cluster\.x-k8s\.io/paused}' ``` The output should be `true`. Do your maintenance The Machine is not touched while it is paused, so do the firmware update, reboot, or hardware work now. Remove the pause when you are done ```console $ kubectl annotate machine -n cluster.x-k8s.io/paused- ``` > [!WARNING] > Double-check the annotation for typos. If it is not set correctly, the node can be flagged unhealthy and reprovisioned while you work on it. ## When remediation should not fire Pause first for anything that makes a healthy node look unhealthy on purpose: rebooting it, taking it off the network, or running long disruptive maintenance. Leave remediation on for everything else. A node that genuinely fails should be replaced, and that is the whole point of the system. ## Related - [Node problem detection and conditions](/docs/hetzner/apalla/servers-and-nodes/access/node-problem-detection-and-conditions) - [Node health conditions](/docs/hetzner/apalla/reference/node-health-conditions) - [Self-healing and node replacement](/docs/hetzner/apalla/concepts/operations/self-healing-and-node-replacement) - [Remove a specific node](/docs/hetzner/apalla/servers-and-nodes/maintenance/remove-a-specific-node)