Syself Autopilot watches every node and replaces the ones that stop being healthy. Most of the time you do nothing.

## What the health check watches

Syself Autopilot watches each node's `Ready` state, the kubelet's summary of whether the node can still run pods. When a node reports itself broken, or stops reporting at all, for longer than a short grace period, it counts as unhealthy and remediation begins.

On cloud pools, Syself Autopilot also acts on a few of its own health signals that mean the node is unlikely to recover on its own. Bare-metal pools are more conservative: a `Ready` failure triggers remediation, but the deeper health signals alert a person instead of reprovisioning the server, because a reinstall does not fix a hardware or storage fault, and the same fault can strike several servers at once.

See [Node problem detection and conditions](/docs/hetzner/apalla/servers-and-nodes/access/node-problem-detection-and-conditions) for what a node reports and how to alert on it.

<Collapsible title="Before the window even starts: in-place repair">

Before any health-check window starts counting, a health daemon on the node tries to fix the smaller problems itself. If the kubelet or the container runtime stumbles, it restarts them in place and gives them a moment to settle. Only when those restarts stop helping does it raise a condition that, on cloud pools, can lead to a reboot and a replacement.

</Collapsible>

> [!TIP]
> A control-plane load balancer outage does not usually flip a node to `Ready=False`. Workers reach the API server through a local failover proxy on `127.0.0.1:7443`. The proxy already knows each control-plane IP address, so if the load balancer goes down, it switches to one directly. Kubelet keeps sending its heartbeat (a periodic write called a **Node Lease**), and the node stays `Ready`.

## What remediation does

Remediation tries a reboot first. If the node comes back, it stays. If not, the machine is drained (its pods move to other nodes) and then replaced.

```mermaid
flowchart LR
  A[Node unhealthy past its window] --> B[Reboot]
  B --> C{Node recovers?}
  C -->|Yes| D[Stays in the cluster]
  C -->|No| E[Drain, then replace]
```

The last step differs by infrastructure. On cloud, Syself Autopilot reboots the node and, if it does not come back, replaces it with a fresh machine. On bare metal it is more patient, retrying the reboot before falling back to reprovisioning the host, since a dedicated server is worth more effort to recover than a disposable VM.

> [!TIP]
> The `Ready=False` window is long enough that a normal reboot or a short network blip resolves before Syself Autopilot notices. If your planned work (a firmware update, a manual reboot, hardware swap) takes longer than that, [pause the Machine first](#pause-remediation-during-maintenance), or Syself Autopilot starts remediating a node you are already fixing. See [Node health conditions](/docs/hetzner/apalla/reference/node-health-conditions) for the exact window.

## Self-healing does not repair hardware

Remediation swaps a node out. It does not fix the physical machine underneath. On bare metal this matters most when a pool's host selector is narrow. If every host the selector matches is already in use, a machine that fails is released and then re-claims the **same** host, because no other host qualifies. The server is wiped and reinstalled, and a fault in the hardware itself comes straight back.

Re-provisioning clears a software or filesystem problem. It does nothing for a failing disk, bad memory, or a dying NIC. So do not lean on self-healing to route around broken hardware. Detect a hardware fault directly instead:

- Alert on the hardware-related [node conditions](/docs/hetzner/apalla/servers-and-nodes/access/node-problem-detection-and-conditions). On bare metal these page a human rather than trigger a reboot, precisely because a reboot would not help.
- Or run your own observability against the servers (SMART data, memory, temperature, link state) and alert from there.

When a server is genuinely broken, take it out of the pool: set `maintenanceMode: true` on its `HetznerBareMetalHost` (or remove it), and open a Hetzner support case for the hardware. Alerting on the node conditions is the reliable way to catch a hardware fault.

## The platform stops before it makes things worse

Remediation is bounded on purpose. Syself Autopilot only replaces nodes while the number of unhealthy machines in a pool stays small, and it never removes a control-plane node in a way that would cost etcd its quorum, the majority of control-plane members the cluster needs to stay writable. A node that is still booting is given time to settle before it counts against that limit at all.

> [!IMPORTANT]
> If more nodes go unhealthy than the limit allows, automatic replacement stops and waits for you. A fleet-wide problem should not trigger fleet-wide reprovisioning. That is your cue to take over: check the [node conditions](/docs/hetzner/apalla/servers-and-nodes/access/node-problem-detection-and-conditions) to see what is happening, [pause remediation](#pause-remediation-during-maintenance) on any node you are working on, and fix the root cause before you let replacement resume.

## Drains and PodDisruptionBudgets

Before a node is removed, its pods are drained, and the drain respects PodDisruptionBudgets. A PodDisruptionBudget is a rule that caps how many replicas of a workload can be down at once. The drain waits for pods to leave within their budget, then proceeds anyway, so a stuck pod cannot block recovery forever.

Set PodDisruptionBudgets on workloads that must not lose too many replicas at once. See [Self-healing and node replacement](/docs/hetzner/apalla/concepts/operations/self-healing-and-node-replacement).

## Pause remediation during maintenance

When you plan to disrupt a node yourself (a firmware update, an in-place reboot, hardware work), pause remediation so Syself Autopilot does not mistake your maintenance for a failure. Pausing a `Machine` stops the management cluster from reconciling it at all, not only from remediating it.

<Steps>
<Step>Annotate the Machine</Step>

In the management cluster:

```console
$ kubectl annotate machine <machine-name> -n <namespace> cluster.x-k8s.io/paused=true
```

<Step>Verify the annotation is set</Step>

Before you start work, check the value:

```console
$ kubectl get machine <machine-name> -n <namespace> \
  -o jsonpath='{.metadata.annotations.cluster\.x-k8s\.io/paused}'
```

The output should be `true`.

<Step>Do your maintenance</Step>

The Machine is not touched while it is paused, so do the firmware update, reboot, or hardware work now.

<Step>Remove the pause when you are done</Step>

```console
$ kubectl annotate machine <machine-name> -n <namespace> cluster.x-k8s.io/paused-
```

</Steps>

> [!WARNING]
> Double-check the annotation for typos. If it is not set correctly, the node can be flagged unhealthy and reprovisioned while you work on it.

## When remediation should not fire

Pause first for anything that makes a healthy node look unhealthy on purpose: rebooting it, taking it off the network, or running long disruptive maintenance. Leave remediation on for everything else. A node that genuinely fails should be replaced, and that is the whole point of the system.