Machine health checks and remediation

Syself Autopilot watches every node and replaces the ones that stop being healthy. Most of the time you do nothing.

What the health check watches#

Syself Autopilot watches each node's Ready state, the kubelet's summary of whether the node can still run pods. When a node reports itself broken, or stops reporting at all, for longer than a short grace period, it counts as unhealthy and remediation begins.

On cloud pools, Syself Autopilot also acts on a few of its own health signals that mean the node is unlikely to recover on its own. Bare-metal pools are more conservative: a Ready failure triggers remediation, but the deeper health signals alert a person instead of reprovisioning the server, because a reinstall does not fix a hardware or storage fault, and the same fault can strike several servers at once.

See Node problem detection and conditions for what a node reports and how to alert on it.

What remediation does#

Remediation tries a reboot first. If the node comes back, it stays. If not, the machine is drained (its pods move to other nodes) and then replaced.

flowchart LR
  A[Node unhealthy past its window] --> B[Reboot]
  B --> C{Node recovers?}
  C -->|Yes| D[Stays in the cluster]
  C -->|No| E[Drain, then replace]

The last step differs by infrastructure. On cloud, Syself Autopilot reboots the node and, if it does not come back, replaces it with a fresh machine. On bare metal it is more patient, retrying the reboot before falling back to reprovisioning the host, since a dedicated server is worth more effort to recover than a disposable VM.

Self-healing does not repair hardware#

Remediation swaps a node out. It does not fix the physical machine underneath. On bare metal this matters most when a pool's host selector is narrow. If every host the selector matches is already in use, a machine that fails is released and then re-claims the same host, because no other host qualifies. The server is wiped and reinstalled, and a fault in the hardware itself comes straight back.

Re-provisioning clears a software or filesystem problem. It does nothing for a failing disk, bad memory, or a dying NIC. So do not lean on self-healing to route around broken hardware. Detect a hardware fault directly instead:

• Alert on the hardware-related node conditions . On bare metal these page a human rather than trigger a reboot, precisely because a reboot would not help.
• Or run your own observability against the servers (SMART data, memory, temperature, link state) and alert from there.

When a server is genuinely broken, take it out of the pool: set maintenanceMode: true on its HetznerBareMetalHost (or remove it), and open a Hetzner support case for the hardware. Alerting on the node conditions is the reliable way to catch a hardware fault.

The platform stops before it makes things worse#

Remediation is bounded on purpose. Syself Autopilot only replaces nodes while the number of unhealthy machines in a pool stays small, and it never removes a control-plane node in a way that would cost etcd its quorum, the majority of control-plane members the cluster needs to stay writable. A node that is still booting is given time to settle before it counts against that limit at all.

Drains and PodDisruptionBudgets#

Before a node is removed, its pods are drained, and the drain respects PodDisruptionBudgets. A PodDisruptionBudget is a rule that caps how many replicas of a workload can be down at once. The drain waits for pods to leave within their budget, then proceeds anyway, so a stuck pod cannot block recovery forever.

Set PodDisruptionBudgets on workloads that must not lose too many replicas at once. See Self-healing and node replacement .

Pause remediation during maintenance#

When you plan to disrupt a node yourself (a firmware update, an in-place reboot, hardware work), pause remediation so Syself Autopilot does not mistake your maintenance for a failure. Pausing a Machine stops the management cluster from reconciling it at all, not only from remediating it.

		$ kubectl annotate machine <machine-name> -n <namespace> cluster.x-k8s.io/paused=true
	

		$ kubectl get machine <machine-name> -n <namespace> \
  -o jsonpath='{.metadata.annotations.cluster\.x-k8s\.io/paused}'
	

		$ kubectl annotate machine <machine-name> -n <namespace> cluster.x-k8s.io/paused-
	

When remediation should not fire#

Pause first for anything that makes a healthy node look unhealthy on purpose: rebooting it, taking it off the network, or running long disruptive maintenance. Leave remediation on for everything else. A node that genuinely fails should be replaced, and that is the whole point of the system.