Self-Healing and Node Replacement

On Syself Autopilot a failed node repairs itself or is replaced on its own. Say kubelet on a worker starts crash-looping. Syself Autopilot restarts it, and if the restart does not help, replaces the machine.

Plain Kubernetes does little about a failed node. If kubelet stops sending heartbeats for about 40 seconds, Kubernetes marks the node NotReady, and a few minutes later it moves the pods elsewhere. The node itself stays broken. Your monitoring notices and an alert fires, but nothing happens until someone logs in to read the logs, restart services, or delete the machine by hand.

Syself Autopilot adds three layers that run at the same time. A health daemon on each node fixes what a restart can fix. A failover proxy keeps workers talking to a control plane when the load balancer path fails. A controller replaces any node that cannot recover on its own. Because they run together, one broken node can have all three working on it, each on a different failure. A person is called only when replacing the machine would not fix the problem.

One safety gate sits over all three layers. The safety gate section below explains it.

flowchart TD
  subgraph node["Running concurrently on the node"]
    L1["Layer 1: syself-agent<br/>(on the node)<br/>checks, in-place restarts,<br/>sets NodeConditions"]
    L2["Layer 2: syself-proxy failover proxy<br/>(on each worker)<br/>keeps kubelet talking to a<br/>control plane if the load balancer fails"]
  end
  L2 -->|"load balancer fails:<br/>node stays Ready"| ok["No escalation needed"]
  L1 -->|"condition persists past<br/>the pool timeout"| L3["Layer 3: replacement controller<br/>(management cluster)"]
  L3 --> gate{"Safety gate:<br/>too many machines in the<br/>pool unhealthy at once?"}
  gate -->|"yes"| hold["Hold replacement,<br/>alert a human"]
  gate -->|"no"| seq["Reboot first, then<br/>cordon, drain (respects<br/>PodDisruptionBudgets), replace"]
  seq --> fresh["Fresh immutable node joins"]

Layers 1 and 2 run on the node, inside its immutable image. Layer 3 runs in the Syself-operated management cluster, which watches conditions across your clusters and acts on the ones that do not clear.

Immutable means the node's operating system cannot change after it is built. There is nothing to patch on a running node, nothing drifts over time, and you can check that a node runs exactly what Syself shipped (see the node lifecycle ).

Layer 1: in-place repair on the node#

Every node runs syself-agent inside the immutable image. It watches the kernel logs, kubelet, the container runtime, the disk, and the clock. Each check runs on its own, so one failing check never stalls the rest. Every couple of minutes, syself-agent writes a short health report onto the Node.

When a check trips, the daemon does one of two things.

If a restart can fix the problem, the daemon restarts the service. A kubelet that fails its health check gets restarted, and the daemon sets KubeletUnhealthy to mark that it is working on kubelet. That flag does not tell layer 3 to delete the node.

If a restart cannot fix the problem, the daemon raises a condition for layer 3 to act on. The condition is a flag on the Node object that names a specific fault, such as ReadonlyFilesystem or ServiceNotRecovering. The daemon reports the fault, and the controller in layer 3 decides what to do.

The full catalog, condition by condition, is in node health conditions .

Layer 2: keep workers connected when the load balancer fails#

The control-plane load balancer is a single point of failure for every worker. Each worker reaches the API server through it. kubelet renews its Node Lease every few seconds. Lose the API server for about 40 seconds and Kubernetes marks the node NotReady and moves its pods. So one load balancer failure marks every worker NotReady at the same moment, even though every node is fine.

A failover proxy named syself-proxy runs on each worker, and kubelet talks to it instead of the load balancer. It keeps its own list of the real control-plane addresses. When the load balancer stops passing traffic, it connects to a live control plane directly. The leases keep renewing, and the node stays Ready.

The proxy runs locally on each worker and does not add load to the rest of the control-plane traffic. The connection through it is still fully verified and encrypted, the same as if kubelet reached the load balancer directly.

The proxy is only for workers. On a control-plane node, kubelet talks straight to the API server on the same node, so no load balancer is involved there and there is nothing to fail over.

Layer 3: replacement, and why bare metal is different#

Layer 3 watches conditions from the management cluster and replaces a machine when a condition it cares about outlasts its timeout. Which conditions it cares about depends on the pool, Hetzner Cloud or bare metal (Hetzner dedicated), because re-provisioning does not fix the same problems on both.

On cloud, replacement fires on a range of conditions:

• Ready=Unknown: the node has gone unreachable.
• Ready=False: the node reports itself unhealthy.
• KernelDeadlock: a kernel deadlock.
• ReadonlyFilesystem: a filesystem that flipped read-only.
• ServiceNotRecovering: a core service that keeps crashing.
• VerityCorruption: an integrity check found an altered block.

On bare metal, only Ready=False triggers replacement. Everything else is set for visibility and alerting only, and triggers no replacement.

The narrower bare-metal list is deliberate. Re-provisioning a cloud server gives you a genuinely new machine, so churning one is cheap and usually works. Re-provisioning a bare-metal server rewrites the OS disk on the same physical host. A node that went Ready=Unknown because the network partitioned does not come back from a reinstall. A KernelDeadlock or a filesystem that flipped read-only is often failing hardware, and that hardware is still there after the disk is rewritten. So on bare metal those conditions raise an alert and wait for a person, because the fix needs someone to look at the physical hardware.

VerityCorruption is the clearest case. On cloud, a block that fails the integrity check means the image is wrong, so the node is replaced. On bare metal the same corruption is at least as likely to be a dying disk as tampering, and re-provisioning would churn the pool without touching the bad drive. So a corrupt-but-Ready bare-metal node alerts a human and keeps running.

Either way, Syself tries a reboot before it deletes anything, and only deletes when the reboot fails. The exact conditions, their timeouts, and the per-pool behavior are in node health conditions .

The safety gate#

When too many machines in a pool are unhealthy at the same time, one safety gate stops replacement and alerts a person instead. A bad rollout or a network incident can make an entire pool report failed at once, when the machines are fine. Replacing them all in parallel turns a temporary failure into a real outage.

How the replacement runs#

When a node does get replaced, it is never removed abruptly. First it is cordoned, so no new pods land on it. Then it is drained, so the pods on it move off. The drain honors PodDisruptionBudgets: it waits a few minutes rather than break them, then proceeds anyway so one stuck pod cannot block healing forever. Then the node is deleted, and a fresh one joins.

This is the same cordon-drain-replace sequence the node lifecycle runs for every change, and it is cheap for the same reasons. Every node is the byte-for-byte same hash-pinned image. The desired state of each node is stored in the Cluster object, so a replacement needs no input from you. Upgrades and scale-downs use this sequence too.

Design your workloads for replacement#

Assume any node can be replaced at any time.

• Run more than one replica. A single-replica Deployment is down for the length of every drain that touches its node.
• Set a PodDisruptionBudget on anything that matters. It names a floor, say two pods, that must stay available for the application. On any drain, whether from self-healing, an upgrade, or a scale-down, Syself keeps that floor running before it evicts the next pod.
• Handle the shutdown signal (SIGTERM). Finish in-flight work and exit inside the grace period. The drain does not wait forever, so a pod that ignores it is killed anyway.
• Keep data off the OS disk. It is rewritten on every re-provision and replacement. What survives a node loss, and how cloud and local volumes differ, is in backup and disaster recovery .
• Never fix a node over SSH. Any change on a live node disappears with the next replacement. Permanent changes belong in your manifests or your cluster's declared configuration.

The same failure: generic cluster vs Syself Autopilot#

Follow that same crash-looping kubelet through both setups.

How you treat an alert depends on the pool. On cloud pools, a Ready=False alert is informational, because the platform is already replacing the node. On bare metal, the alert-only conditions are your queue. There the platform has decided, on purpose, that a person should look at the hardware before anything is rewritten.

Syself tests these recovery paths by failing nodes and breaking the network on purpose.