Node health conditions

Every node in a Syself Autopilot cluster runs a health daemon. The daemon continuously checks the node and reports problems as Kubernetes NodeConditions (flags on a Node object that describe its health state), events, and metrics. It also writes a JSON health report to the node, as a set of autopilot.syself.com/* annotations, one key per domain.

What a condition does when it fires depends on the pool. A few conditions trigger automatic machine replacement; the rest are there for visibility and alerting. Automatic replacement lists the exact set per pool.

The two layers: report, then react#

The daemon and the replacement logic are separate. Keep them distinct.

The daemon reports. It detects a problem and writes it as a NodeCondition, an event, and a metric. That is all it does for most problems. It does not cordon, drain, or replace nodes. It does not set the Ready condition; the kubelet owns that.

Something else reacts. A few different actors can respond:

• In-place repair restarts a broken unit (kubelet, containerd) on the node.
• Syself's replacement logic reboots the node and, if that fails, replaces it. It is the only automatic actor that deletes a node, and it watches Ready on bare metal, plus a few daemon conditions on cloud pools.
• You can force a replacement by hand at any time: delete the Machine object, and the controllers drain the node and bring up a fresh one in its place.
• A human reads an alert and decides what to do.

The mapping from condition to actor is fixed. It is called the disposition and is defined in the next section.

Events vs conditions#

Every monitor rule is one of two types.

Temp (temporary) rules emit a one-shot Node event and increment the problem_counter metric. They do not set a condition. Use these for things that happen once and pass (a single disk read error, a corrected memory error).

Perm (permanent) rules set a NodeCondition on the Node and hold it. status: True means the problem is present. status: False means healthy. If the daemon stops, its conditions keep their last value and go stale; they do not clear automatically.

An event says "this happened". A condition says "this is the state right now".

Dispositions: what happens when a condition fires#

Every condition has exactly one disposition. The disposition decides what happens when the condition turns True.

In-place repair#

Before anything heavier runs, a broken critical unit is restarted on the node.

kubelet: the health daemon probes /healthz every 10 seconds. On a failed probe it restarts kubelet.service and waits a 1-minute cooldown before trying again.

containerd: the daemon probes the CRI socket (CRI, the Container Runtime Interface, connects kubelet to the container runtime) every 10 seconds. On failure it restarts containerd.service with a 2-minute cooldown.

systemd also restarts these units on its own (Restart=always).

Every restart increments systemd's NRestarts counter for that unit.

When repair is not working. The service-not-recovering check runs every minute. For the critical units (kubelet, containerd, and on a worker the failover proxy), it sets ServiceNotRecovering = True when a unit is either in the systemd failed state, or flapping (its NRestarts went up by 3 or more since the last check). ServiceNotRecovering has the alert disposition, so it surfaces for you to alert on (see How to alert on a condition).

The daemon also exports syself_node_unit_restarts{unit} as a Prometheus metric, so you can see repair activity as a time series.

Automatic replacement#

Syself's replacement logic is the only thing that deletes and re-creates a machine. On bare-metal pools it watches only the Ready condition. On cloud pools it also watches four of the daemon's own conditions: ReadonlyFilesystem, KernelDeadlock, ServiceNotRecovering, and VerityCorruption. Rebooting the node fixes all of these on a fresh VM.

Cloud pools (HCloud control plane and HCloud workers)#

Syself reboots the server once, allowing up to 180 seconds for the reboot to complete. If the condition is still true after it has held for the time shown above, Syself replaces the machine with a new VM.

Bare-metal pools (bare-metal control plane and bare-metal workers)#

On bare metal, Ready=Unknown alerts a human instead of triggering remediation. It is not wired into automatic replacement.

Syself reboots the physical server up to two times. If the node is still unhealthy after 300 seconds, Syself re-provisions the same physical host (wipe and reinstall). The longer timeout gives hardware more time to stabilise after a reboot before Syself escalates to re-provisioning.

Safety gates and startup grace#

These prevent automatic replacement from making an incident worse or from firing on a node that is still booting.

If more nodes are unhealthy than the gate allows, automatic replacement stops. This is intentional: a fleet-wide outage is almost never caused by individual nodes, and replacing them all would make things worse.

These values (the startup grace, the drain timeout, the volume-detach timeout, the concurrency gate, and the rollout maxSurge/maxUnavailable) are upstream Cluster API settings. The cluster stack ships them as tested defaults on the generated MachineDeployment and KubeadmControlPlane. They are not exposed as topology variables today, and the topology controller reverts a hand edit to a derived object on its next reconcile, so treat them as fixed for now. Making the drain and reschedule behavior tunable per pool is planned for a future release.

Condition catalog#

Conditions from custom checks#

Each check runs on an interval and sets one condition, then holds it.

Conditions from the kernel monitor (kmsg)#

The kmsg monitor watches /dev/kmsg (the kernel's message log) and sets these conditions on matching lines, then holds them.

The tamper conditions#

SealedOSTampered and VerityCorruption both cover the sealed OS layers but catch different attacks. VerityCorruption fires when the kernel hits a bad block on a read, meaning a sealed layer was changed in place. SealedOSTampered fires when the health daemon compares the root hash the kernel is enforcing against the one recorded at provision time, so it catches a layer swapped for a different image that is internally consistent under its own hash. The daemon reads this state for the sealed OS layers.

See Node and OS security for what is baselined, and Respond to a tampered node for the runbook.

Events only (no condition set)#

The kmsg monitor emits one-shot Node events (plus a problem_counter increment, but no lasting condition) for: TaskHung, Ext4Error, Ext4Warning, IOError, BlockIOError, NvmeError, AtaError, NvidiaXidError, MemoryReadError, UnregisterNetDevice, and CPER corrected/recoverable errors.

OOMKilling and KernelOops are not emitted. A pod OOM is a workload signal the kubelet already reports. A kernel oops is rare and informational, so both were only noise as Node events.

Reading conditions and the health report#

The daemon writes two things to the Node object.

NodeConditions on Node.status.conditions: the binary alert state.

Conditions carry no severity field. The disposition decides how each one is routed.

The report annotations under autopilot.syself.com/: five keys, one per domain, refreshed every 2 minutes. Each value is a self-contained JSON document carrying updatedAt and role plus that one domain, so you can read the certs without pulling the rest.

		// autopilot.syself.com/storage
{ "updatedAt": "<timestamp>", "role": "controlplane | worker",
  "filesystems": [ ... ], "volumeGroups": [ ... ] }
 
// autopilot.syself.com/disks
{ "updatedAt": "<timestamp>", "role": "...", "disks": [ ... ] }
 
// autopilot.syself.com/certs
{ "updatedAt": "<timestamp>", "role": "...", "certs": [ ... ] }
 
// autopilot.syself.com/tamper
{ "updatedAt": "<timestamp>", "role": "...", "tamper": { ... } }
 
// autopilot.syself.com/services
{ "updatedAt": "<timestamp>", "role": "...", "services": [ ... ] }
	

The condition tells you that something is wrong. The report tells you the detail.

How to read them on a node#

		# Show all conditions and their current status (type=status reason):
$ kubectl get node <name> \
  -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}{"\n"}{end}'
 
# Show one domain of the health report. There is no single key that returns all of it:
$ kubectl get node <name> \
  -o jsonpath='{.metadata.annotations.autopilot\.syself\.com/disks}' | jq .
 
# The other four keys work the same way:
#   autopilot.syself.com/storage, /certs, /tamper, /services
	

A stale condition means the daemon is down. In that case, alert on Ready for "node gone" and on the report's updatedAt timestamp for "daemon gone".

How to alert on a condition#

The daemon sets conditions; it does not send anyone a notification. Alerting is yours to wire up. The report-only conditions (dispositions alert, operability, and security) do nothing on their own out of the box.

The path is kube-state-metrics . It turns every NodeCondition into a Prometheus metric, kube_node_status_condition{condition="...",status="true"}, and you write a Prometheus or Alertmanager rule on it. A rule that fires when kube_node_status_condition{condition="DisksFailure",status="true"} == 1 gives you the page the alert disposition is meant to produce.

See set up Prometheus for installing the metrics pipeline, and Metrics reference for the daemon's own metrics on port 20257.

Related#

• Self-healing and node replacement : how Syself's replacement logic fits into the wider recovery model.
• Node and OS security : how tamper and verity conditions relate to node integrity.
• Node labels and annotations : the node's labels and the autopilot.syself.com/* health annotations.
• Verify node integrity : how to investigate a NodeTampered or VerityCorruption condition.
• Ports and listeners : port 20257 where the health daemon exposes its Prometheus metrics.