Kubernetes Node Health Monitoring and Self-Healing

Syself Health is our approach to Kubernetes node health monitoring: an agent that runs on every node and watches it in real time. When it finds a problem, it reports it as a structured condition on the Node object, a health flag you can read and alert on, and it triggers the loop that reboots or replaces the node. New in 1.36, it also tracks bare-metal disk health, so a failing drive shows up as node data before it becomes an outage.

Nodes here are disposable, so something has to decide when one should be replaced. Syself Health is that something.

What does Syself Health check?

Generic Kubernetes setups lean on an external monitoring stack and slow default checks. Syself Health runs a daemon right on each node and does hundreds of lightweight checks in real time:

• hardware degradation across CPU, disk, and memory
• kubelet and container-runtime health
• network latency, connectivity, and packet loss

The platform also checks each node's health from outside the node. That's how it catches a silent network partition or an unreachable node that an on-node check alone would miss. A node that's cut off doesn't just sit there looking fine.

What happens when it finds a problem?

Detection drives self-healing. On cloud servers, a troubled node is rebooted first and, if it doesn't recover, replaced from a fresh image. On bare-metal servers the node is rebooted but never wiped automatically, since a bad signal there is as likely a failing disk as anything else. So most bare-metal conditions are alert-only: you see them and decide.

The agent runs as a background service instead of a pod, so it doesn't count against your per-node pod limit. It also saves a short diagnostic report in the node's data for later. And 1.36 adds a log collector that gathers what's needed to diagnose a problem in one step, so opening a support case is quick instead of a hunt for the right logs.

You read the conditions with kubectl and wire them into your own alerting. The full loop, from a failed check to a replaced node, is in the self-healing docs.

Who keeps the record after a node is replaced?

Syself Health keeps the cluster healthy. It doesn't keep the record for you. The moment self-healing replaces a node, that node's local logs go with it, so any evidence you'd investigate with later has to be shipped off-node before then.

Detection and retention are separate jobs. We handle detection, in real time, on and around every node. Retention is yours, and it needs a log store that outlives any single node. Set it up early, because a healthy fleet with no history is how you end up unable to explain what happened last Tuesday. The split is spelled out in detection vs retention, and shipping the audit trail off-node is covered in audit logs with the real client IP.

FAQ

What does Syself Health actually detect?

Hardware degradation, kubelet and runtime health, and network problems: hundreds of checks per node, plus health checks from outside the node to catch silent partitions and unreachable nodes.

Does a failing node get replaced automatically?

On cloud servers, yes. It's rebooted first, then replaced from a fresh image if it doesn't recover. On bare metal it's rebooted but never wiped automatically, so you stay in the loop on hardware faults.

How do I see a node's health?

Problems appear as conditions on the Node object, which you read with kubectl and connect to your alerting. While a node recovers, the control-plane failover proxy keeps workers talking to a healthy control plane.

Detection is what makes disposable nodes practical. You can't replace a bad node until something notices it's bad, and Syself Health is what notices.