Node problem detection and conditions
Every node reports its own health as conditions, the status flags Kubernetes keeps on a Node. On top of the standard Kubernetes conditions, Syself Autopilot adds its own, so trouble shows up as a clear signal before a node fails outright. You get early warning, automatic recovery where it is safe, and one clean place to hook your own alerting.
What conditions give you#
- Early warning. A node tells you it is degrading, a disk wearing out, a certificate not renewing, a service struggling, before it drops out of the cluster.
- Automatic recovery. For the problems a restart or a replacement can fix, Syself Autopilot acts on its own. You do not wire anything up for that.
- One place to alert from. The conditions that actually need a person are the ones you route to your monitoring, and there are only a handful.
What the conditions cover#
Beyond the standard Ready, MemoryPressure, DiskPressure, and PIDPressure that every Kubernetes node reports, Syself Autopilot adds health signals for:
- the sealed OS and kernel
- the kubelet and container runtime
- core node services and networking
- disks: wear, temperature, and failures
- certificates
- GPUs
- hardware faults and node integrity
Ready is still the signal that drives replacement. The rest give you and the platform a head start. The full catalog, with every signal and what it means, is in the Node health conditions reference .
What happens when one fires#
Each signal has a fixed outcome, so the behavior is predictable:
- Fixed in place. For a service that has stumbled, Syself Autopilot restarts it on the node. No machine is touched.
- Reboot, then replace. For a problem that means the node is unlikely to recover, a cloud node is rebooted and, if that does not help, replaced.
- Reported for a human. Integrity and hardware signals are report-only. They tell you something needs a look; the platform does not act on its own.
Bare metal is more conservative: a dedicated server is not reprovisioned for a fault a reboot would not fix, or for one that could hit many servers at once, so those signals alert a person instead of churning hardware. For exactly what gets replaced and when, see Machine health checks and remediation .
See a node's conditions#
List a node's conditions and the short reason on each:
$ kubectl get node <node-name> -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}{"\n"}{end}'
Syself Autopilot also writes a fuller health report onto the node, so you can read the detail behind a signal without connecting to it. See Node labels and annotations for how to read it.
Note
Watch that the report keeps updating. If it goes stale while Ready still looks fine, the health reporting itself has stopped, which is worth an alert of its own.
Alert on the ones that need you#
Most signals either self-heal or trigger a replacement, and you can leave those to the platform. Route the rest, the certificate, disk-failure, hardware, and integrity signals, to your own monitoring, because they do not fix themselves and a person has to act.
Syself Autopilot exposes node health as metrics you can pull into your existing stack; see Scrape control-plane metrics . For the security and integrity signals, forward the records to your SIEM: see Ship audit logs to a SIEM . At a minimum, alert on two things: Ready going bad (the node is gone) and the health report going stale (the reporting is gone).
When an integrity signal fires, treat the node as untrusted, reprovision it, and follow Respond to a tampered node . For a failing certificate, see Certificate expiry and rotation .