Debug a node
Most node problems never reach you. Syself Autopilot spots them as node conditions and, where it safely can, replaces the node on its own (see Machine health checks and remediation ). You only reach for this page when a node is stuck below Kubernetes and you want to see why yourself. When that happens, get onto the node and read its boot and join logs. Most lines are noise; a few point to a real cause.
Where to look first#
Get onto the node with emergency SSH , or read the serial console if it will not boot. Then, in rough order:
| Source | What it tells you |
|---|---|
| Serial console | Kernel panics and dm-verity errors, before systemd even starts |
/var/log/syself-activate.log | The provisioning and join sequence. Read it with cat: journalctl -u syself-activate shows only systemd's start and stop lines, not the activation output itself. |
crictl ps -a / crictl pods | Whether the container runtime is up and what is running |
journalctl -u kubelet -b / -u containerd -b | kubelet and runtime logs for this boot |
dmsetup status root | The verity device state (a healthy root shows verity V) |
The standard diagnostics (dmesg, lsblk, mount, ip, ss, crictl, dmsetup) are present in the base OS.
Common failures#
For most problems, the node's health conditions name the cause faster than the logs do, see Node problem detection and conditions . One log line is worth knowing on sight:
| Symptom (log line) | Cause | Fix |
|---|---|---|
device-mapper: verity: ... data block ... is corrupted | The root image failed its integrity check: a corrupted or incomplete write, or a change to the OS after it was sealed. | Reprovision the node. There is no verity-off boot entry, so the OS cannot be repaired in place. |
Benign log noise you can ignore#
Healthy nodes print plenty of harmless boot and disconnect messages. Do not chase them. A closed kubectl logs or exec stream, occasional pooled-connection churn, headless-server and cloud-VM boot quirks, and brief startup races before the network is up are all normal and clear on their own. When in doubt, watch the node's health conditions instead of the raw log: they tell you whether something is actually wrong.
Warning
One thing is not noise: containerd memory that climbs and stays high is worth investigating as a possible leak.
If you would rather watch health signals than chase log lines, see Node problem detection and conditions .