Skip to main content

Log sources on a sealed node

Inspect 1.36

A Syself Autopilot node runs , a sealed OS: the root filesystem is read-only, and only /var is writable. Every log stream lives on /var, which means every log stream disappears when the node is reprovisioned. Knowing what is there, and that it is temporary, is the reason the rest of this sub-topic ships logs off-node.

The streams#

Source Path Nodes What it records
Container logs /var/log/pods/ all stdout and stderr from every container
journald (the systemd log) /var/log/journal/ all systemd service output (kubelet, containerd, the node daemons)
auditd (the Linux OS audit log) /var/log/audit/audit.log all OS-level events: file and identity changes, privilege use, module loads, every command in a login session
Kubernetes API audit /var/log/kube-apiserver/kube-apiserver.log control planes who did what to which Kubernetes object
KubeGate audit /var/log/kubegate/audit.log control planes the allow and deny decisions the API server's front gate made (TLS and source-address checks)

The first three are on every node. The two API-side audit streams exist only on control-plane nodes, because that is where the API server and its front gate run. Each API server logs only the requests it handled, so on a multi-control-plane cluster the full API trail is the union of the per-node files.

Warning

Everything in /var is lost when a node is replaced, and the platform replaces nodes rather than repairing them. The on-node copies rotate and are a bounded hot buffer, not a system of record. Ship what you need off-node before a node goes. See and .

Why you cannot install a log agent in the OS#

The root filesystem is read-only and there is no package manager, so you cannot install a collector into the OS the way you would on a normal server. That is not a limitation to work around; it is why every collector here runs as a Kubernetes DaemonSet, a pod that runs on every node. The DaemonSet mounts the host log paths and streams them out. The node stays sealed, and the shipping is a workload you own and upgrade like any other.

For the on-node size caps and how to size retention in your own store, see .