Collect container and journald logs

Two operational log streams exist on every node, and both are temporary. Container stdout and stderr land under /var/log/pods/, and the systemd journal holds the output of the node's own services. Both live on the writable /var partition, which means both vanish when Syself Autopilot replaces the node, so both have to be shipped off it. The System Alloy does that as part of its config, mounting the host's /var/log read-only and tailing what it finds there.

The collector runs as a DaemonSet because nothing can be installed into the sealed OS (Log sources on a sealed node ).

Container logs, and the labels that make them queryable

Raw container logs are files named after pod UIDs, which cannot be queried. What makes them worth storing is the Kubernetes metadata attached to each line: namespace, pod, container, and node. With those labels a query reads the way you think about the system:

		{namespace="team-a", container="api"} |= "timeout"
	

Two mechanics behind that are worth knowing when it does not work.

• The log lines on disk are wrapped in the CRI format: one envelope per line, carrying a timestamp and a stream name around your actual message. The pipeline unwraps them before storage. Skip that and every line arrives with the wrapper embedded in it.
• The file paths have to be discovered by expanding a glob per container, which is a separate step from tailing. Handing a glob straight to a file source tails nothing at all, silently, which is the single most common reason container logs never appear.

journald, and why you filter it

The journal holds output from the node's own services: the kubelet, containerd, and the syself-* daemons that run health checking, the failover proxy, and the tunnel. That is the useful part. It also holds output from every other unit on the machine, most of which nobody will ever read, and you pay for all of it in ingest and storage.

The collector keeps the units that matter and drops the rest. The System Alloy's config filters on the unit name for exactly this, keeping kubelet, containerd, and the syself-* services. Widen it when you are chasing something specific, and narrow it again afterward.

Handle the burst

Log volume jumps during an incident, exactly when the store is also under load. Alloy buffers the overflow to disk at its storagePath, so a burst becomes delay rather than loss. Size that buffer for a realistic peak: the same agent carries the node's metrics, so a log spike must not take it down. Log retention and sizing has the per-node estimates to size against.

Confirm lines land

In Grafana's Explore view against the Loki data source, query a namespace you know is busy:

		{namespace="kube-system"}
	

If nothing appears, Troubleshoot the collectors covers finding where the pipeline stops.

Once the operational streams flow, add the ones that exist for compliance rather than debugging: Ship audit logs off-node .