Two operational log streams exist on every node, and both are temporary. Container stdout and stderr land under `/var/log/pods/`, and the systemd journal holds the output of the node's own services. Both live on the writable `/var` partition, which means both vanish when the platform replaces the node, so both have to be shipped off it. The [System Alloy](/docs/hetzner/apalla/observability/collection/deploy-the-system-alloy) does that as part of its config, mounting the host's `/var/log` read-only and tailing what it finds there.

The collector is a DaemonSet rather than an agent installed in the OS because the root filesystem is read-only and there is no package manager to install into ([Log sources on a sealed node](/docs/hetzner/apalla/observability/logs/log-sources-on-a-sealed-node)). The node stays sealed, and the shipping is a workload you own and upgrade like any other.

## Container logs, and the labels that make them queryable

Raw container logs are files named after pod UIDs, which cannot be queried. What makes them worth storing is the Kubernetes metadata attached to each line: namespace, pod, container, and node. With those labels a query reads the way you think about the system:

```text
{namespace="team-a", container="api"} |= "timeout"
```

Two mechanics behind that are worth knowing when it does not work. The log lines on disk are wrapped in the CRI format, one envelope per line carrying a timestamp and a stream name around your actual message, so the pipeline unwraps them before storage; skip that and every line arrives with the wrapper embedded in it. The file paths have to be discovered by expanding a glob per container, which is a separate step from tailing. Handing a glob straight to a file source tails nothing at all, silently, which is the single most common reason container logs never appear.

> [!IMPORTANT]
> Keep the label set small and stable. Namespace, pod, container, and node are worth indexing. A label carrying a request id, a trace id, or anything else with unbounded values multiplies Loki's stream count the same way a high-cardinality metric label multiplies series, and it degrades the store rather than the collector. Put that detail in the log line, where a query can filter on it, not in a label.

## journald, and why you filter it

The journal holds output from the node's own services: the kubelet, containerd, and the `syself-*` daemons that run health checking, the failover proxy, and the tunnel. That is the useful part. It also holds output from every other unit on the machine, most of which nobody will ever read, and you pay for all of it in ingest and storage.

The collector keeps the units that matter and drops the rest. The System Alloy's config filters on the unit name for exactly this, keeping `kubelet`, `containerd`, and the `syself-*` services. Widen it when you are chasing something specific, and narrow it again afterward.

> [!NOTE]
> Some collector versions read the journal from `/run/log/journal/` rather than `/var/log/journal/`. If journald produces nothing while container logs arrive normally, check which path systemd is writing to on that node and mount both.

## Handle the burst

Log volume is not steady. During an incident, error logging from a crash-looping workload can jump several-fold at exactly the moment the store is also under load, and the collector has to decide what to do with the overflow. Alloy buffers to disk at its `storagePath`, so a burst becomes delay rather than loss, and the buffer is bounded so it cannot consume the node.

The failure to avoid is the one where a log spike takes the collector down and you lose the node's metrics with it. Size the buffer volume for a realistic peak, and remember that the same agent is carrying this node's kubelet and etcd metrics. [Log retention and sizing](/docs/hetzner/apalla/observability/logs/log-retention-and-sizing) has the per-node estimates to size against.

## Confirm lines land

In Grafana's Explore view against the Loki data source, query a namespace you know is busy:

```text
{namespace="kube-system"}
```

If nothing appears, check which files the agent has actually opened before touching the config, because "discovered a target" and "opened a file" are different states:

```console
$ kubectl -n monitoring port-forward daemonset/system-alloy 12345:12345
$ curl -s localhost:12345/metrics | grep loki_source_file_file_bytes_total
```

A file per running container on that node means the container pipeline works; the journal has its own counter alongside it. [Troubleshoot the collectors](/docs/hetzner/apalla/observability/collection/troubleshoot-the-collectors) covers both when they are empty.

Once the operational streams flow, add the ones that exist for compliance rather than debugging: [Ship audit logs off-node](/docs/hetzner/apalla/observability/logs/ship-audit-logs).