Three audit streams are your evidence trail, and all three live on the node and rotate away. Ship them off-node, because a node that fails and gets replaced takes them with it, and an audit trail that a node loss can erase is not much of an audit trail.

## The audit streams

The three streams are the host `auditd` log, the Kubernetes API audit log, and the KubeGate audit log. For what each records, which nodes carry it, and its path, see the inventory on [Ship audit logs to a SIEM](/docs/hetzner/apalla/security/ship-audit-logs-to-a-siem).

The API and KubeGate streams exist only on control-plane nodes, because that is where the API server and its front gate run. Each API server logs only the requests it handled, so on a multi-control-plane cluster the full API trail is the union of the per-node files, which is a reason to tag each stream with its node.

## The collector already has them

The [System Alloy](/docs/hetzner/apalla/observability/collection/deploy-the-system-alloy) runs on every node, control planes included, and its config tails all three paths. If you deployed it, these streams are already shipping and you do not need a second DaemonSet for them. Two properties of that setup are what make it work here, and both are easy to get wrong:

- **It runs as root.** All three streams are mode `0600`. `auditd` and the API audit log are owned by root; the KubeGate log is owned by the unprivileged user KubeGate itself runs as (UID `65532`). No unprivileged collector can read any of them, so a collector that drops to a non-root user starts cleanly, finds the files, and reads nothing.
- **It lands on control-plane nodes.** Without a toleration for the control-plane taint, the two most important streams are never collected at all, and nothing about the setup looks broken.

On worker nodes the two control-plane paths are absent or empty. The agent finds no file to tail and stays quiet, which is expected rather than a fault.

## Parse the API audit stream

The Kubernetes API audit log is JSON, one event per line, and the fields you query on are inside it. Lifting them out at collection time makes them queryable without parsing at read time:

```terraform
loki.process "api_audit" {
  stage.json {
    expressions = {verb = "verb", user = "user.username", uri = "requestURI"}
  }
  forward_to = [loki.write.central.receiver]
}
```

Extract the fields you actually query on, and no more. Every extracted field you promote to a label multiplies Loki's stream count, and a user name or a request URI has enough distinct values to do real damage. Keep them as extracted fields for filtering, and label only the stream and the node.

The `auditd` stream is not JSON and goes through unparsed as raw text. The KubeGate stream is JSON, but it also goes through unparsed in this pipeline; its fields are best extracted dynamically at query time (e.g., `| json` in LogQL) rather than turned into permanent stream labels. Tag each stream with a distinct `job` so the three never mix.

> [!NOTE]
> The two highest-volume Cilium datapath events are not shipped, which is most of the raw volume, because Hubble already captures them and can attribute each one to a workload. What reaches your store is the security-relevant remainder.

> [!IMPORTANT]
> A single API audit event can exceed Loki's default 256 KB entry limit, and Loki rejects an oversized entry rather than truncating it, so the largest events are dropped while every ordinary one arrives. Raise `max_line_size` when you [set up Loki](/docs/hetzner/apalla/observability/logs/set-up-loki), and alert on `loki_write_dropped_entries_total{reason="ingester_error"}` so a silently incomplete trail cannot pass an audit.

## Keep them write-once

The durable copy has to live where the cluster's own credentials cannot rewrite it, or an attacker who reached the cluster edits the trail after the fact. Ship to a store outside the cluster with write-once, append-only retention. The full tamper-proof checklist (object-lock or WORM, write-only credentials, a separate trust domain) is on [Ship audit logs to a SIEM](/docs/hetzner/apalla/security/ship-audit-logs-to-a-siem). To read a stream by hand without a pipeline, see [Retrieve audit logs](/docs/hetzner/apalla/security/retrieve-audit-logs).

## Compliance retention windows

Your framework sets how long to keep the trail (many require a year or more), which is far longer than the on-node buffer holds. Set the retention on the off-node store to the window your framework requires, and keep the three streams for the same period so an auditor gets a complete picture.

This is also where the cost argument splits from operational logs. Audit streams are small and high-value, so keeping them for a year on cheap object storage costs little; container logs are large and mostly worthless after a week. Treat them differently rather than applying one retention to everything. The on-node caps are in [Log retention and sizing](/docs/hetzner/apalla/observability/logs/log-retention-and-sizing).