Ship audit logs off-node

Three audit streams are your evidence trail, and all three live on the node and rotate away. Ship them off-node, because a node that fails and gets replaced takes them with it.

The audit streams

The three streams are the host auditd log, the Kubernetes API audit log, and the KubeGate audit log. For what each records, which nodes carry it, and its path, see the table on Log sources on a sealed node .

The API and KubeGate streams exist only on control-plane nodes, and each control plane holds only its own slice of the API trail, so tag each stream with the node it came from.

The collector already has them

The System Alloy runs on every node, control planes included, and its config tails all three paths. If you deployed it, these streams are already shipping and you do not need a second DaemonSet for them. Two properties of that setup are what make it work here, and both are easy to get wrong:

• It runs as root. All three streams are mode 0600. auditd and the API audit log are owned by root; the KubeGate log is owned by the unprivileged user KubeGate itself runs as (UID 65532). No unprivileged collector can read any of them, so a collector that drops to a non-root user starts cleanly, finds the files, and reads nothing.
• It lands on control-plane nodes. Without a toleration for the control-plane taint, the two most important streams are never collected at all, and nothing about the setup looks broken.

On worker nodes the two control-plane paths are absent or empty. The agent finds no file to tail and stays quiet, which is expected rather than a fault.

Parse the API audit stream

The Kubernetes API audit log is JSON, one event per line, and the fields you query on are inside it. Lifting them out at collection time makes them queryable without parsing at read time:

		loki.process "api_audit" {
  stage.json {
    expressions = {verb = "verb", user = "user.username", uri = "requestURI"}
  }
  forward_to = [loki.write.central.receiver]
}
	

Extract only the fields you query on. Every field you promote to a label multiplies Loki's stream count, and a user name or request URI has enough distinct values to do real damage. Keep them as extracted fields for filtering, and label only the stream and the node.

The auditd stream is not JSON and ships as raw text. The KubeGate stream is JSON but ships unparsed too; extract its fields at query time with | json in LogQL instead of turning them into labels. Tag each stream with a distinct job so the three never mix.

Keep them write-once

Store the off-node copy outside the cluster, with write-once, append-only retention, so an attacker who reaches the cluster cannot edit the trail afterwards. The full checklist (object-lock, write-only credentials, a separate trust domain) is on Ship audit logs to a SIEM . To read a stream by hand, see Retrieve audit logs .

Compliance retention windows

Keep the trail as long as your compliance framework requires, often a year or more, and apply the same retention to all three streams so an auditor gets a complete picture. That is affordable: audit streams are small, so a year on object storage costs little, unlike container logs, which are large and rarely worth keeping past a week. The on-node caps are in Log retention and sizing .