Kubernetes Audit Logs With the Real Client IP

Kubernetes audit logs answer a plain question after an incident: who did what, and when. Syself Autopilot 1.36 produces four audit signals on every node, all on by default, and because KubeGate carries the client's real address through to the API server, your audit log names the actual caller instead of a load balancer's IP. Together the four cover who reached the API, who changed what, and what the operating system itself saw.

Every node here runs on the public internet and gets replaced rather than patched. The audit trail is built to hold up on exactly that kind of node.

What are the four audit signals?

• Host audit (auditd). Every node runs the Linux audit daemon with a fixed ruleset scoped for the BSI C5 cloud security standard, built from source and locked until reboot. It records what happened on the node itself: file and identity changes, privilege use, module loads, every command in a login session.
• Kubernetes API audit. On control-plane nodes, a record of who did what to which object through the API server. Writes are logged in full; secrets and configmaps are recorded at metadata level, so their values never land in the log.
• KubeGate audit. The allow and deny decisions the API server's front gate makes, each recorded with its real origin.
• Tamper detection. The node's few writable paths are watched, and a change to a protected file raises a NodeTampered condition you can alert on. The rest of the OS is read-only and verified on every read.

Why does the real client IP matter?

An audit log that pins every request on the same internal hop can't tell you who actually did something. Behind a load balancer, a plain API server can only record the balancer's address, not the caller's, so every request pins to that one hop. KubeGate opens the connection from the client's genuine address, so the last address in each audit entry is the real caller, written by the kernel rather than read from a header. A client can't forge it. For an investigation, or a compliance framework, that's the difference between a log and evidence.

The forgery protection covers that last address only. A client can still put whatever it likes into the earlier entries of the record, so alert on the last address and never trust the first.

Where do the logs go? Retention is your job

Nothing ships off-node by default. Each node keeps its own logs, and they hold only a short window: auditd keeps up to about 400 MB, the API audit log about a gigabyte or 90 days, KubeGate about 30 days. After that, old entries are overwritten. And because a node gets replaced rather than patched, its local logs are erased the moment self-healing swaps it out.

So ship whatever you need to keep. Run a log shipper as a DaemonSet: it mounts the three log paths read-only and forwards them to durable storage, whether that's a SIEM, Loki, or write-once object storage for compliance. This is the same pipeline our own release checks use. Send the logs to a store that refuses deletes, in a separate account, so a compromised node can't erase what it already sent. Ship audit logs to a SIEM has the copy-paste manifests, and Retrieve audit logs covers reading a log by hand.

FAQ

Which audit signals do I get without configuring anything?

All four — host auditd, the Kubernetes API audit, the KubeGate audit, and tamper detection — are on by default. What you add is where they ship and how long they're kept.

Are the audit logs suitable for compliance?

The host audit ruleset is scoped for BSI C5, the German government's cloud-security standard and one of the most rigorous anywhere. Together the four signals give you what frameworks like ISO 27001, SOC 2, BSI C5, and NIS2 ask for: a record of who did what, with the real caller named and tampering flagged. Those frameworks also want the trail kept off the machine that produced it, so a node loss or an attacker can't erase it. That part is the log shipping you set up.

How do I keep audit logs after a node is replaced?

Ship them off-node before it happens. A DaemonSet that tails the three log paths and forwards them to durable storage keeps the record after the node is gone.

Which IP address in the audit log can I trust?

The last address in the sourceIPs list. The kernel writes it and no client can influence it, so it names the real caller. The earlier entries come from client-supplied headers: keep them for context, but alert on the last one.

Audit tells you what happened. Catching trouble as it happens, and fixing it, is a different job on the node.