Retrieve audit logs

A Syself Autopilot cluster keeps three audit logs on its nodes. One records OS-level events, one records Kubernetes API activity, and one records the decisions of the API server's front gate. This page covers where each log lives, what each captures, and how to pull them over SSH or with kubectl. All three have local size limits, and nothing ships them by default, so for compliance you need to retrieve them and send them to durable off-node storage.

The audit logs are the security view of the observability model : the platform writes the records on the node, and you own the storage and the history.

Prerequisites#

• SSH key for port 100, or kubectl access with permission to create Jobs in kube-system.
• For log shipping: a destination bucket or SIEM (Security Information and Event Management system, a platform that collects and searches security logs) with object-lock or WORM (write-once read-many) retention.

The three audit logs#

The host auditd log is at /var/log/audit/audit.log on every node. The Kubernetes API audit log is at /var/log/kube-apiserver/kube-apiserver.log and the KubeGate audit log at /var/log/kubegate/audit.log, both on control-plane nodes only. For what each one records, see the inventory on Ship audit logs to a SIEM . Two points matter when you read them by hand:

• The API audit log is per control-plane node. Each API server logs only the requests it handled. To see all API activity, collect from every control-plane node.
• All three logs are stored only on the node, with limited size and retention. Old entries are eventually overwritten. auditd keeps up to 400 MB (8 rotated files of 50 MB). The API audit log rotates at 100 MB per file and keeps up to 10 rotated files (about 1 GB total), or 90 days, whichever comes first. KubeGate keeps about 30 days. These are short-term buffers of days to weeks, not the system of record. To keep records long-term and tamper-resistant, ship all three logs off-node. See Ship logs off-node below.

What the Kubernetes API audit captures#

The audit policy at /etc/kubernetes/audit-policy.yaml is read-only and platform-managed. It records events at different detail levels:

• Every write (create, update, patch, delete) is logged in full, including CRDs (Custom Resource Definitions) and Cilium network policies. Three exceptions keep the log usable: create on Events is dropped (their update, patch and delete are still recorded in full, so covering tracks is audited), writes to pods/status are recorded at metadata level, and nodes/status updates from kubelets are recorded without the response body.
• secrets, configmaps, and token requests are logged at metadata level only. The actual values and tokens are never written to the log.
• Impersonation is logged with both identities: who impersonated whom.
• High-volume, low-risk reads are dropped to keep the log useful: kube-proxy watches on endpoints and services, kubelet gets on Node objects, event reads, and health-check and other non-resource URLs. Kubelet watches themselves are still recorded, at request level.

Reading sourceIPs: which address you can trust#

Every event carries a sourceIPs array. It is not a list of hops you can trust end to end, and reading it as one is the most common mistake made with Kubernetes audit logs. The API server builds it from the X-Forwarded-For header, then X-Real-Ip, then appends the address of the TCP connection it actually received. It does not validate any of the headers, so any client can put any addresses at the front of that array, including addresses that belong to your own infrastructure.

The last element is different: the kernel writes it, and no client can influence it.

Use exactly this mapping in your SIEM:

Preserving the client address does not stop header forgery. It makes the one element that was already unforgeable, the last one, carry a real address instead of a loopback one. Everything before it stays client-supplied.

What the last element tells you#

sourceIPs[-1] takes one of these shapes, and they mean different things:

Both 127.0.7.1 and 127.0.7.2 are fixed markers for a category, not an encoding of anyone's address. A client that puts either in a header only lands it in sourceIPs[0], which you never trust anyway.

The split matters. Before it existed, a degraded gate and a bypassed gate both read 127.0.0.1 and no query could tell them apart. Now 127.0.0.1 means only "did not come through the gate", which is the case no metric on the node can see, and 127.0.7.2 means "the gate handled this and could not name the caller", which the node does report as a metric and a node condition.

The rules that follow from that:

Alert when sourceIPs[-1] == "127.0.0.1" exceeds your baseline. A small, steady volume is normal: the control-plane kubelet talks to its API server over loopback. A rise above that baseline means something is reaching the API server on the node itself rather than through the front gate. Failed authentication is audited too, so this catches unauthenticated probing as well as authenticated activity.

Alert when sourceIPs[-1] == "127.0.7.2" appears at all. It means client attribution on that node is broken: the gate served the connection and could not name the caller. Run syself-node auditsrc status on the node.

One case is not a fault. A node image older than syself-auditsrc.service starts the helper after the gate, so every boot on such a node produces a burst while the gate serves traffic the helper cannot yet attribute. On a node that runs that unit the helper starts first, and a boot burst there means the ordering regressed. systemctl status syself-auditsrc.service tells the two apart.

One case has already been fixed by the time you look. A connection keeps the source address it was opened with, so a client that connected during a boot burst goes on recording 127.0.7.2 on a node where the helper is now healthy and syself-node auditsrc status comes back clean. It stops when that client reconnects, and kube-controller-manager never does on its own. The signal is that the events all belong to one caller and arrive at a steady rate rather than in a burst. Deleting that pod clears it, at the cost of a leader election.

One case is not the node's fault either, and it is the one to rule out before you go looking at the node. The gate learns a client's address from a header the load balancer adds. If the balancer stops adding it (proxy protocol switched off, or a balancer replaced), every request that arrives through it is marked 127.0.7.2 while syself-node auditsrc status on every node comes back clean. The signal is that the burst covers all external traffic on every control plane at once and starts at a configuration change rather than at a boot.

The reason this shows up as 127.0.7.2 rather than as a plausible address is worth knowing, because it decides what you can trust the table above to mean. Without the header, the address the gate can see is the load balancer's own. Recording that would put a real, routable address in sourceIPs[-1] and attribute every external client on the cluster to the balancer, with nothing anywhere in the event to say the attribution was wrong. The gate refuses to do that and records the marker instead: it would rather tell you it does not know than tell you something false. So if your own control-plane load balancer's address ever appears as a client in this field, treat it as a finding and not as traffic.

Alert when sourceIPs[-1] == "127.0.7.1" appears at all. It is not an error and nothing is broken, but it means an IPv6 client is reaching your control-plane endpoint and every request it makes is unattributable. The fix is in DNS, not on the cluster: drop the AAAA record.

Delete any rule that suppresses 127.0.0.1 as control-plane noise. With such a rule in place, sending X-Forwarded-For: 127.0.0.1 becomes the cheapest possible way to make an event invisible to your SIEM.

Retrieve a log from a node#

Read a log directly on the node over SSH, or pull it through the API server with a temporary kubectl Job when SSH is not available.

		# Tail the Kubernetes API audit log as JSON (run on a control-plane node):
$ ssh -p 100 -i <your-key> root@<cp-node-ip> \
  'tail -n 50 /var/log/kube-apiserver/kube-apiserver.log' | jq .
 
# Find every write to secrets by a given user:
$ ssh -p 100 -i <your-key> root@<cp-node-ip> \
  'grep secrets /var/log/kube-apiserver/kube-apiserver.log' | jq 'select(.verb!="get")'
 
# The host audit log:
$ ssh -p 100 -i <your-key> root@<node-ip> 'tail -n 50 /var/log/audit/audit.log'
	

		$ ssh -p 100 -i <your-key> root@<node-ip> 'syself-log-collector'
$ scp -P 100 -i <your-key> -o IdentitiesOnly=yes \
  root@<node-ip>:/var/log/syself-node-report/latest.tar.gz .
	

		$ kubectl apply -f - <<'EOF'
apiVersion: batch/v1
kind: Job
metadata:
  name: log-collector
  namespace: kube-system
spec:
  template:
    spec:
      nodeName: <NODE_NAME>
      hostPID: true
      restartPolicy: Never
      containers:
        - name: collect
          image: busybox
          securityContext: { privileged: true }
          command: ["chroot", "/host", "/usr/bin/syself-log-collector",
                    "--stdout", "--stdout-encoding=base64"]
          volumeMounts: [{ name: host, mountPath: /host }]
      volumes:
        - name: host
          hostPath: { path: / }
EOF
 
# Decode the streamed bundle:
$ kubectl -n kube-system logs job/log-collector \
  | sed -n '/BEGIN SYSELF BUNDLE/,/END SYSELF BUNDLE/p' | sed '1d;$d' \
  | base64 -d > report.tar.gz
$ kubectl -n kube-system delete job log-collector
	

Ship logs off-node#

The node keeps audit logs locally only, and nothing ships them by default. Retention and immutability are your responsibility. Local logs are tamper-evident, not tamper-proof: the node raises a NodeTampered condition when a protected file changes, but a root-level attacker who stops the daemon can still alter local files. Only evidence that already left the node survives an incident.

Ship all three logs (host auditd, Kubernetes API, and KubeGate) to a store that refuses deletes. Ship audit logs to a SIEM is the copy-paste pipeline, a Grafana Alloy DaemonSet that tails all three logs, plus the full tamper-proof destination checklist and the end-to-end verification. Forward the integrity signals alongside them: watch each Node object for NodeTampered transitions and the autopilot.syself.com/tamper annotation (see Node health conditions ).