A Syself Autopilot cluster keeps three audit logs on its nodes. One records OS-level events, one records Kubernetes API activity, and one records the decisions of the API server's front gate. This page covers where each log lives, what each captures, and how to pull them over SSH or with kubectl. All three have local size limits, and nothing ships them by default, so for compliance you need to retrieve them and send them to durable off-node storage. The audit logs are the security view of the [observability model](/docs/hetzner/apalla/concepts/operations/observability-model): the platform writes the records on the node, and you own the storage and the history. ## Prerequisites - SSH key for port 100, or `kubectl` access with permission to create Jobs in `kube-system`. - For log shipping: a destination bucket or SIEM (Security Information and Event Management system, a platform that collects and searches security logs) with object-lock or WORM (write-once read-many) retention. ## The three audit logs The host `auditd` log is at `/var/log/audit/audit.log` on every node. The Kubernetes API audit log is at `/var/log/kube-apiserver/kube-apiserver.log` and the KubeGate audit log at `/var/log/kubegate/audit.log`, both on control-plane nodes only. For what each one records, see the inventory on [Ship audit logs to a SIEM](/docs/hetzner/apalla/security/ship-audit-logs-to-a-siem). Two points matter when you read them by hand: - The API audit log is per control-plane node. Each API server logs only the requests it handled. To see all API activity, collect from every control-plane node. - All three logs are stored only on the node, with limited size and retention. Old entries are eventually overwritten. `auditd` keeps up to 400 MB (8 rotated files of 50 MB). The API audit log rotates at 100 MB per file and keeps up to 10 rotated files (about 1 GB total), or 90 days, whichever comes first. KubeGate keeps about 30 days. These are short-term buffers of days to weeks, not the system of record. To keep records long-term and tamper-resistant, ship all three logs off-node. See [Ship logs off-node](#ship-logs-off-node) below. ### What the Kubernetes API audit captures The audit policy at `/etc/kubernetes/audit-policy.yaml` is read-only and platform-managed. It records events at different detail levels: - Every write (`create`, `update`, `patch`, `delete`) is logged in full, including CRDs (Custom Resource Definitions) and Cilium network policies. Three exceptions keep the log usable: `create` on Events is dropped (their `update`, `patch` and `delete` are still recorded in full, so covering tracks is audited), writes to `pods/status` are recorded at metadata level, and `nodes/status` updates from kubelets are recorded without the response body. - `secrets`, `configmaps`, and token requests are logged at metadata level only. The actual values and tokens are never written to the log. - Impersonation is logged with both identities: who impersonated whom. - High-volume, low-risk reads are dropped to keep the log useful: kube-proxy watches on endpoints and services, kubelet `get`s on Node objects, event reads, and health-check and other non-resource URLs. Kubelet watches themselves are still recorded, at request level. ### Reading `sourceIPs`: which address you can trust Every event carries a `sourceIPs` array. It is not a list of hops you can trust end to end, and reading it as one is the most common mistake made with Kubernetes audit logs. The API server builds it from the `X-Forwarded-For` header, then `X-Real-Ip`, then appends the address of the TCP connection it actually received. It does not validate any of the headers, so **any client can put any addresses at the front of that array**, including addresses that belong to your own infrastructure. The last element is different: the kernel writes it, and no client can influence it. Use exactly this mapping in your SIEM: | Field | Expression | Meaning | | ------------------- | -------------------- | -------------------------------------------------------------------------------------------------------------------------- | | `client_ip_trusted` | `sourceIPs[-1]` | the real peer. Unforgeable. Alert on this one. | | `client_ip_claimed` | `sourceIPs[0]` | whatever the client asked to be recorded as. Keep it for context, **never** alert on it. | | `source_ip_forged` | `len(sourceIPs) > 1` | the client sent forwarding headers. On this platform nothing legitimately does, so treat it as a strong signal on its own. | Preserving the client address does **not** stop header forgery. It makes the one element that was already unforgeable, the last one, carry a real address instead of a loopback one. Everything before it stays client-supplied. ### What the last element tells you `sourceIPs[-1]` takes one of these shapes, and they mean different things: | `sourceIPs[-1]` | Meaning | What to do | | ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | a routable address | attributed. This is the real caller. | normal operation | | `127.0.7.1` | an IPv6 client. Its address arrived at the load balancer but cannot be carried onto an IPv4 connection, so it is unattributable by construction. The client was served normally. | remove the `AAAA` record for your control-plane endpoint | | `127.0.7.2` | the request came through the front gate, which could not name the caller. Either the helper on the node was not serving yet or its routing broke, or the load balancer stopped sending the header that carries the client's address. The client was served normally. | run `syself-node auditsrc status` on the node and check `syself-auditsrc.service`; if both are healthy, check proxy protocol on the load balancer. See the alert below | | `127.0.0.1` | the request did not come through the front gate at all. Something reached the API server on the node itself. | investigate; see the alert below | Both `127.0.7.1` and `127.0.7.2` are fixed markers for a category, not an encoding of anyone's address. A client that puts either in a header only lands it in `sourceIPs[0]`, which you never trust anyway. The split matters. Before it existed, a degraded gate and a bypassed gate both read `127.0.0.1` and no query could tell them apart. Now `127.0.0.1` means only "did not come through the gate", which is the case no metric on the node can see, and `127.0.7.2` means "the gate handled this and could not name the caller", which the node does report as a metric and a node condition. The rules that follow from that: **Alert when `sourceIPs[-1] == "127.0.0.1"` exceeds your baseline.** A small, steady volume is normal: the control-plane kubelet talks to its API server over loopback. A rise above that baseline means something is reaching the API server on the node itself rather than through the front gate. Failed authentication is audited too, so this catches unauthenticated probing as well as authenticated activity. **Alert when `sourceIPs[-1] == "127.0.7.2"` appears at all.** It means client attribution on that node is broken: the gate served the connection and could not name the caller. Run `syself-node auditsrc status` on the node. One case is not a fault. A node image older than `syself-auditsrc.service` starts the helper after the gate, so every boot on such a node produces a burst while the gate serves traffic the helper cannot yet attribute. On a node that runs that unit the helper starts first, and a boot burst there means the ordering regressed. `systemctl status syself-auditsrc.service` tells the two apart. One case has already been fixed by the time you look. A connection keeps the source address it was opened with, so a client that connected during a boot burst goes on recording `127.0.7.2` on a node where the helper is now healthy and `syself-node auditsrc status` comes back clean. It stops when that client reconnects, and `kube-controller-manager` never does on its own. The signal is that the events all belong to one caller and arrive at a steady rate rather than in a burst. Deleting that pod clears it, at the cost of a leader election. One case is not the node's fault either, and it is the one to rule out before you go looking at the node. The gate learns a client's address from a header the load balancer adds. If the balancer stops adding it (proxy protocol switched off, or a balancer replaced), every request that arrives through it is marked `127.0.7.2` while `syself-node auditsrc status` on every node comes back clean. The signal is that the burst covers all external traffic on every control plane at once and starts at a configuration change rather than at a boot. The reason this shows up as `127.0.7.2` rather than as a plausible address is worth knowing, because it decides what you can trust the table above to mean. Without the header, the address the gate can see is the **load balancer's own**. Recording that would put a real, routable address in `sourceIPs[-1]` and attribute every external client on the cluster to the balancer, with nothing anywhere in the event to say the attribution was wrong. The gate refuses to do that and records the marker instead: it would rather tell you it does not know than tell you something false. So if your own control-plane load balancer's address ever appears as a client in this field, treat it as a finding and not as traffic. **Alert when `sourceIPs[-1] == "127.0.7.1"` appears at all.** It is not an error and nothing is broken, but it means an IPv6 client is reaching your control-plane endpoint and every request it makes is unattributable. The fix is in DNS, not on the cluster: drop the `AAAA` record. **Delete any rule that suppresses `127.0.0.1` as control-plane noise.** With such a rule in place, sending `X-Forwarded-For: 127.0.0.1` becomes the cheapest possible way to make an event invisible to your SIEM. > [!NOTE] > **Publish only an `A` record for your control-plane endpoint, never an `AAAA`.** Client attribution is carried over IPv4. A client that reaches the endpoint over IPv6 keeps working exactly as before (this costs attribution, not reachability), but its address cannot be recorded, and its events are marked `127.0.7.1` instead of naming it. Hetzner load balancers are assigned both an IPv4 and an IPv6 address by default, so if your endpoint is a hostname it is easy to publish an `AAAA` by accident and quietly lose attribution for everyone who resolves it. ## Retrieve a log from a node Read a log directly on the node over SSH, or pull it through the API server with a temporary kubectl Job when SSH is not available. SSH runs on **port 100**, key-only. Read a log directly or copy it to your machine: ```console # Tail the Kubernetes API audit log as JSON (run on a control-plane node): $ ssh -p 100 -i root@ \ 'tail -n 50 /var/log/kube-apiserver/kube-apiserver.log' | jq . # Find every write to secrets by a given user: $ ssh -p 100 -i root@ \ 'grep secrets /var/log/kube-apiserver/kube-apiserver.log' | jq 'select(.verb!="get")' # The host audit log: $ ssh -p 100 -i root@ 'tail -n 50 /var/log/audit/audit.log' ``` To capture a full bundle including the API audit directory and node diagnostics, use the node's log collector: Nothing produces this bundle on a schedule, so run the collector first and then copy the refreshed bundle. Copying without running it either fails with "No such file or directory" or hands you a stale bundle from whenever someone last ran it. ```console $ ssh -p 100 -i root@ 'syself-log-collector' $ scp -P 100 -i -o IdentitiesOnly=yes \ root@:/var/log/syself-node-report/latest.tar.gz . ``` When SSH is not available but the API server is reachable, run the log collector in a temporary privileged pod and stream the bundle through `kubectl logs`. Note: `kubectl cp` does not work for this. ```console $ kubectl apply -f - <<'EOF' apiVersion: batch/v1 kind: Job metadata: name: log-collector namespace: kube-system spec: template: spec: nodeName: hostPID: true restartPolicy: Never containers: - name: collect image: busybox securityContext: { privileged: true } command: ["chroot", "/host", "/usr/bin/syself-log-collector", "--stdout", "--stdout-encoding=base64"] volumeMounts: [{ name: host, mountPath: /host }] volumes: - name: host hostPath: { path: / } EOF # Decode the streamed bundle: $ kubectl -n kube-system logs job/log-collector \ | sed -n '/BEGIN SYSELF BUNDLE/,/END SYSELF BUNDLE/p' | sed '1d;$d' \ | base64 -d > report.tar.gz $ kubectl -n kube-system delete job log-collector ``` The collector's `audit` component bundles the Kubernetes API audit directory (on control-plane nodes only). Read the host `auditd` log directly from `/var/log/audit/`, over SSH or with the privileged pod above. ## Ship logs off-node The node keeps audit logs locally only, and nothing ships them by default. Retention and immutability are your responsibility. Local logs are tamper-evident, not tamper-proof: the node raises a `NodeTampered` condition when a protected file changes, but a root-level attacker who stops the daemon can still alter local files. Only evidence that already left the node survives an incident. Ship all three logs (host `auditd`, Kubernetes API, and KubeGate) to a store that refuses deletes. [Ship audit logs to a SIEM](/docs/hetzner/apalla/security/ship-audit-logs-to-a-siem) is the copy-paste pipeline, a Grafana Alloy DaemonSet that tails all three logs, plus the full tamper-proof destination checklist and the end-to-end verification. Forward the integrity signals alongside them: watch each Node object for `NodeTampered` transitions and the `autopilot.syself.com/tamper` annotation (see [Node health conditions](/docs/hetzner/apalla/reference/node-health-conditions)).