Nodes keep audit logs locally, with size caps, and nothing ships them by default. The pipeline here is a Grafana Alloy DaemonSet that tails all three audit logs on every node and forwards them to a SIEM, Loki, or write-once storage. To read a log manually instead, see [Retrieve audit logs](/docs/hetzner/apalla/security/retrieve-audit-logs). If you already run the [System Alloy](/docs/hetzner/apalla/observability/collection/deploy-the-system-alloy), it tails these three logs on every node already; [Ship audit logs off-node](/docs/hetzner/apalla/observability/logs/ship-audit-logs) covers that route. This page is the standalone pipeline with the full SIEM and WORM setup. Off-node storage is your side of the [observability model](/docs/hetzner/apalla/concepts/operations/observability-model): the platform produces the audit trail, you decide where it lives and how long it is kept. ## Prerequisites - `kubectl` access to the workload cluster, with permission to create a namespace and a DaemonSet. No SSH access is needed for the pipeline. - A destination reachable from the cluster: a Loki instance, a SIEM (Security Information and Event Management system, a platform that collects and searches security logs), or an object store. For compliance, the destination needs object-lock or WORM (write-once read-many) retention. See [Make the destination tamper-proof](#make-the-destination-tamper-proof). ## The three audit logs | Log | What it records | Which nodes | Path on the node | | ------------------------- | ------------------------------------------------------------------------------------------------------------ | ------------------------ | -------------------------------------------- | | **Host audit (`auditd`)** | OS-level events: file and identity changes, privilege use, module loads, every command in a login session | every node | `/var/log/audit/audit.log` | | **Kubernetes API audit** | who did what to which Kubernetes object through the API server | control-plane nodes only | `/var/log/kube-apiserver/kube-apiserver.log` | | **KubeGate audit** | allow/deny decisions the apiserver front gate made: TLS (Transport Layer Security) and source-address checks | control-plane nodes only | `/var/log/kubegate/audit.log` | Together, the three logs answer: who reached the API server (KubeGate), who did what to which object (API audit), and what the OS saw (auditd). Where each log is written shapes the pipeline: - The API server and KubeGate run as static pods on control-plane nodes and write their logs through host-path mounts. The files are written to the node's `/var`, so a pod that mounts the host paths can read them. You do not need to reach inside the apiserver pod. - The API and KubeGate logs exist only on control-plane nodes, and each control-plane node logs only the requests it handled. The shipper must run on every node, control planes included. Control-plane nodes are tainted, so the DaemonSet needs a toleration to run there. The manifest below uses `operator: Exists`, which tolerates every taint. ## Deploy the shipper The manifest below runs Grafana Alloy as a DaemonSet. The ConfigMap holds the Alloy config (River syntax). One `loki.source.file` tails the three logs, tags each with a `job` label (`auditd`, `apiserver_audit`, `kubegate_audit`), and forwards to one `loki.write`. Edit the `loki.write` URL to point at your destination before you apply the manifest. ```yaml apiVersion: v1 kind: Namespace metadata: name: audit-shipping --- apiVersion: v1 kind: ConfigMap metadata: name: alloy-config namespace: audit-shipping data: config.alloy: | loki.write "default" { endpoint { // EDIT ME: your Loki push endpoint, in-cluster or external. url = "https://loki.example.com/loki/api/v1/push" } } loki.source.file "audit" { targets = [ {__path__ = "/var/log/audit/audit.log", job = "auditd"}, {__path__ = "/var/log/kube-apiserver/kube-apiserver.log", job = "apiserver_audit"}, {__path__ = "/var/log/kubegate/audit.log", job = "kubegate_audit"}, ] forward_to = [loki.write.default.receiver] } --- apiVersion: apps/v1 kind: DaemonSet metadata: name: audit-shipper namespace: audit-shipping spec: selector: matchLabels: app: audit-shipper template: metadata: labels: app: audit-shipper spec: # Land on every node, including tainted control planes. tolerations: - operator: Exists securityContext: runAsUser: 0 containers: - name: alloy image: grafana/alloy:v1.5.1 args: - run - /etc/alloy/config.alloy - --storage.path=/var/lib/alloy - --server.http.listen-addr=0.0.0.0:12345 securityContext: privileged: true volumeMounts: - name: config mountPath: /etc/alloy - name: positions mountPath: /var/lib/alloy - name: audit mountPath: /var/log/audit readOnly: true - name: apiaudit mountPath: /var/log/kube-apiserver readOnly: true - name: kgaudit mountPath: /var/log/kubegate readOnly: true volumes: - name: config configMap: name: alloy-config - name: positions hostPath: path: /var/lib/alloy type: DirectoryOrCreate - name: audit hostPath: path: /var/log/audit - name: apiaudit hostPath: path: /var/log/kube-apiserver - name: kgaudit hostPath: path: /var/log/kubegate ``` > [!NOTE] > The pod runs as root and privileged because the `auditd` log is owned by root with mode 0600. A non-root reader cannot open it. The host mounts are read-only, so the shipper can tail the logs but not change them. On worker nodes, `/var/log/kube-apiserver` does not exist and `/var/log/kubegate` exists but stays empty. Either way the mounts carry nothing, Alloy finds no file to tail, and only the `auditd` target ships. That is expected. ## Persist the read positions Alloy tracks how far it has read each file under `--storage.path`. The manifest above persists that path with a hostPath at `/var/lib/alloy`, so after a pod restart Alloy resumes from the last read position. (The platform's probe uses an `emptyDir` instead, which is fine for a one-shot test but wrong for production: every restart would re-ship each file from the beginning, duplicating entries in your store.) A small per-node PVC (PersistentVolumeClaim) works too. What matters is that the positions survive pod restarts. They do not need to survive node replacement: a replaced node starts with fresh logs anyway. ## Send to a real destination The config above pushes to any Loki-compatible endpoint (Grafana Cloud, a self-hosted Loki, or a SIEM that accepts the Loki push API). To install Loki in the cluster, see [Collect logs](/docs/hetzner/apalla/observability/logs/collect-container-and-journald-logs). For other destinations, keep the `loki.source.file` block and replace the output: - **Elasticsearch or OpenSearch**: convert the stream with `otelcol.receiver.loki`, then export with `otelcol.exporter.elasticsearch` (or `otelcol.exporter.otlp` to a collector in front of OpenSearch). - **Object store (S3-compatible)**: same conversion, then `otelcol.exporter.awss3`. An object store is the natural place for the write-once copy. - **Any SIEM that accepts OTLP (OpenTelemetry Protocol)**: `otelcol.receiver.loki` plus `otelcol.exporter.otlp`. Alloy is not the only option. Vector and Fluent Bit read the same three files from the same host mounts, with `file`/`tail` sources and sinks for Loki, Elasticsearch, S3, and most SIEMs. The DaemonSet setup stays the same: root, privileged, `tolerations: operator: Exists`, three read-only host mounts, persisted positions. Only the config format changes. ## Make the destination tamper-proof Shipping the logs is only half of the control. Local logs are tamper-evident, not tamper-proof: an attacker with root access can alter files that are still on the node. Only evidence that has already left the node survives an incident, and only if the destination refuses deletes: - **Object-lock or WORM in compliance mode.** The strictest setting: even an administrator cannot delete an object before the retention period expires. - **Write-only credentials.** The shipper's credentials can append, never delete or overwrite. A compromised node must not be able to erase what it already sent. - **A separate trust domain.** Keep the store in a different account or project from the cluster. Compromising the cluster must not give an attacker control over its own audit trail. - **Retention to your policy.** Match your compliance framework. PCI-DSS (Payment Card Industry Data Security Standard) expects roughly one year, with the most recent three months immediately searchable. [Retrieve audit logs](/docs/hetzner/apalla/security/retrieve-audit-logs) explains the reasoning behind each point and lists the local retention caps. Use that page to read logs manually, and this page for the pipeline. ## Verify the pipeline Test end to end, not just that pods are running: 1. Create and delete a test Secret: ```console $ kubectl create secret generic audit-pipeline-test --from-literal=probe=1 $ kubectl delete secret audit-pipeline-test ``` 2. Find both events in the store. In Loki: ```text {job="apiserver_audit"} |= "audit-pipeline-test" ``` You should see a `create` and a `delete` event at metadata level (Secret values are never written to the audit log). If nothing appears, check that the DaemonSet has a pod on every control-plane node: `kubectl -n audit-shipping get pods -o wide`. 3. Confirm the `auditd` and `kubegate_audit` job labels carry data too. KubeGate audits every decision it makes, allow and deny (`audit.level: all`), so on a running control plane the `kubegate_audit` stream always has content. An empty stream there means the pipeline is not reading `/var/log/kubegate/audit.log`, not that the cluster has been quiet. 4. Try to delete an object from the store before its retention expires. The delete must fail. If it succeeds, the store is not yet WORM and the compliance control is not in place.