Runtime threat detection
The sealed OS and admission guardrails prevent a lot, but prevention is never complete. Runtime threat detection watches what actually happens inside running containers and alerts on the behavior that gets past the preventive controls: a shell spawned in a container, an unexpected outbound connection, a privilege change. Falco is the common tool; deploy it, tune its rules, and wire it into a detect-and-respond loop.
Deploy Falco as a DaemonSet#
Falco runs a sensor on every node. On the sealed OS there is one thing to get right: use the eBPF driver, not a kernel module. The kernel is built with MODULE_SIG_FORCE, so an unsigned kernel module cannot load, and Falco's kernel-module driver would be rejected. Falco's eBPF probe needs no module and is the supported path here. The Helm chart selects it:
$ helm repo add falcosecurity https://falcosecurity.github.io/charts
$ helm install falco falcosecurity/falco \
--namespace falco --create-namespace \
--set driver.kind=ebpf
Which rules matter most#
Falco ships a default ruleset; the high-value rules for a Kubernetes cluster are the ones that flag post-compromise behavior:
- A shell in a container (
bash,sh) where the workload should never spawn one, the classic sign of an attacker exploring after a breach. - Unexpected outbound connections, a pod dialing a destination it never has before (pair this with Hubble flow export for the network view).
- Privilege changes, a process trying to escalate, write to a sensitive path, or load code.
Tune out the noise for your workloads (a legitimate debug container, an expected admin action) so a real alert is not buried, then keep the rules under version control.
Route alerts to your SIEM#
A detection is only useful if someone sees it. Send Falco's alerts to the same SIEM or store you ship audit logs to , so runtime alerts sit next to the audit trail. Together, Falco (what a process did), the audit streams (who touched the API and what the OS saw), and Hubble (what talked to what) give you the three views an investigation needs. Falco is a runtime layer you add on top of the built-in observability model , and its alerts land in storage you own, the same as every other signal.
The detect-and-respond runbook#
When Falco fires on a workload you believe is compromised:
Isolate the pod #
Apply a NetworkPolicy that denies the pod all ingress and egress, so it cannot spread or exfiltrate while you look. Do not delete it yet.
Capture evidence #
Preserve the Falco alert, the pod's logs, and the relevant audit and Hubble records off-node, before the pod is gone. A replaced node takes its local evidence with it.
Redeploy clean #
Once you have what you need, delete the pod and let its controller recreate it from the clean image. If the node itself is suspect, treat it as a tampered node : reprovision, never repair in place.
Close the gap #
Fix what let it in (the vulnerable image, the missing network policy, the over-broad RBAC) so the same path does not reopen.
Runtime detection complements the sealed OS and admission; it does not replace them. Keep the preventive controls tight and use Falco to catch what gets through, then respond fast enough that a detection does not become an incident.
Ship audit logs to a SIEM
A copy-paste Grafana Alloy DaemonSet that tails the host auditd, Kubernetes API, and KubeGate audit logs on every Syself Linux node and forwards them to Loki, a SIEM, or write-once object storage.
Cluster variables
Complete reference for every cluster variable you can set under spec.topology.variables on a Hetzner cluster in Syself Autopilot, including defaults and what each one does.