Skip to main content

Runtime threat detection

Inspect 1.36

The sealed OS and admission guardrails prevent a lot, but prevention is never complete. Runtime threat detection watches what actually happens inside running containers and alerts on the behavior that gets past the preventive controls: a shell spawned in a container, an unexpected outbound connection, a privilege change. Falco is the common tool; deploy it, tune its rules, and wire it into a detect-and-respond loop.

Deploy Falco as a DaemonSet#

Falco runs a sensor on every node. On the sealed OS there is one thing to get right: use the eBPF driver, not a kernel module. The kernel is built with MODULE_SIG_FORCE, so an unsigned kernel module cannot load, and Falco's kernel-module driver would be rejected. Falco's eBPF probe needs no module and is the supported path here. The Helm chart selects it:

		$ helm repo add falcosecurity https://falcosecurity.github.io/charts
$ helm install falco falcosecurity/falco \
  --namespace falco --create-namespace \
  --set driver.kind=ebpf
	

Which rules matter most#

Falco ships a default ruleset; the high-value rules for a Kubernetes cluster are the ones that flag post-compromise behavior:

  • A shell in a container (bash, sh) where the workload should never spawn one, the classic sign of an attacker exploring after a breach.
  • Unexpected outbound connections, a pod dialing a destination it never has before (pair this with for the network view).
  • Privilege changes, a process trying to escalate, write to a sensitive path, or load code.

Tune out the noise for your workloads (a legitimate debug container, an expected admin action) so a real alert is not buried, then keep the rules under version control.

Route alerts to your SIEM#

A detection is only useful if someone sees it. Send Falco's alerts to the same SIEM or store you , so runtime alerts sit next to the audit trail. Together, Falco (what a process did), the (who touched the API and what the OS saw), and Hubble (what talked to what) give you the three views an investigation needs. Falco is a runtime layer you add on top of the built-in , and its alerts land in storage you own, the same as every other signal.

The detect-and-respond runbook#

When Falco fires on a workload you believe is compromised:

Isolate the pod #

Apply a that denies the pod all ingress and egress, so it cannot spread or exfiltrate while you look. Do not delete it yet.

Capture evidence #

Preserve the Falco alert, the pod's logs, and the relevant audit and Hubble records off-node, before the pod is gone. A replaced node takes its local evidence with it.

Redeploy clean #

Once you have what you need, delete the pod and let its controller recreate it from the clean image. If the node itself is suspect, treat it as a : reprovision, never repair in place.

Close the gap #

Fix what let it in (the vulnerable image, the missing network policy, the over-broad RBAC) so the same path does not reopen.

Runtime detection complements the sealed OS and admission; it does not replace them. Keep the preventive controls tight and use Falco to catch what gets through, then respond fast enough that a detection does not become an incident.