A rise in dropped packets is a signal worth paging on: a workload is being blocked, or something is probing it. Alert on `hubble_drop_total`, and keep a queryable log of the drops so you can go from "drops are up" to "which pod, to where, blocked by what." This assumes you scrape the Hubble metrics from [See flows with Hubble](/docs/hetzner/apalla/observability/network-flows/see-flows-with-hubble).

## Alert on the drop rate

`hubble_drop_total` counts dropped packets, and its `reason` and `protocol` labels tell you what kind. Alert on a nonzero rate, broken out by reason:

```yaml title="hubble-drop-rules.yaml"
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: hubble-drops
  namespace: monitoring
  labels:
    prometheus: main
spec:
  groups:
    - name: hubble-drops
      rules:
        - alert: HubbleDropsRising
          expr: sum(rate(hubble_drop_total[5m])) by (reason, protocol) > 0
          for: 10m
          labels: {severity: warning}
          annotations:
            summary: "packets dropped ({{ $labels.reason }}, {{ $labels.protocol }})"
            description: "Cilium is dropping traffic. Check whether a policy is blocking a legitimate connection, or something is probing the cluster."
```

## Policy misconfig or intrusion

A drop has two innocent-to-serious explanations, and the `reason` separates them:

- **A policy misconfig:** a legitimate connection you have not allowed yet. The reason points at a policy denial, and the source is one of your own workloads. Fix it in [network policy](/docs/hetzner/apalla/security/segment-with-network-policies).
- **A probe or attack:** drops from an unexpected source, or a workload suddenly reaching a new external destination. Alert on unexpected outbound flows the same way, because a pod connecting somewhere it never has before is worth investigating.

## Keep a queryable flow log

Hubble keeps flows in memory, so the history is gone when the agent restarts or the node is replaced. To keep a durable, queryable record of verdicts (the equivalent of a managed provider's network-policy log), turn on Hubble flow export and ship the file like any other log.

There is no topology variable for this, but you can configure it via a `CiliumNodeConfig` object. This custom resource overrides the agent configuration natively without modifying the platform-managed `cilium-config` ConfigMap, which means your settings will survive platform upgrades.

```yaml title="hubble-flow-export.yaml"
apiVersion: cilium.io/v2
kind: CiliumNodeConfig
metadata:
  namespace: kube-system
  name: hubble-flow-export
spec:
  nodeSelector:
    matchLabels: {} # Apply to all nodes
  defaults:
    hubble-export-file-max-size-mb: "50"
    hubble-export-file-max-backups: "5"
    hubble-export-file-path: "/var/run/cilium/hubble/events.log"
    # Keep the volume down: export these verdicts, not every packet. The key is
    # "allowlist", one word; "allow-list" is silently ignored and you get
    # everything. The value is a bare JSON object, never a JSON array.
    hubble-export-allowlist: '{"verdict":["DROPPED","ERROR"]}'
```

Apply the configuration and restart the Cilium agents to load it:

```console
$ kubectl apply -f hubble-flow-export.yaml
$ kubectl -n kube-system rollout restart daemonset/cilium
```

Then tail the file like any other log source. The [System Alloy](/docs/hetzner/apalla/observability/collection/deploy-the-system-alloy) already mounts `/var/run/cilium/hubble/` and has a source block for `events.log` ready to enable. Each line is a JSON flow with source and destination identity, port, and verdict, so in Grafana you can query "which pod was denied reaching which service, and by which policy" right next to the drop metrics. Widen the allow list to include `FORWARDED` for a full connection log rather than only drops.

> [!NOTE]
> Flow export produces high volume. A busy node produces a large stream even filtered to drops, and widening it to `FORWARDED` multiplies that by every connection in the cluster. Size the retention on this stream separately from your other logs, and treat it as a diagnostic you turn up during an investigation rather than something to keep for a year.