Alert on dropped packets

A rise in dropped packets is a signal worth paging on: a workload is being blocked, or something is probing it. Alert on hubble_drop_total, and keep a queryable log of the drops so you can go from "drops are up" to "which pod, to where, blocked by what." This assumes you scrape the Hubble metrics from See flows with Hubble .

Alert on the drop rate#

hubble_drop_total counts dropped packets, and its reason and protocol labels tell you what kind. Alert on a nonzero rate, broken out by reason:

		apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: hubble-drops
  namespace: monitoring
  labels:
    prometheus: main
spec:
  groups:
    - name: hubble-drops
      rules:
        - alert: HubbleDropsRising
          expr: sum(rate(hubble_drop_total[5m])) by (reason, protocol) > 0
          for: 10m
          labels: {severity: warning}
          annotations:
            summary: "packets dropped ({{ $labels.reason }}, {{ $labels.protocol }})"
            description: "Cilium is dropping traffic. Check whether a policy is blocking a legitimate connection, or something is probing the cluster."
	

Policy misconfig or intrusion#

A drop has two innocent-to-serious explanations, and the reason separates them:

• A policy misconfig: a legitimate connection you have not allowed yet. The reason points at a policy denial, and the source is one of your own workloads. Fix it in network policy .
• A probe or attack: drops from an unexpected source, or a workload suddenly reaching a new external destination. Alert on unexpected outbound flows the same way, because a pod connecting somewhere it never has before is worth investigating.

Keep a queryable flow log#

Hubble keeps flows in memory, so the history is gone when the agent restarts or the node is replaced. To keep a durable, queryable record of verdicts (the equivalent of a managed provider's network-policy log), turn on Hubble flow export and ship the file like any other log.

There is no topology variable for this, but you can configure it via a CiliumNodeConfig object. This custom resource overrides the agent configuration natively without modifying the platform-managed cilium-config ConfigMap, which means your settings will survive platform upgrades.

		apiVersion: cilium.io/v2
kind: CiliumNodeConfig
metadata:
  namespace: kube-system
  name: hubble-flow-export
spec:
  nodeSelector:
    matchLabels: {} # Apply to all nodes
  defaults:
    hubble-export-file-max-size-mb: "50"
    hubble-export-file-max-backups: "5"
    hubble-export-file-path: "/var/run/cilium/hubble/events.log"
    # Keep the volume down: export these verdicts, not every packet. The key is
    # "allowlist", one word; "allow-list" is silently ignored and you get
    # everything. The value is a bare JSON object, never a JSON array.
    hubble-export-allowlist: '{"verdict":["DROPPED","ERROR"]}'
	

Apply the configuration and restart the Cilium agents to load it:

		$ kubectl apply -f hubble-flow-export.yaml
$ kubectl -n kube-system rollout restart daemonset/cilium
	

Then tail the file like any other log source. The System Alloy already mounts /var/run/cilium/hubble/ and has a source block for events.log ready to enable. Each line is a JSON flow with source and destination identity, port, and verdict, so in Grafana you can query "which pod was denied reaching which service, and by which policy" right next to the drop metrics. Widen the allow list to include FORWARDED for a full connection log rather than only drops.