See flows with Hubble

Hubble is Cilium's network flow visibility, on in every Syself Autopilot cluster. It shows you the connections in your cluster and which packets get blocked. Each recorded connection is a flow, and each blocked packet is a drop. Hubble records every flow and every drop, with the reason for each drop. Scrape its metrics for the trend, and use the CLI to watch flows live when you are investigating a specific problem.

What Hubble collects#

The cluster enables these Hubble metric groups. Only httpV2 carries source and destination namespace and workload labels; the rest carry only their own labels like reason and protocol. DNS query names and HTTP paths are not included.

Scrape the metrics#

The Hubble metrics server runs inside the Cilium agent on the host network, and because nodes carry public IPs it binds to 127.0.0.1:9965. Binding to loopback keeps the metrics on the node and off the network, and the host firewall keeps the endpoint private. The System Alloy reaches it there as part of its node-local jobs, so if that agent is running these series are already arriving (ss -tlnp | grep 9965 from a node shell confirms the port is listening).

Connect the CLI to the relay#

The Hubble CLI talks to Hubble Relay for cluster-wide flows, and on a 1.36 cluster the relay serves its flow API over TLS only. A bare hubble observe speaks plaintext and fails with a TLS handshake error before printing anything. Set the CLI up once against the Cilium CA:

		$ cilium hubble port-forward &
	

		$ kubectl get secret -n kube-system cilium-ca -o 'jsonpath={.data.ca\.crt}' | base64 -d > hubble-ca.crt
	

		$ hubble config set tls true
$ hubble config set tls-ca-cert-files "$PWD/hubble-ca.crt"
$ hubble config set tls-server-name "*.hubble-relay.cilium.io"
	

After that, every hubble observe works. To avoid storing CLI config, pass --tls --tls-ca-cert-files ./hubble-ca.crt --tls-server-name "*.hubble-relay.cilium.io" on each command instead.

Watch flows live#

		$ hubble observe --verdict DROPPED --namespace team-a
$ hubble observe --from-label app=frontend --verdict DROPPED -f
	

Each DROPPED line shows the source, destination, port, and drop reason, usually enough to tell a policy misconfiguration from a genuine intrusion attempt. To fix a policy drop, see Segment with network policies .

Next, turn the drops into alerts and a queryable log: Alert on dropped packets .