See flows with Hubble

Hubble runs on every Syself Autopilot cluster as part of Cilium. It records the connections in your cluster, including every packet Cilium drops and why. The metrics need no setup other than running the System Alloy. Hubble CLI needs a one-time TLS step before it will connect.

What Hubble collects

The cluster enables these Hubble metric groups. Only httpV2 carries source and destination namespace and workload labels; the rest carry only their own labels like reason and protocol. DNS query names and HTTP paths are not included.

Scrape the metrics

The Hubble metrics server runs inside the Cilium agent on the host network, and because nodes carry public IPs it binds to 127.0.0.1:9965. Binding to loopback keeps the metrics on the node and off the network. The System Alloy reaches it there as part of its node-local jobs, so if that agent is running, these series are already arriving. From a node shell, ss -tlnp | grep 9965 confirms the port is listening.

Connect the CLI to the relay

The Hubble CLI talks to Hubble Relay for cluster-wide flows, and on a 1.36 cluster the relay serves its flow API over TLS only. With the port-forward, a bare hubble observe still speaks plaintext and fails with error reading server preface: EOF before printing anything. Set the CLI up once against the Cilium CA:

		$ cilium hubble port-forward &
	

		$ kubectl get secret -n kube-system cilium-ca -o 'jsonpath={.data.ca\.crt}' | base64 -d > hubble-ca.crt
	

		$ hubble config set tls true
$ hubble config set tls-ca-cert-files "$PWD/hubble-ca.crt"
$ hubble config set tls-server-name "*.hubble-relay.cilium.io"
	

After that, every hubble observe works. To avoid storing CLI config, pass --tls --tls-ca-cert-files ./hubble-ca.crt --tls-server-name "*.hubble-relay.cilium.io" on each command instead.

Watch flows live

		$ hubble observe --verdict DROPPED --namespace team-a
$ hubble observe --from-label app=frontend --verdict DROPPED -f
	

Each DROPPED line shows the source, destination, port, and drop reason, usually enough to tell a policy misconfiguration from a genuine intrusion attempt. To fix a policy drop, see Segment with network policies .

Next, turn the drops into alerts and a queryable log: Alert on dropped packets .