Hubble is Cilium's network flow visibility, on in every Syself Autopilot cluster. It shows you the connections in your cluster and which packets get blocked. Each recorded connection is a flow, and each blocked packet is a drop. Hubble records every flow and every drop, with the reason for each drop. Scrape its metrics for the trend, and use the CLI to watch flows live when you are investigating a specific problem.

## What Hubble collects

The cluster enables these Hubble metric groups. Only `httpV2` carries source and destination namespace and workload labels; the rest carry only their own labels like `reason` and `protocol`. DNS query names and HTTP paths are not included.

| Group    | Shows                                                      |
| -------- | ---------------------------------------------------------- |
| `drop`   | dropped packets and the reason (the main detection signal) |
| `flow`   | connection-level flows between sources and destinations    |
| `tcp`    | TCP-level counters                                         |
| `dns`    | DNS request counts (no per-domain label)                   |
| `icmp`   | ICMP flows                                                 |
| `httpV2` | HTTP flows with namespace, workload, and direction labels  |

## Scrape the metrics

The Hubble metrics server runs inside the Cilium agent on the host network, and because nodes carry public IPs it binds to `127.0.0.1:9965`. Binding to loopback keeps the metrics on the node and off the network, and the host firewall keeps the endpoint private. The [System Alloy](/docs/hetzner/apalla/observability/collection/deploy-the-system-alloy) reaches it there as part of its node-local jobs, so if that agent is running these series are already arriving (`ss -tlnp | grep 9965` from a node shell confirms the port is listening).

## Connect the CLI to the relay

The Hubble CLI talks to Hubble Relay for cluster-wide flows, and on a 1.36 cluster the relay serves its flow API over TLS only. A bare `hubble observe` speaks plaintext and fails with a TLS handshake error before printing anything. Set the CLI up once against the Cilium CA:

<Steps>
<Step>Port-forward the relay</Step>

```console
$ cilium hubble port-forward &
```

<Step>Export the Cilium CA</Step>

The relay's certificate is signed by the Cilium CA (Cilium's internal certificate authority) for the name `*.hubble-relay.cilium.io`.

```console
$ kubectl get secret -n kube-system cilium-ca -o 'jsonpath={.data.ca\.crt}' | base64 -d > hubble-ca.crt
```

<Step>Point the CLI at it over TLS</Step>

```console
$ hubble config set tls true
$ hubble config set tls-ca-cert-files "$PWD/hubble-ca.crt"
$ hubble config set tls-server-name "*.hubble-relay.cilium.io"
```

</Steps>

After that, every `hubble observe` works. To avoid storing CLI config, pass `--tls --tls-ca-cert-files ./hubble-ca.crt --tls-server-name "*.hubble-relay.cilium.io"` on each command instead.

## Watch flows live

```console
$ hubble observe --verdict DROPPED --namespace team-a
$ hubble observe --from-label app=frontend --verdict DROPPED -f
```

Each `DROPPED` line shows the source, destination, port, and drop reason, usually enough to tell a policy misconfiguration from a genuine intrusion attempt. To fix a policy drop, see [Segment with network policies](/docs/hetzner/apalla/security/segment-with-network-policies).

Next, turn the drops into alerts and a queryable log: [Alert on dropped packets](/docs/hetzner/apalla/observability/network-flows/alert-on-drops).