The System Alloy is the one agent that runs on every node, and it collects everything the node itself knows: the control-plane metrics on loopback, the kubelet and cadvisor series, the Cilium and cluster component endpoints, and the container, systemd, and audit logs. One Helm release covers all of it. Application metrics are deliberately absent, for the reason in [Using Alloy for observability](/docs/hetzner/apalla/observability/collection/using-alloy-for-observability).

**Prerequisites:** a metrics backend that accepts remote-write and a Loki that accepts pushes, both reachable from inside the cluster ([Set up Prometheus](/docs/hetzner/apalla/observability/metrics/set-up-prometheus) and [Set up Loki](/docs/hetzner/apalla/observability/logs/set-up-loki)), plus Helm and kubectl access.

> [!WARNING]
> This DaemonSet is a privileged collector. On the host network it reaches every loopback listener on the node, including the unauthenticated ones (etcd on `2381`, KubeGate on `8080`, `syself-agent` on `20257`), and it reads root-owned audit files. Keep it in a namespace only your platform team can write to, run no application workloads there, and review its config like a firewall rule.

## What the config has to solve

Four things make this config specific to Syself Autopilot rather than a stock Kubernetes install.

1. **Loopback binding.** Most platform metrics endpoints bind to `127.0.0.1`, so the agent shares the node's network namespace with `hostNetwork: true` and dials loopback directly. `dnsPolicy: ClusterFirstWithHostNet` keeps cluster DNS resolvable from there, which the backend Service names need.

2. **A per-node identity.** Every agent dials the same `127.0.0.1:<port>`, so without intervention every node reports the same `instance` label. A remote-write receiver then sees many agents writing one series, interleaves the samples, and rejects the overlap as out of order (`err-mimir-sample-out-of-order` on Grafana Mimir). The config stamps the node name as `instance` and `node` on every target, injected through the Downward API.

3. **Node-role gating.** etcd, kube-controller-manager, kube-scheduler, KubeGate, and syself-tunnel-server run only on control planes; `syself-proxy` only on workers; the CSI node plugin only on **cloud** workers. Give every node the same target list and the nodes missing a component report `connection refused` forever, behind a `TargetDown` alert that never clears. The config reads the local node's labels and keeps each job only where that component actually runs. Control-plane nodes carry `node-role.kubernetes.io/control-plane` with an **empty value**, so the gate checks that the label is there rather than what it says; workers carry `node.kubernetes.io/worker=true`, and cloud nodes carry `autopilot.syself.com/machine-type=hcloud`. On a cluster with bare-metal workers, the CSI node job needs both labels: a worker-only gate keeps the bare-metal nodes, where that port is never listening.

4. **Self-signed certificates on loopback.** kube-controller-manager, kube-scheduler, and the kubelet serve HTTPS with a certificate the agent cannot verify from loopback: self-signed, or with the node IP as its only SAN. Those jobs set `insecure_skip_verify`. The connection never leaves the node, so there is nothing on the wire to intercept. The API server is the exception: it listens on loopback too and its serving certificate names `127.0.0.1`, so that job verifies against the ServiceAccount CA bundle rather than skipping verification.

## Deploy it

<Steps>
<Step>Write the values file</Step>

The chart's own RBAC covers everything this agent needs. `rbac.create: true` grants the discovery permissions plus `nodes/metrics` and `get` on the `/metrics` non-resource URL, which is exactly what the token-authenticated control-plane and kubelet jobs authorize against, so no extra ClusterRole is required.

<Collapsible title="Why those two permissions differ">

kube-controller-manager, kube-scheduler, and the kubelet all delegate authentication to the API server, so the agent authenticates with its ServiceAccount token. The controller-manager and scheduler check the `nonResourceURLs: ["/metrics"]` rule. The kubelet maps its `/metrics` path to the resource `nodes/metrics` and checks that instead, which also covers `/metrics/cadvisor`, so the cadvisor job needs nothing extra. etcd and KubeGate serve plain HTTP with no authentication and check neither.

</Collapsible>

`mounts.varlog: true` mounts the host's whole `/var/log`, which is where the container logs, the journal, and all three audit streams live, so one flag replaces a list of hostPath volumes. All three audit streams are mode `0600`, owned by root except the KubeGate log, which belongs to the unprivileged user KubeGate runs as. Either way no unprivileged reader can open them, so the container runs as root.

<Collapsible title="The full values file">

```yaml title="system-alloy-values.yaml"
controller:
  type: daemonset
  # Reach the node's loopback endpoints, and keep cluster DNS working from there.
  hostNetwork: true
  dnsPolicy: ClusterFirstWithHostNet
  # Tolerate every taint, so the agent lands on all nodes: control planes,
  # where etcd, the scheduler, the controller-manager, and KubeGate live, and
  # any node your own workloads have tainted. A node without this agent is a
  # node with no telemetry at all.
  tolerations:
    - operator: Exists
  volumes:
    extra:
      # Hubble flow export, if you enabled it. Drop this if you did not.
      - name: hubble
        hostPath: {path: /var/run/cilium/hubble}

alloy:
  # Read the 0600 root-owned audit files.
  securityContext:
    runAsUser: 0
    runAsNonRoot: false
  mounts:
    # /var/log covers container logs, the journal, and the audit streams.
    varlog: true
    extra:
      - name: hubble
        mountPath: /var/run/cilium/hubble
        readOnly: true
  extraEnv:
    - name: NODE_NAME
      valueFrom:
        fieldRef: {fieldPath: spec.nodeName}
  # Buffer samples on disk so a backend restart costs latency, not data.
  storagePath: /var/lib/alloy
  configMap:
    content: |
      // ===================== Destinations =====================

      prometheus.remote_write "central" {
        endpoint {
          url = "http://prometheus-operated.monitoring.svc.cluster.local:9090/api/v1/write"
        }
      }

      loki.write "central" {
        endpoint {
          url = "http://loki-gateway.monitoring.svc.cluster.local/loki/api/v1/push"
        }
      }

      // ===================== Node identity and role gates =====================

      discovery.kubernetes "nodes" {
        role = "node"
      }

      // Keep only this node's entry, and stamp its name so every node's series
      // stays distinct at the receiver.
      discovery.relabel "local_node" {
        targets = discovery.kubernetes.nodes.targets

        rule {
          source_labels = ["__meta_kubernetes_node_name"]
          regex         = sys.env("NODE_NAME")
          action        = "keep"
        }
        rule {
          source_labels = ["__meta_kubernetes_node_name"]
          target_label  = "instance"
        }
        rule {
          source_labels = ["__meta_kubernetes_node_name"]
          target_label  = "node"
        }
      }

      // Yields one target on a control plane and nothing elsewhere. The role
      // label's value is empty here, so test for its presence.
      discovery.relabel "control_plane" {
        targets = discovery.relabel.local_node.output

        rule {
          source_labels = ["__meta_kubernetes_node_labelpresent_node_role_kubernetes_io_control_plane"]
          regex         = "true"
          action        = "keep"
        }
      }

      // Yields one target on a worker and nothing elsewhere.
      discovery.relabel "worker" {
        targets = discovery.relabel.local_node.output

        rule {
          source_labels = ["__meta_kubernetes_node_label_node_kubernetes_io_worker"]
          regex         = "true"
          action        = "keep"
        }
      }

      // The CSI node plugin runs on cloud workers only, so its gate needs the
      // machine type as well. A worker-only gate keeps bare-metal nodes too,
      // where port 9189 is never listening.
      discovery.relabel "cloud_worker" {
        targets = discovery.relabel.worker.output

        rule {
          source_labels = ["__meta_kubernetes_node_label_autopilot_syself_com_machine_type"]
          regex         = "hcloud"
          action        = "keep"
        }
      }

      // ===================== Every node, plain HTTP =====================

      prometheus.scrape "node_local" {
        targets = [
          {__address__ = "127.0.0.1:9962",  job = "cilium-agent",        instance = sys.env("NODE_NAME")},
          {__address__ = "127.0.0.1:9965",  job = "hubble",              instance = sys.env("NODE_NAME")},
          {__address__ = "127.0.0.1:20257", job = "syself-agent",        instance = sys.env("NODE_NAME")},
          {__address__ = "127.0.0.1:8182",  job = "syself-tunnel-agent", instance = sys.env("NODE_NAME")},
        ]
        forward_to = [prometheus.remote_write.central.receiver]
      }

      // containerd serves its metrics on /v1/metrics, not the usual /metrics,
      // so it needs its own block rather than a line in the list above.
      prometheus.scrape "containerd" {
        targets = [
          {__address__ = "127.0.0.1:1338", job = "containerd", instance = sys.env("NODE_NAME")},
        ]
        metrics_path = "/v1/metrics"
        forward_to   = [prometheus.remote_write.central.receiver]
      }

      // The agent's own metrics. Without this nothing watches the collector,
      // and a collector that stops looks exactly like a quiet cluster.
      prometheus.scrape "alloy_self" {
        targets = [
          {__address__ = "127.0.0.1:12345", job = "alloy-system", instance = sys.env("NODE_NAME")},
        ]
        forward_to = [prometheus.remote_write.central.receiver]
      }

      // node-exporter is your install, not the platform's.
      // Delete this block if you skipped it.
      prometheus.scrape "node_exporter" {
        targets = [
          {__address__ = "127.0.0.1:9100", job = "node-exporter", instance = sys.env("NODE_NAME")},
        ]
        forward_to = [prometheus.remote_write.central.receiver]
      }

      // ===================== Every node, HTTPS with a token =====================

      prometheus.scrape "kubelet" {
        targets = [
          {__address__ = "127.0.0.1:10250", job = "kubelet", instance = sys.env("NODE_NAME")},
        ]
        scheme           = "https"
        bearer_token_file = "/var/run/secrets/kubernetes.io/serviceaccount/token"
        tls_config {
          insecure_skip_verify = true
        }
        forward_to = [prometheus.remote_write.central.receiver]
      }

      // Per-container CPU, memory, and network, from the kubelet's own path.
      prometheus.scrape "cadvisor" {
        targets = [
          {__address__ = "127.0.0.1:10250", job = "cadvisor", instance = sys.env("NODE_NAME")},
        ]
        scheme            = "https"
        metrics_path      = "/metrics/cadvisor"
        bearer_token_file = "/var/run/secrets/kubernetes.io/serviceaccount/token"
        tls_config {
          insecure_skip_verify = true
        }
        forward_to = [prometheus.remote_write.central.receiver]
      }

      // ===================== Control planes only =====================

      discovery.relabel "etcd" {
        targets = discovery.relabel.control_plane.output
        rule {
          target_label = "__address__"
          replacement  = "127.0.0.1:2381"
        }
      }

      prometheus.scrape "etcd" {
        targets    = discovery.relabel.etcd.output
        job_name   = "etcd"
        forward_to = [prometheus.remote_write.central.receiver]
      }

      // The API server listens on loopback as well as the node IP, and its
      // serving certificate names 127.0.0.1, so this is the one control-plane
      // job that verifies TLS properly instead of skipping it.
      discovery.relabel "apiserver" {
        targets = discovery.relabel.control_plane.output
        rule {
          target_label = "__address__"
          replacement  = "127.0.0.1:6443"
        }
      }

      prometheus.scrape "apiserver" {
        targets           = discovery.relabel.apiserver.output
        job_name          = "kube-apiserver"
        scheme            = "https"
        bearer_token_file = "/var/run/secrets/kubernetes.io/serviceaccount/token"
        tls_config {
          ca_file = "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
        }
        forward_to = [prometheus.remote_write.central.receiver]
      }

      discovery.relabel "kubegate" {
        targets = discovery.relabel.control_plane.output
        rule {
          target_label = "__address__"
          replacement  = "127.0.0.1:8080"
        }
      }

      prometheus.scrape "kubegate" {
        targets    = discovery.relabel.kubegate.output
        job_name   = "kubegate"
        forward_to = [prometheus.remote_write.central.receiver]
      }

      discovery.relabel "tunnel_server" {
        targets = discovery.relabel.control_plane.output
        rule {
          target_label = "__address__"
          replacement  = "127.0.0.1:8181"
        }
      }

      prometheus.scrape "tunnel_server" {
        targets    = discovery.relabel.tunnel_server.output
        job_name   = "syself-tunnel-server"
        forward_to = [prometheus.remote_write.central.receiver]
      }

      discovery.relabel "controller_manager" {
        targets = discovery.relabel.control_plane.output
        rule {
          target_label = "__address__"
          replacement  = "127.0.0.1:10257"
        }
      }

      prometheus.scrape "controller_manager" {
        targets           = discovery.relabel.controller_manager.output
        job_name          = "kube-controller-manager"
        scheme            = "https"
        bearer_token_file = "/var/run/secrets/kubernetes.io/serviceaccount/token"
        tls_config {
          insecure_skip_verify = true
        }
        forward_to = [prometheus.remote_write.central.receiver]
      }

      discovery.relabel "scheduler" {
        targets = discovery.relabel.control_plane.output
        rule {
          target_label = "__address__"
          replacement  = "127.0.0.1:10259"
        }
      }

      prometheus.scrape "scheduler" {
        targets           = discovery.relabel.scheduler.output
        job_name          = "kube-scheduler"
        scheme            = "https"
        bearer_token_file = "/var/run/secrets/kubernetes.io/serviceaccount/token"
        tls_config {
          insecure_skip_verify = true
        }
        forward_to = [prometheus.remote_write.central.receiver]
      }

      // ===================== Workers only =====================

      // syself-proxy runs on every worker; the CSI node plugin only on cloud
      // workers, so the two jobs use different gates.

      discovery.relabel "csi_node" {
        targets = discovery.relabel.cloud_worker.output
        rule {
          target_label = "__address__"
          replacement  = "127.0.0.1:9189"
        }
      }

      prometheus.scrape "csi_node" {
        targets    = discovery.relabel.csi_node.output
        job_name   = "csi-node"
        forward_to = [prometheus.remote_write.central.receiver]
      }

      discovery.relabel "syself_proxy" {
        targets = discovery.relabel.worker.output
        rule {
          target_label = "__address__"
          replacement  = "127.0.0.1:9587"
        }
      }

      prometheus.scrape "syself_proxy" {
        targets    = discovery.relabel.syself_proxy.output
        job_name   = "syself-proxy"
        forward_to = [prometheus.remote_write.central.receiver]
      }

      // ===================== Addons on the pod network =====================

      // The Cilium operator, the CCM, hubble-relay, the CSI controller, and
      // CoreDNS run as ordinary pods. Each agent scrapes only the ones on its
      // own node, so the work spreads without any target being scraped twice.
      discovery.kubernetes "addon_pods" {
        role = "pod"
        namespaces {
          names = ["kube-system"]
        }
      }

      discovery.relabel "addons" {
        targets = discovery.kubernetes.addon_pods.targets

        rule {
          source_labels = ["__meta_kubernetes_pod_node_name"]
          regex         = sys.env("NODE_NAME")
          action        = "keep"
        }
        rule {
          source_labels = ["__meta_kubernetes_pod_node_name"]
          target_label  = "node"
        }
      }

      // Cilium operator binds its metrics to loopback like the agent does.
      discovery.relabel "cilium_operator" {
        targets = discovery.relabel.addons.output

        rule {
          source_labels = ["__meta_kubernetes_pod_label_app_kubernetes_io_name"]
          regex         = "cilium-operator"
          action        = "keep"
        }
        // These pods declare more than one port, and pod discovery emits a
        // target for each. Keep the metrics port so the endpoint is scraped
        // once, not once per port.
        rule {
          source_labels = ["__meta_kubernetes_pod_container_port_name"]
          regex         = "prometheus"
          action        = "keep"
        }
        rule {
          target_label = "__address__"
          replacement  = "127.0.0.1:9963"
        }
        rule {
          source_labels = ["__meta_kubernetes_pod_node_name"]
          target_label  = "instance"
        }
      }

      prometheus.scrape "cilium_operator" {
        targets    = discovery.relabel.cilium_operator.output
        job_name   = "cilium-operator"
        forward_to = [prometheus.remote_write.central.receiver]
      }

      // The rest serve on the pod network, so scrape the pod IP.
      discovery.relabel "ccm" {
        targets = discovery.relabel.addons.output

        rule {
          source_labels = ["__meta_kubernetes_pod_label_app_kubernetes_io_name"]
          regex         = "ccm"
          action        = "keep"
        }
        rule {
          source_labels = ["__meta_kubernetes_pod_container_port_name"]
          regex         = "metrics"
          action        = "keep"
        }
        rule {
          source_labels = ["__address__"]
          regex         = "([^:]+)(?::\\d+)?"
          replacement   = "$1:8233"
          target_label  = "__address__"
        }
      }

      prometheus.scrape "ccm" {
        targets    = discovery.relabel.ccm.output
        job_name   = "ccm"
        forward_to = [prometheus.remote_write.central.receiver]
      }

      discovery.relabel "hubble_relay" {
        targets = discovery.relabel.addons.output

        rule {
          source_labels = ["__meta_kubernetes_pod_label_k8s_app"]
          regex         = "hubble-relay"
          action        = "keep"
        }
        rule {
          source_labels = ["__meta_kubernetes_pod_container_port_name"]
          regex         = "prometheus"
          action        = "keep"
        }
        rule {
          source_labels = ["__address__"]
          regex         = "([^:]+)(?::\\d+)?"
          replacement   = "$1:9966"
          target_label  = "__address__"
        }
      }

      prometheus.scrape "hubble_relay" {
        targets    = discovery.relabel.hubble_relay.output
        job_name   = "hubble-relay"
        forward_to = [prometheus.remote_write.central.receiver]
      }

      discovery.relabel "csi_controller" {
        targets = discovery.relabel.addons.output

        rule {
          source_labels = ["__meta_kubernetes_pod_label_app_kubernetes_io_name", "__meta_kubernetes_pod_label_app_kubernetes_io_component"]
          separator     = ";"
          regex         = "csi;controller"
          action        = "keep"
        }
        rule {
          source_labels = ["__meta_kubernetes_pod_container_port_name"]
          regex         = "metrics"
          action        = "keep"
        }
        rule {
          source_labels = ["__address__"]
          regex         = "([^:]+)(?::\\d+)?"
          replacement   = "$1:9189"
          target_label  = "__address__"
        }
      }

      prometheus.scrape "csi_controller" {
        targets    = discovery.relabel.csi_controller.output
        job_name   = "csi-controller"
        forward_to = [prometheus.remote_write.central.receiver]
      }

      discovery.kubernetes "endpoints" {
        role = "endpoints"
      }

      discovery.relabel "coredns" {
        targets = discovery.kubernetes.endpoints.targets

        rule {
          source_labels = ["__meta_kubernetes_service_name", "__meta_kubernetes_endpoint_port_name"]
          separator     = ";"
          regex         = "kube-dns;metrics"
          action        = "keep"
        }
        rule {
          source_labels = ["__meta_kubernetes_endpoint_node_name"]
          regex         = sys.env("NODE_NAME")
          action        = "keep"
        }
      }

      prometheus.scrape "coredns" {
        targets    = discovery.relabel.coredns.output
        job_name   = "coredns"
        forward_to = [prometheus.remote_write.central.receiver]
      }

      // ===================== Container logs =====================

      discovery.kubernetes "pods" {
        role = "pod"
      }

      discovery.relabel "pod_logs" {
        targets = discovery.kubernetes.pods.targets

        rule {
          source_labels = ["__meta_kubernetes_pod_node_name"]
          regex         = sys.env("NODE_NAME")
          action        = "keep"
        }
        rule {
          source_labels = ["__meta_kubernetes_namespace"]
          target_label  = "namespace"
        }
        rule {
          source_labels = ["__meta_kubernetes_pod_name"]
          target_label  = "pod"
        }
        rule {
          source_labels = ["__meta_kubernetes_pod_container_name"]
          target_label  = "container"
        }
        rule {
          source_labels = ["__meta_kubernetes_pod_node_name"]
          target_label  = "node"
        }
        // /var/log/pods/<namespace>_<pod>_<uid>/<container>/*.log
        rule {
          source_labels = ["__meta_kubernetes_pod_uid", "__meta_kubernetes_pod_container_name"]
          separator     = "/"
          replacement   = "/var/log/pods/*$1/*.log"
          target_label  = "__path__"
        }
      }

      // discovery.relabel produced a glob per container; local.file_match is
      // what expands it into the real files. loki.source.file does not expand
      // globs itself, so skipping this step tails nothing at all.
      local.file_match "pod_logs" {
        path_targets = discovery.relabel.pod_logs.output
      }

      loki.source.file "pod_logs" {
        targets    = local.file_match.pod_logs.targets
        forward_to = [loki.process.pod_logs.receiver]
      }

      loki.process "pod_logs" {
        // Unwrap the CRI envelope so the message, not the wrapper, is the line.
        stage.cri {}
        forward_to = [loki.write.central.receiver]
      }

      // ===================== journald =====================

      discovery.relabel "journal" {
        targets = []

        // Keep the units worth reading. Everything else is volume you pay for
        // and never query.
        rule {
          source_labels = ["__journal__systemd_unit"]
          regex         = "(kubelet|containerd|syself-.*)\\.service"
          action        = "keep"
        }
        rule {
          source_labels = ["__journal__systemd_unit"]
          target_label  = "unit"
        }
      }

      loki.source.journal "host" {
        path          = "/var/log/journal"
        relabel_rules = discovery.relabel.journal.rules
        labels        = {job = "journald", node = sys.env("NODE_NAME")}
        forward_to    = [loki.write.central.receiver]
      }

      // ===================== Audit streams =====================

      // The Linux audit daemon, on every node.
      local.file_match "auditd" {
        path_targets = [{
          __path__ = "/var/log/audit/audit.log",
          job      = "auditd",
          log_type = "audit",
          node     = sys.env("NODE_NAME"),
        }]
      }

      loki.source.file "auditd" {
        targets    = local.file_match.auditd.targets
        forward_to = [loki.write.central.receiver]
      }

      // The Kubernetes API and KubeGate audit trails, control planes only. On a
      // worker these paths are absent or empty, so nothing is tailed and the
      // agent stays quiet.
      local.file_match "api_audit" {
        path_targets = [{
          __path__ = "/var/log/kube-apiserver/kube-apiserver.log",
          job      = "kube-apiserver-audit",
          log_type = "audit",
          node     = sys.env("NODE_NAME"),
        }]
      }

      loki.source.file "api_audit" {
        targets    = local.file_match.api_audit.targets
        forward_to = [loki.process.api_audit.receiver]
      }

      // Lift the fields you query on out of the JSON envelope into structured metadata.
      loki.process "api_audit" {
        stage.json {
          expressions = {verb = "verb", user = "user.username", uri = "requestURI"}
        }
        stage.structured_metadata {
          values = {verb = "verb", user = "user", uri = "uri"}
        }
        forward_to = [loki.write.central.receiver]
      }

      local.file_match "kubegate_audit" {
        path_targets = [{
          __path__ = "/var/log/kubegate/audit.log",
          job      = "kubegate-audit",
          log_type = "audit",
          node     = sys.env("NODE_NAME"),
        }]
      }

      loki.source.file "kubegate_audit" {
        targets    = local.file_match.kubegate_audit.targets
        forward_to = [loki.write.central.receiver]
      }
```

</Collapsible>

<Step>Install the chart</Step>

```console
$ helm repo add grafana https://grafana.github.io/helm-charts
$ helm upgrade -i system-alloy grafana/alloy \
  --namespace monitoring --create-namespace \
  --version <chart-version> \
  --values system-alloy-values.yaml
```

Pin `--version` to a chart release you have tested, so an upgrade does not change collection behavior underneath you.

<Step>Confirm a pod runs on every node</Step>

```console
$ kubectl -n monitoring get pods -l app.kubernetes.io/instance=system-alloy -o wide
```

Expect one pod per node, control planes included. A control-plane node with no pod means the toleration did not apply.

<Step>Check the targets</Step>

Alloy serves a UI that lists every component and its targets, which is the fastest way to see what a single node is actually collecting:

```console
$ kubectl -n monitoring port-forward daemonset/system-alloy 12345:12345
```

Open `http://localhost:12345` and look at the `prometheus.scrape` components. On a control-plane pod the `etcd`, `kubegate`, `controller_manager`, `scheduler`, and `tunnel_server` components each show one target up, and `csi_node` and `syself_proxy` show none. On a worker pod that reverses. A component with no targets on the node where the component does not belong is the gating working, not a fault.

Then confirm the samples arrive at the backend, one series per node rather than one shared series:

```text
count(up{job="etcd"}) by (instance)
count(up{job="cilium-agent"}) by (instance)
```

</Steps>

## Add the Hubble flow log

If you turned on Hubble flow export ([Alert on dropped packets](/docs/hetzner/apalla/observability/network-flows/alert-on-drops)), the verdict log is a file on the node like any other, and the values file above already mounts its directory. Add a source for it:

```terraform
local.file_match "hubble_flows" {
  path_targets = [{
    __path__ = "/var/run/cilium/hubble/events.log",
    job      = "hubble-flows",
    node     = sys.env("NODE_NAME"),
  }]
}

loki.source.file "hubble_flows" {
  targets    = local.file_match.hubble_flows.targets
  forward_to = [loki.write.central.receiver]
}
```

## Keep the log volume down

The journald rule above keeps `kubelet`, `containerd`, and the `syself-*` units and drops the rest, which is most of the volume for none of the value. Container logs are the other large contributor and vary entirely with your workloads, so drop or sample the high-volume namespaces rather than paying to store what you will not query. When Loki is slow or an incident floods logs, Alloy's disk buffer at `storagePath` absorbs the burst; size that volume so a spike degrades into delay instead of taking the node's memory with it. [Log retention and sizing](/docs/hetzner/apalla/observability/logs/log-retention-and-sizing) has the per-node estimates to size against.

With the node signals flowing, add [the Application Alloy](/docs/hetzner/apalla/observability/collection/deploy-the-application-alloy) for your workloads' metrics and traces. For what the series you just started collecting actually tell you, see [Control-plane metrics](/docs/hetzner/apalla/observability/metrics/control-plane-metrics) and [Data-plane and cluster component metrics](/docs/hetzner/apalla/observability/metrics/data-plane-and-addon-metrics). If a target will not come up, [Troubleshoot the collectors](/docs/hetzner/apalla/observability/collection/troubleshoot-the-collectors) covers the failures worth knowing by name.