Put every scrape job in one node-level agent and a single application can blind the whole node. An application that labels a metric with something unique, a request ID for example, turns one series into hundreds of thousands, and the agent runs out of memory. You lose the kubelet, etcd, the health daemon, and log shipping along with the application's own dashboard. The telemetry you need to debug the incident is the first thing the incident takes.

So we split collection in two. A DaemonSet on the host network collects infrastructure telemetry, which is the only way to reach the node's loopback endpoints. A separate clustered agent collects application telemetry and scales out on its own. When an application buries its collector, the node's view of itself keeps running.

```mermaid
flowchart TD
  subgraph infra["Infrastructure telemetry"]
    node["Worker and control-plane nodes<br/>loopback metrics endpoints"]
    logs["Container, journald,<br/>and audit logs"]
    ds["System Alloy<br/>host-network DaemonSet"]
    node -->|"scrape 127.0.0.1"| ds
    logs -->|"tail /var/log"| ds
  end
  subgraph apps["Application telemetry"]
    app["Application pods<br/>Prometheus endpoints"]
    inst["Instrumented services<br/>OTLP exporters"]
    sts["Application Alloy<br/>clustered StatefulSet"]
    app -->|"scrape ServiceMonitor targets"| sts
    inst -->|"push OTLP"| sts
  end
  ds -->|"infrastructure metrics and logs"| backend["Your backend<br/>Prometheus or Mimir, Loki, Tempo"]
  sts -->|"application metrics and traces"| backend
```

Both agents write to the same backend, so the split stops at the collection tier. Your dashboards query everything as one dataset.

## What each agent owns

|              | System Alloy                                                          | Application Alloy                          |
| ------------ | --------------------------------------------------------------------- | ------------------------------------------ |
| Shape        | DaemonSet, one pod per node, host network                             | StatefulSet, a few replicas, pod network   |
| Metrics      | Control-plane, kubelet, node, Cilium, and cluster component endpoints | Your workloads' `/metrics` endpoints       |
| Logs         | Container, journald, and the audit streams                            | none                                       |
| Traces       | none                                                                  | OTLP over gRPC and HTTP, plus span metrics |
| Scales with  | Node count, and nothing else                                          | Application count and trace volume         |
| Blast radius | One node's infrastructure view                                        | Application telemetry only                 |

One rule keeps that separation intact: **never add an application scrape job to the DaemonSet.** A platform component emits a predictable number of series. An application emits as many as its labels allow, and that number is out of your hands.

## Host network access

Most of the platform's metrics endpoints bind to `127.0.0.1`, and several of them serve plain HTTP with no authentication. Loopback is what keeps them safe, because nothing off the node can reach them at all. That also puts them out of reach of an ordinary scraper: inside a pod, `127.0.0.1` is the pod's own loopback, not the node's, so a `ServiceMonitor` or a normal Prometheus pod never sees them. A pod sharing the node's network namespace does. [The observability model](/docs/hetzner/apalla/concepts/operations/observability-model) has the background, and [Reference: metrics per component](/docs/hetzner/apalla/observability/reference/metrics-per-component) lists which endpoint sits where.

That one constraint shapes the System Alloy's pod spec. Each setting answers a part of it:

- **`hostNetwork: true`**: puts the agent in the node's network namespace, so its `127.0.0.1` is the node's.
- **`dnsPolicy: ClusterFirstWithHostNet`**: keeps cluster DNS working from the host network, so the agent can still resolve the backend's Service name.
- **Read-only host mounts**: `/var/log/pods`, `/var/log/journal`, and the audit paths, so one agent carries logs as well as metrics and you do not run a second one per signal.
- **Tolerating all taints**: so the agent lands on every node, including tainted control planes (etcd, kube-scheduler, kube-controller-manager, KubeGate) and any node your own workloads taint. A skipped node is a node with no telemetry.
- **Node-role discovery and local relabeling**: decides which jobs run where. Control-plane components exist only on control planes, `syself-proxy` only on workers, and the CSI node plugin only on cloud VM workers, so a static target list would produce permanent `connection refused` and false `TargetDown` alerts.
- **Write-ahead log on disk**: buffers samples when the backend is unreachable, so a backend restart costs you latency instead of data.

> [!WARNING]
> The System Alloy is a highly privileged collector. Operating on the host network, it can access every loopback listener on the node, including unauthenticated services (like etcd on `2381`, KubeGate on `8080`, and `syself-agent` on `20257`). Additionally, its ServiceAccount holds cluster-wide `get` permissions for `/metrics` and `nodes/metrics`. You must isolate it in a namespace restricted to your platform team, avoid running any application workloads in that namespace, and treat its scrape configuration with the same scrutiny as a firewall rule.

## How the application agent scales

Application load scales with your workloads, not your machines, so splitting the work by node is the wrong split. Alloy clusters instead. The replicas form a hash ring and divide the scrape targets between them, rebalancing as replicas join or leave, so every target is scraped exactly once and adding a replica adds capacity. The same pool terminates OTLP, which puts span metrics next to the traces they are generated from.

## Alloy, and the alternatives

Alloy carries metrics, logs, and traces in one binary. The usual alternative is three: a Prometheus agent, Promtail, and an OpenTelemetry Collector, which means three config languages, three upgrade cycles, and three sets of host mounts on every node. Its Kubernetes service discovery also saves you a hand-written target list that breaks every time a component moves.

None of this is specific to Alloy. Vector or Fluent Bit read the same log paths, the OpenTelemetry Collector takes the same OTLP, and a plain Prometheus agent scrapes the same loopback endpoints. What matters is the split: infrastructure collection in a host-network DaemonSet, application collection in a pool that scales sideways. The rest of this guide uses Alloy because it lets us give you one coherent set of configs.

To get started, follow [Deploy the System Alloy](/docs/hetzner/apalla/observability/collection/deploy-the-system-alloy) to establish node and control-plane telemetry. Once you have applications running, proceed to [Deploy the Application Alloy](/docs/hetzner/apalla/observability/collection/deploy-the-application-alloy) to monitor your workloads.