Using Alloy for observability

Running every scrape job through a single node-level agent means one bad application can take down monitoring for the whole node. An application that labels a metric with something unique, a request ID for example, can generate hundreds of thousands of series. The agent runs out of memory and crashes. While it is down, nothing on that node is collected: not the kubelet, not etcd, not the health daemon, not the logs, and not the application that caused it. The data you need to debug the incident disappears exactly when you need it.

So we split collection in two. A DaemonSet handles infrastructure telemetry, which is required because it is the only way to reach the node's loopback endpoints. A separate agent handles application telemetry and scales on its own. If an application overwhelms its own collector, the node's monitoring keeps running.

flowchart TD
  subgraph infra["Infrastructure telemetry"]
    node["Worker and control-plane nodes<br/>loopback metrics endpoints"]
    logs["Container, journald,<br/>and audit logs"]
    ds["System Alloy<br/>host-network DaemonSet"]
    node -->|"scrape 127.0.0.1"| ds
    logs -->|"tail /var/log"| ds
  end
  subgraph apps["Application telemetry"]
    app["Application pods<br/>Prometheus endpoints"]
    inst["Instrumented services<br/>OTLP exporters"]
    sts["Application Alloy<br/>clustered StatefulSet"]
    app -->|"scrape ServiceMonitor targets"| sts
    inst -->|"push OTLP"| sts
  end
  ds -->|"infrastructure metrics and logs"| backend["Your backend<br/>Prometheus or Mimir, Loki, Tempo"]
  sts -->|"application metrics and traces"| backend

Both agents write to the same backend, so the split stops at the collection tier. Your dashboards query everything as one dataset.

What each agent owns

One rule keeps that separation intact: never add an application scrape job to the DaemonSet. A platform component emits a predictable number of series. An application emits as many as its labels allow, and that number is out of your hands.

Host network access

Most of the platform's metrics endpoints bind to 127.0.0.1 and use plain HTTP with no authentication. This is intentional: loopback-only means nothing off the node can reach them.

This also means an ordinary scraper cannot reach them. Inside a pod, 127.0.0.1 refers to the pod's own loopback, not the node's. Only a pod sharing the node's network namespace can see these endpoints. See The observability model for background, and Reference: metrics per component for which endpoint lives where.

This is why the System Alloy pod is configured the way it is:

• hostNetwork: true: puts the agent on the node's network namespace, so 127.0.0.1 resolves to the node.
• dnsPolicy: ClusterFirstWithHostNet: keeps cluster DNS working, so the agent can still resolve the backend's Service name.
• Read-only host mounts (/var/log/pods, /var/log/journal, audit paths): lets one agent collect both logs and metrics, instead of running a separate agent per signal.
• Tolerates all taints: the agent runs on every node, including tainted control planes and any nodes your own workloads taint. A skipped node means no telemetry from that node.
• Node-role discovery and relabeling: routes each job to the right nodes. Control-plane metrics only exist on control planes, syself-proxy only on workers, and the CSI node plugin only on cloud VM workers. Without this, a static target list would cause constant connection refused errors and false TargetDown alerts.
• Write-ahead log on disk: buffers samples when the backend is unreachable, so a backend restart costs you latency instead of data.

How the application agent scales

Application load scales with your workloads, not your machines, so splitting the work by node is the wrong split. The Application Alloy runs as a cluster of replicas instead. The replicas form a hash ring and divide the scrape targets between them, rebalancing as replicas join or leave, so every target is scraped exactly once and adding a replica adds capacity. The same pool terminates OTLP, which puts span metrics next to the traces they are generated from.

Alloy, and the alternatives

Alloy carries metrics, logs, and traces in one binary. The usual alternative is three: a Prometheus agent, Promtail, and an OpenTelemetry Collector, which means three config languages, three upgrade cycles, and three sets of host mounts on every node. Its Kubernetes service discovery also saves you a hand-written target list that breaks every time a component moves.

The architecture here does not depend on Alloy. Vector or Fluent Bit read the same log paths, the OpenTelemetry Collector takes the same OTLP, and a plain Prometheus agent scrapes the same loopback endpoints. What matters is the split: infrastructure collection in a host-network DaemonSet, application collection in a pool that scales by adding replicas. The rest of this guide uses Alloy because it lets us give you one coherent set of configs.

Start with Deploy the System Alloy for node and control-plane telemetry. Add Deploy the Application Alloy once you have applications to monitor.