Skip to main content

Deploy the Application Alloy

Inspect 1.36

Application telemetry scales with your workloads, not with machine count, so it needs an agent that scales the same way rather than a DaemonSet. The Application Alloy is a StatefulSet whose replicas form a cluster, shard the scrape targets between them, and rebalance as replicas join or leave. The same pool also receives OTLP traces, so deriving span metrics is cheap: the traces are already in memory.

Keeping this separate from is the whole point of the split. If a workload ships a high-cardinality metric and the collector runs out of memory, only application telemetry is affected; the node's own view, the kubelet, etcd, and log shipping, keeps running.

Prerequisites: Prometheus operator's CRDs, which its bundle installs ( ), a remote-write endpoint, and a trace backend if you collect traces ( ).

Your workloads keep the interface they already have

Application teams do not have to learn Alloy. The agent reads the ServiceMonitor and PodMonitor resources they already use with the Prometheus Operator and turns them into scrape targets, so deploying a ServiceMonitor next to an application starts its metrics flowing with no central configuration change. The metricRelabelings in those objects still apply, so the cardinality controls in keep working.

The only change is what does the scraping: the Application Alloy becomes the collector, and Prometheus is left as the storage and alerting plane.

Deploy it

Write the values file

alloy.clustering.enabled is what turns the replicas into one cluster instead of three agents each scraping every target. Every target-discovery component also needs its own clustering block to opt into the sharding, so that each series is collected once across the pool.

Note

The wal volume uses storageClassName: standard, which provisions only on cloud nodes, so the values file pins the StatefulSet to cloud nodes. Without that nodeSelector, a replica scheduled onto a bare-metal node cannot provision its claim and stays Pending with no topology key found. Keep the default: the write-ahead log is small, and a standard volume follows its replica through a node replacement.

Install the chart 

		$ helm upgrade -i app-alloy grafana/alloy \
  --namespace monitoring \
  --version <chart-version> \
  --values app-alloy-values.yaml

Confirm the cluster forms

The replicas must form a cluster before sharding happens. Alloy has a built-in clustering UI that shows the peers each replica sees:

		$ kubectl -n monitoring port-forward svc/app-alloy 12345:12345

Open http://localhost:12345/clustering in your browser and check that every replica sees every other one. A pod that only sees itself has not joined the cluster and is scraping every target on its own.

Confirm the sharding

Clustering exists so a target is scraped once across the pool rather than once per replica. In the Alloy UI (http://localhost:12345), open the prometheus.operator.servicemonitors.apps component and check that the targets are spread across the replicas.

Note

Broken sharding shows up as out-of-order sample rejections at the receiver, not as extra instance values, so confirm it in the Alloy UI rather than with a PromQL count.

Then point a service at the trace receiver and confirm spans arrive. Applications export OTLP to the chart's Service endpoint:

text
		http://app-alloy.monitoring.svc.cluster.local:4318

For instructions on configuring applications to send traces, see .

Sizing and isolation

Start with three replicas, and scale on the agent's memory use and scrape duration rather than raw target counts. Set a PodDisruptionBudget so a node drain does not take the pool below quorum, and give the write-ahead log volume enough room to buffer during backend restarts.

Keep the isolation. When the Application Alloy is under load, it is tempting to move scrape jobs onto the System Alloy DaemonSet to use its spare capacity. That gives up the one property worth keeping: the DaemonSet survives when everything else fails. If the application agent needs more capacity, add a replica.

If a scrape target fails to appear or the cluster refuses to converge, refer to for debugging steps.

Deploy the System Alloy

Install one Grafana Alloy DaemonSet on the host network that scrapes every loopback and cluster component metrics endpoint and ships container, journald, and audit logs off the node.

Troubleshoot the collectors

The failure modes worth knowing by name, from a target that is down and a series scraped twice to out-of-order samples and a cardinality spike.