Deploy the Application Alloy

Inspect 1.36

Application telemetry scales with your workloads, not with machine count, so it needs an agent that scales the same way rather than a DaemonSet. The Application Alloy is a StatefulSet whose replicas form a cluster, shard the scrape targets between them, and rebalance as replicas join or leave. The same pool also receives OTLP traces, so deriving span metrics is cheap: the traces are already in memory.

Keeping this separate from the System Alloy is the whole point of the split. If a workload ships a high-cardinality metric and the collector runs out of memory, only application telemetry is affected; the node's own view, the kubelet, etcd, and log shipping, keeps running.

Prerequisites: Prometheus operator's CRDs, which its bundle installs (Set up Prometheus ), a remote-write endpoint, and a trace backend if you collect traces (Install Tempo or Jaeger ).

Your workloads keep the interface they already have#

Application teams do not have to learn Alloy. The agent reads the ServiceMonitor and PodMonitor resources they already use with the Prometheus Operator and turns them into scrape targets, so deploying a ServiceMonitor next to an application starts its metrics flowing with no central configuration change. The metricRelabelings in those objects still apply, so the cardinality controls in Custom application metrics keep working.

The only change is what does the scraping: the Application Alloy becomes the collector, and Prometheus is left as the storage and alerting plane.

Deploy it#

Write the values file #

alloy.clustering.enabled is what turns the replicas into one cluster instead of three agents each scraping every target. Every target-discovery component also needs its own clustering block to opt into the sharding, so that each series is collected once across the pool.

app-alloy-values.yamlyaml

		controller:
  type: statefulset
  replicas: 3
  # A pod-network agent: no host network, no control-plane toleration.
  # standard volumes provision only on cloud nodes, so keep the replicas there.
  # Drop this if you point storageClassName at a local class instead.
  nodeSelector:
    autopilot.syself.com/machine-type: hcloud
  volumeClaimTemplates:
    - metadata: {name: wal}
      spec:
        accessModes: [ReadWriteOnce]
        # standard = Hetzner Cloud volumes, cloud nodes only. The nodeSelector
        # above keeps this on a cloud node. To run on a bare-metal node instead,
        # drop it and name the local class matching that node's disks.
        # (local-nvme, local-ssd, or local-hdd through TopoLVM).
        storageClassName: standard
        resources: {requests: {storage: 10Gi}}
 
# Let the agent be scraped for its own metrics. It reads every ServiceMonitor,
# including this one, so it ends up monitoring itself.
serviceMonitor:
  enabled: true
 
alloy:
  clustering:
    enabled: true
  storagePath: /var/lib/alloy
  mounts:
    extra:
      - {name: wal, mountPath: /var/lib/alloy}
  # Expose the OTLP receivers through the chart's Service.
  extraPorts:
    - {name: otlp-grpc, port: 4317, targetPort: 4317, protocol: TCP}
    - {name: otlp-http, port: 4318, targetPort: 4318, protocol: TCP}
  configMap:
    content: |
      // ===================== Destinations =====================
 
      prometheus.remote_write "central" {
        endpoint {
          url = "http://prometheus-operated.monitoring.svc.cluster.local:9090/api/v1/write"
        }
      }
 
      otelcol.exporter.otlp "tempo" {
        client {
          endpoint = "tempo.monitoring.svc.cluster.local:4317"
          tls {
            insecure = true
          }
        }
      }
 
      // ===================== Application metrics =====================
 
      // Read the ServiceMonitor objects your teams already write. The
      // clustering block is what shards the resulting targets across replicas,
      // so each endpoint is scraped once rather than once per replica.
      prometheus.operator.servicemonitors "apps" {
        forward_to = [prometheus.remote_write.central.receiver]
 
        clustering {
          enabled = true
        }
      }
 
      // The same, for workloads with no Service in front of them.
      prometheus.operator.podmonitors "apps" {
        forward_to = [prometheus.remote_write.central.receiver]
 
        clustering {
          enabled = true
        }
      }
 
      // ===================== Traces =====================
 
      otelcol.receiver.otlp "default" {
        grpc {
          endpoint = "0.0.0.0:4317"
        }
        http {
          endpoint = "0.0.0.0:4318"
        }
 
        output {
          traces = [otelcol.processor.batch.default.input]
        }
      }
 
      otelcol.processor.batch "default" {
        output {
          traces = [
            otelcol.connector.spanmetrics.default.input,
            otelcol.exporter.otlp.tempo.input,
          ]
        }
      }
 
      // Turn spans into request-rate and latency series, so a service gets its
      // golden signals without being instrumented for metrics separately.
      otelcol.connector.spanmetrics "default" {
        namespace = "traces_span_metrics"
 
        histogram {
          explicit {
            buckets = ["10ms", "50ms", "100ms", "500ms", "1s", "5s"]
          }
        }
 
        output {
          metrics = [otelcol.exporter.prometheus.spanmetrics.input]
        }
      }
 
      otelcol.exporter.prometheus "spanmetrics" {
        forward_to = [prometheus.remote_write.central.receiver]
      }

Note

The wal volume sets storageClassName: standard, which is backed by Hetzner Cloud volumes and provisions only on cloud nodes, which is why the values file pins the StatefulSet to cloud nodes. Without that nodeSelector, on a cluster with bare-metal nodes any replica scheduled onto one cannot provision its claim there and stays Pending with no topology key found. Keep the default: the write-ahead log is small, it is not latency-sensitive, and a standard volume follows its replica, so a node replacement does not cost you the buffer. Move off it when the cluster has no cloud nodes, or when you want the log on local disk on purpose. To run the replicas on bare metal, change the selector and set storageClassName to the local class that matches those nodes' disks (local-nvme, local-ssd, or local-hdd) through TopoLVM . Check which one the node offers before you pick:

		$ kubectl get node <node-name> -o json \
  | jq -r '.metadata.annotations | to_entries[]
           | select(.key | startswith("capacity.topolvm.io")) | "\(.key)\t\(.value)"'
capacity.topolvm.io/00default	0
capacity.topolvm.io/dc-ssd	501118664704

Each line is a device class and its free bytes. Here only dc-ssd has capacity, so local-ssd is the class that binds and local-nvme would leave the claim Pending with did not have enough free storage.

Install the chart #

		$ helm upgrade -i app-alloy grafana/alloy \
  --namespace monitoring \
  --version <chart-version> \
  --values app-alloy-values.yaml

Confirm the cluster forms #

The replicas must form a cluster before sharding happens. Alloy has a built-in clustering UI that shows the peers each replica sees:

		$ kubectl -n monitoring port-forward svc/app-alloy 12345:12345

Open http://localhost:12345/clustering in your browser and check that every replica sees every other one. A pod that only sees itself has not joined the cluster and is scraping every target on its own.

Confirm the sharding #

Clustering exists so a target is scraped once across the pool rather than once per replica. Confirm target distribution across replicas in the Alloy UI (http://localhost:12345) or check component target ownership under prometheus.operator.servicemonitors.apps.

Note

count(count by (instance) (up{job="my-app"})) measures distinct target pod instances, not collector replica counts. If sharding were broken, all replicas would scrape the same targets and emit duplicate series with identical labels, which causes out-of-order sample rejections at the receiver rather than extra instance rows. Confirm sharding in the Alloy UI target list instead.

Then point a service at the trace receiver and confirm spans arrive. Applications export OTLP to the chart's Service endpoint:

text

		http://app-alloy.monitoring.svc.cluster.local:4318

For instructions on configuring applications to send traces, see Instrument a workload .

Size it, and let it be the thing that breaks#

Start with three replicas, and scale on the agent's memory use and scrape duration rather than raw target counts. Set a PodDisruptionBudget so a node drain does not take the pool below quorum, and give the write-ahead log volume enough room to buffer during backend restarts.

Keep the isolation. When the Application Alloy is under load, moving scrape jobs onto the System Alloy DaemonSet to use its spare node capacity is tempting, but it gives up the one property worth keeping: the DaemonSet surviving when everything else fails. If the application agent needs more capacity, add a replica.

If a scrape target fails to appear or the cluster refuses to converge, refer to Troubleshoot the collectors for debugging steps.

Deploy the System Alloy

Install one Grafana Alloy DaemonSet on the host network that scrapes every loopback and cluster component metrics endpoint and ships container, journald, and audit logs off the node.

Troubleshoot the collectors

The failure modes worth knowing by name, from a target that is down and a series scraped twice to out-of-order samples and a cardinality spike.