Application telemetry scales with your workloads, not with machine count, so it needs an agent that scales the same way rather than a DaemonSet. The Application Alloy is a StatefulSet whose replicas form a cluster, shard the scrape targets between them, and rebalance as replicas join or leave. The same pool also receives OTLP traces, so deriving span metrics is cheap: the traces are already in memory.

Keeping this separate from [the System Alloy](/docs/hetzner/apalla/observability/collection/deploy-the-system-alloy) is the whole point of the split. If a workload ships a high-cardinality metric and the collector runs out of memory, only application telemetry is affected; the node's own view, the kubelet, etcd, and log shipping, keeps running.

**Prerequisites**: Prometheus operator's CRDs, which its bundle installs ([Set up Prometheus](/docs/hetzner/apalla/observability/metrics/set-up-prometheus)), a remote-write endpoint, and a trace backend if you collect traces ([Install Tempo or Jaeger](/docs/hetzner/apalla/observability/traces/install-tempo-or-jaeger)).

## Your workloads keep the interface they already have

Application teams do not have to learn Alloy. The agent reads the `ServiceMonitor` and `PodMonitor` resources they already use with the Prometheus Operator and turns them into scrape targets, so deploying a `ServiceMonitor` next to an application starts its metrics flowing with no central configuration change. The `metricRelabelings` in those objects still apply, so the cardinality controls in [Custom application metrics](/docs/hetzner/apalla/observability/metrics/custom-application-metrics) keep working.

The only change is what does the scraping: the Application Alloy becomes the collector, and Prometheus is left as the storage and alerting plane.

## Deploy it

<Steps>
<Step>Write the values file</Step>

`alloy.clustering.enabled` is what turns the replicas into one cluster instead of three agents each scraping every target. Every target-discovery component also needs its own `clustering` block to opt into the sharding, so that each series is collected once across the pool.

```yaml title="app-alloy-values.yaml"
controller:
  type: statefulset
  replicas: 3
  # A pod-network agent: no host network, no control-plane toleration.
  # standard volumes provision only on cloud nodes, so keep the replicas there.
  # Drop this if you point storageClassName at a local class instead.
  nodeSelector:
    autopilot.syself.com/machine-type: hcloud
  volumeClaimTemplates:
    - metadata: {name: wal}
      spec:
        accessModes: [ReadWriteOnce]
        # standard = Hetzner Cloud volumes, cloud nodes only. The nodeSelector
        # above keeps this on a cloud node. To run on a bare-metal node instead,
        # drop it and name the local class matching that node's disks.
        # (local-nvme, local-ssd, or local-hdd through TopoLVM).
        storageClassName: standard
        resources: {requests: {storage: 10Gi}}

# Let the agent be scraped for its own metrics. It reads every ServiceMonitor,
# including this one, so it ends up monitoring itself.
serviceMonitor:
  enabled: true

alloy:
  clustering:
    enabled: true
  storagePath: /var/lib/alloy
  mounts:
    extra:
      - {name: wal, mountPath: /var/lib/alloy}
  # Expose the OTLP receivers through the chart's Service.
  extraPorts:
    - {name: otlp-grpc, port: 4317, targetPort: 4317, protocol: TCP}
    - {name: otlp-http, port: 4318, targetPort: 4318, protocol: TCP}
  configMap:
    content: |
      // ===================== Destinations =====================

      prometheus.remote_write "central" {
        endpoint {
          url = "http://prometheus-operated.monitoring.svc.cluster.local:9090/api/v1/write"
        }
      }

      otelcol.exporter.otlp "tempo" {
        client {
          endpoint = "tempo.monitoring.svc.cluster.local:4317"
          tls {
            insecure = true
          }
        }
      }

      // ===================== Application metrics =====================

      // Read the ServiceMonitor objects your teams already write. The
      // clustering block is what shards the resulting targets across replicas,
      // so each endpoint is scraped once rather than once per replica.
      prometheus.operator.servicemonitors "apps" {
        forward_to = [prometheus.remote_write.central.receiver]

        clustering {
          enabled = true
        }
      }

      // The same, for workloads with no Service in front of them.
      prometheus.operator.podmonitors "apps" {
        forward_to = [prometheus.remote_write.central.receiver]

        clustering {
          enabled = true
        }
      }

      // ===================== Traces =====================

      otelcol.receiver.otlp "default" {
        grpc {
          endpoint = "0.0.0.0:4317"
        }
        http {
          endpoint = "0.0.0.0:4318"
        }

        output {
          traces = [otelcol.processor.batch.default.input]
        }
      }

      otelcol.processor.batch "default" {
        output {
          traces = [
            otelcol.connector.spanmetrics.default.input,
            otelcol.exporter.otlp.tempo.input,
          ]
        }
      }

      // Turn spans into request-rate and latency series, so a service gets its
      // golden signals without being instrumented for metrics separately.
      otelcol.connector.spanmetrics "default" {
        namespace = "traces_span_metrics"

        histogram {
          explicit {
            buckets = ["10ms", "50ms", "100ms", "500ms", "1s", "5s"]
          }
        }

        output {
          metrics = [otelcol.exporter.prometheus.spanmetrics.input]
        }
      }

      otelcol.exporter.prometheus "spanmetrics" {
        forward_to = [prometheus.remote_write.central.receiver]
      }
```

> [!NOTE]
> The `wal` volume sets `storageClassName: standard`, which is backed by Hetzner Cloud volumes and provisions only on cloud nodes, which is why the values file pins the StatefulSet to cloud nodes. Without that `nodeSelector`, on a cluster with bare-metal nodes any replica scheduled onto one cannot provision its claim there and stays `Pending` with `no topology key found`. Keep the default: the write-ahead log is small, it is not latency-sensitive, and a `standard` volume follows its replica, so a node replacement does not cost you the buffer. Move off it when the cluster has no cloud nodes, or when you want the log on local disk on purpose. To run the replicas on bare metal, change the selector and set `storageClassName` to the local class that matches those nodes' disks (`local-nvme`, `local-ssd`, or `local-hdd`) through [TopoLVM](/docs/hetzner/apalla/storage/local/local-nvme-with-topolvm). Check which one the node offers before you pick:
>
> ```console
> $ kubectl get node <node-name> -o json \
>   | jq -r '.metadata.annotations | to_entries[]
>            | select(.key | startswith("capacity.topolvm.io")) | "\(.key)\t\(.value)"'
> capacity.topolvm.io/00default	0
> capacity.topolvm.io/dc-ssd	501118664704
> ```
>
> Each line is a device class and its free bytes. Here only `dc-ssd` has capacity, so `local-ssd` is the class that binds and `local-nvme` would leave the claim `Pending` with `did not have enough free storage`.

<Step>Install the chart</Step>

```console
$ helm upgrade -i app-alloy grafana/alloy \
  --namespace monitoring \
  --version <chart-version> \
  --values app-alloy-values.yaml
```

<Step>Confirm the cluster forms</Step>

The replicas must form a cluster before sharding happens. Alloy has a built-in clustering UI that shows the peers each replica sees:

```console
$ kubectl -n monitoring port-forward svc/app-alloy 12345:12345
```

Open `http://localhost:12345/clustering` in your browser and check that every replica sees every other one. A pod that only sees itself has not joined the cluster and is scraping every target on its own.

<Step>Confirm the sharding</Step>

Clustering exists so a target is scraped once across the pool rather than once per replica. Confirm target distribution across replicas in the Alloy UI (`http://localhost:12345`) or check component target ownership under `prometheus.operator.servicemonitors.apps`.

> [!NOTE]
> `count(count by (instance) (up{job="my-app"}))` measures distinct target pod instances, not collector replica counts. If sharding were broken, all replicas would scrape the same targets and emit duplicate series with identical labels, which causes out-of-order sample rejections at the receiver rather than extra instance rows. Confirm sharding in the Alloy UI target list instead.

Then point a service at the trace receiver and confirm spans arrive. Applications export OTLP to the chart's Service endpoint:

```text
http://app-alloy.monitoring.svc.cluster.local:4318
```

For instructions on configuring applications to send traces, see [Instrument a workload](/docs/hetzner/apalla/observability/traces/instrument-a-workload).

</Steps>

## Size it, and let it be the thing that breaks

Start with three replicas, and scale on the agent's memory use and scrape duration rather than raw target counts. Set a `PodDisruptionBudget` so a node drain does not take the pool below quorum, and give the write-ahead log volume enough room to buffer during backend restarts.

Keep the isolation. When the Application Alloy is under load, moving scrape jobs onto the System Alloy DaemonSet to use its spare node capacity is tempting, but it gives up the one property worth keeping: the DaemonSet surviving when everything else fails. If the application agent needs more capacity, add a replica.

If a scrape target fails to appear or the cluster refuses to converge, refer to [Troubleshoot the collectors](/docs/hetzner/apalla/observability/collection/troubleshoot-the-collectors) for debugging steps.