Your own workloads are the easy case. The platform's components keep their metrics private on loopback and need an agent on the node to reach them; your app runs on the pod network, where an ordinary scrape works. Instrument it, expose a `/metrics` endpoint, and declare a `ServiceMonitor` or `PodMonitor` next to it. The [Application Alloy](/docs/hetzner/apalla/observability/collection/deploy-the-application-alloy) reads those objects and scrapes what they select.

The interface is the same one the Prometheus operator uses, so nothing about writing them changes: a `ServiceMonitor` selects a Service, a `PodMonitor` selects pods directly for a workload with no Service in front of it.

## Instrument and expose

Add a Prometheus client library for your language, register your counters and histograms, and serve them on an HTTP path, `/metrics` by convention, on a named container port:

```yaml
ports:
  - name: metrics
    containerPort: 8080
```

## Declare the monitor

The Application Alloy reads every `ServiceMonitor` in every namespace unless you narrow it with a selector, so a new object is picked up with no collector change:

```yaml title="app-servicemonitor.yaml"
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-app
  namespace: my-app-namespace
spec:
  selector:
    matchLabels:
      app: my-app
  endpoints:
    - port: metrics # the named port on the Service
      interval: 30s
```

Keep the monitor in the same namespace as the workload it selects. That way the team that owns the app owns its scrape config too, and the collector needs no change when a new service appears.

> [!NOTE]
> Because the agent reads every monitor by default, a `ServiceMonitor` anywhere in the cluster becomes a scrape target. Narrow the agent with a namespace list or a label selector if you want an explicit allow-list instead. Nothing else competes for those targets: the store selects no monitors at all ([Set up Prometheus](/docs/hetzner/apalla/observability/metrics/set-up-prometheus)), so the agent is the only scraper reading them.

## Why application metrics go to their own collector

A platform component's metrics are bounded: the cluster decides how many nodes and pods exist, so the series count moves slowly and predictably. An application's are not. One histogram with a label carrying a request id, a user id, or a full URL path turns into hundreds of thousands of series in an afternoon, and the collector scraping it runs out of memory.

That is survivable when the collector only handles applications. It is not survivable when the same agent is the node's only source of kubelet, etcd, Cilium, and log data, because the incident then deletes the evidence you would investigate it with. Application scrape jobs stay on the clustered Application Alloy and never move to the node-level DaemonSet, no matter how convenient the spare capacity looks. [Using Alloy for observability](/docs/hetzner/apalla/observability/collection/using-alloy-for-observability) is the longer version of this argument.

Clustering also has a practical benefit: the replicas shard your targets between them, so a service with hundreds of pods spreads across the pool instead of loading one agent.

## Keep the series count sane

Drop what you will not query before it reaches storage, with `metricRelabelings` on the monitor. The Application Alloy honors them, so this is still the right place for the fix, and it lives next to the metric's owner:

```yaml
endpoints:
  - port: metrics
    interval: 30s
    metricRelabelings:
      - sourceLabels: [__name__]
        regex: "myapp_debug_.*"
        action: drop
```

Prefer dropping a label to dropping a metric when the metric itself is useful. A histogram is worth keeping; the same histogram split by request id is not.

## Verify and scope

Check that the target appears and is up. Alloy's UI lists the targets each component resolved, which tells you whether the monitor was read at all:

```console
$ kubectl -n monitoring port-forward svc/app-alloy 12345:12345
```

Two things explain a missing target almost every time: the `release` label does not match, or the workload's Service has no port with the name the monitor references. If the target is there but the series arrive at multiples of the real rate, clustering is not sharding, which [Troubleshoot the collectors](/docs/hetzner/apalla/observability/collection/troubleshoot-the-collectors) covers.

For metrics you can trace back to a specific request, add exemplars so a spike on a graph links to the trace behind it. See [Correlate traces, metrics, and logs](/docs/hetzner/apalla/observability/traces/correlate-traces-metrics-and-logs).