This page covers collecting metrics from your own applications. Unlike the platform components, which keep their metrics on each node's loopback, your app runs on the pod network, where an ordinary scrape works. Instrument it, expose a `/metrics` endpoint, and declare a `ServiceMonitor` or `PodMonitor` next to it. The [Application Alloy](/docs/hetzner/apalla/observability/collection/deploy-the-application-alloy) reads those objects and scrapes what they select.

The interface is the same one the Prometheus operator uses, so nothing about writing them changes: a `ServiceMonitor` selects a Service, a `PodMonitor` selects pods directly for a workload with no Service in front of it.

## Instrument and expose

Add a Prometheus client library for your language, register your counters and histograms, and serve them on an HTTP path, `/metrics` by convention, on a named container port:

```yaml
ports:
  - name: metrics
    containerPort: 8080
```

## Declare the monitor

The Application Alloy reads every `ServiceMonitor` in every namespace unless you narrow it with a selector, so a new object is picked up with no collector change:

```yaml title="app-servicemonitor.yaml"
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-app
  namespace: my-app-namespace
spec:
  selector:
    matchLabels:
      app: my-app
  endpoints:
    - port: metrics # the named port on the Service
      interval: 30s
```

Keep the monitor in the same namespace as the workload it selects. That way the team that owns the app owns its scrape config too, and the collector needs no change when a new service appears.

> [!NOTE]
> Because the agent reads every monitor by default, a `ServiceMonitor` anywhere in the cluster becomes a scrape target. Narrow the agent with a namespace list or a label selector if you want an explicit allow-list instead.

## Why application metrics go to their own collector

A platform component's metrics are bounded: the cluster decides how many nodes and pods exist, so the series count moves slowly and predictably. An application's are not. One histogram with a label carrying a request id, a user id, or a full URL path turns into hundreds of thousands of series in an afternoon, and the collector scraping it runs out of memory.

That is survivable when the collector only handles applications. It is not survivable when the same agent also collects the node's data (kubelet, etcd, Cilium, and logs), because the crash then deletes the evidence you would investigate it with. Application scrape jobs stay on the clustered Application Alloy and never move to the node-level DaemonSet ([Using Alloy for observability](/docs/hetzner/apalla/observability/collection/using-alloy-for-observability)).

Clustering also has a practical benefit: the replicas shard your targets between them, so a service with hundreds of pods spreads across the pool instead of loading one agent.

## Keep the series count sane

Drop what you will not query before it reaches storage, with `metricRelabelings` on the monitor. The Application Alloy honors them, so this is still the right place for the fix, and it lives next to the metric's owner:

```yaml
endpoints:
  - port: metrics
    interval: 30s
    metricRelabelings:
      - sourceLabels: [__name__]
        regex: "myapp_debug_.*"
        action: drop
```

Prefer dropping a label to dropping a metric when the metric itself is useful. A latency histogram is worth keeping, but with a request id label on it, it splits into one series per request, and that label is what to drop.

## Verify and scope

Check that the target appears and is up. Alloy's UI lists the targets each component resolved, which tells you whether the monitor was read at all:

```console
$ kubectl -n monitoring port-forward svc/app-alloy 12345:12345
```

Two things explain a missing target almost every time: the monitor's `selector` does not match the Service's labels, or the Service has no port with the name the monitor references. The other problem is duplicated data: the same sample shows up in Prometheus two or three times. That means every Alloy replica is scraping your app on its own instead of sharing the work. [Troubleshoot the collectors](/docs/hetzner/apalla/observability/collection/troubleshoot-the-collectors) covers that failure.

For metrics you can trace back to a specific request, add exemplars so a spike on a graph links to the trace behind it. See [Correlate traces, metrics, and logs](/docs/hetzner/apalla/observability/traces/correlate-traces-metrics-and-logs).