Custom application metrics

This page covers collecting metrics from your own applications. Unlike the platform components, which keep their metrics on each node's loopback, your app runs on the pod network, where an ordinary scrape works. Instrument it, expose a /metrics endpoint, and declare a ServiceMonitor or PodMonitor next to it. The Application Alloy reads those objects and scrapes what they select.

The interface is the same one the Prometheus operator uses, so nothing about writing them changes: a ServiceMonitor selects a Service, a PodMonitor selects pods directly for a workload with no Service in front of it.

Instrument and expose

Add a Prometheus client library for your language, register your counters and histograms, and serve them on an HTTP path, /metrics by convention, on a named container port:

		ports:
  - name: metrics
    containerPort: 8080
	

Declare the monitor

The Application Alloy reads every ServiceMonitor in every namespace unless you narrow it with a selector, so a new object is picked up with no collector change:

		apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-app
  namespace: my-app-namespace
spec:
  selector:
    matchLabels:
      app: my-app
  endpoints:
    - port: metrics # the named port on the Service
      interval: 30s
	

Keep the monitor in the same namespace as the workload it selects. That way the team that owns the app owns its scrape config too, and the collector needs no change when a new service appears.

Why application metrics go to their own collector

A platform component's metrics are bounded: the cluster decides how many nodes and pods exist, so the series count moves slowly and predictably. An application's are not. One histogram with a label carrying a request id, a user id, or a full URL path turns into hundreds of thousands of series in an afternoon, and the collector scraping it runs out of memory.

That is survivable when the collector only handles applications. It is not survivable when the same agent also collects the node's data (kubelet, etcd, Cilium, and logs), because the crash then deletes the evidence you would investigate it with. Application scrape jobs stay on the clustered Application Alloy and never move to the node-level DaemonSet (Using Alloy for observability ).

Clustering also has a practical benefit: the replicas shard your targets between them, so a service with hundreds of pods spreads across the pool instead of loading one agent.

Keep the series count sane

Drop what you will not query before it reaches storage, with metricRelabelings on the monitor. The Application Alloy honors them, so this is still the right place for the fix, and it lives next to the metric's owner:

		endpoints:
  - port: metrics
    interval: 30s
    metricRelabelings:
      - sourceLabels: [__name__]
        regex: "myapp_debug_.*"
        action: drop
	

Prefer dropping a label to dropping a metric when the metric itself is useful. A latency histogram is worth keeping, but with a request id label on it, it splits into one series per request, and that label is what to drop.

Verify and scope

Check that the target appears and is up. Alloy's UI lists the targets each component resolved, which tells you whether the monitor was read at all:

		$ kubectl -n monitoring port-forward svc/app-alloy 12345:12345
	

Two things explain a missing target almost every time: the monitor's selector does not match the Service's labels, or the Service has no port with the name the monitor references. The other problem is duplicated data: the same sample shows up in Prometheus two or three times. That means every Alloy replica is scraping your app on its own instead of sharing the work. Troubleshoot the collectors covers that failure.

For metrics you can trace back to a specific request, add exemplars so a spike on a graph links to the trace behind it. See Correlate traces, metrics, and logs .