Custom application metrics

Your own workloads are the easy case. The platform's components keep their metrics private on loopback and need an agent on the node to reach them; your app runs on the pod network, where an ordinary scrape works. Instrument it, expose a /metrics endpoint, and declare a ServiceMonitor or PodMonitor next to it. The Application Alloy reads those objects and scrapes what they select.

The interface is the same one the Prometheus operator uses, so nothing about writing them changes: a ServiceMonitor selects a Service, a PodMonitor selects pods directly for a workload with no Service in front of it.

Instrument and expose#

Add a Prometheus client library for your language, register your counters and histograms, and serve them on an HTTP path, /metrics by convention, on a named container port:

		ports:
  - name: metrics
    containerPort: 8080
	

Declare the monitor#

The Application Alloy reads every ServiceMonitor in every namespace unless you narrow it with a selector, so a new object is picked up with no collector change:

		apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-app
  namespace: my-app-namespace
spec:
  selector:
    matchLabels:
      app: my-app
  endpoints:
    - port: metrics # the named port on the Service
      interval: 30s
	

Keep the monitor in the same namespace as the workload it selects. That way the team that owns the app owns its scrape config too, and the collector needs no change when a new service appears.

Why application metrics go to their own collector#

A platform component's metrics are bounded: the cluster decides how many nodes and pods exist, so the series count moves slowly and predictably. An application's are not. One histogram with a label carrying a request id, a user id, or a full URL path turns into hundreds of thousands of series in an afternoon, and the collector scraping it runs out of memory.

That is survivable when the collector only handles applications. It is not survivable when the same agent is the node's only source of kubelet, etcd, Cilium, and log data, because the incident then deletes the evidence you would investigate it with. Application scrape jobs stay on the clustered Application Alloy and never move to the node-level DaemonSet, no matter how convenient the spare capacity looks. Using Alloy for observability is the longer version of this argument.

Clustering also has a practical benefit: the replicas shard your targets between them, so a service with hundreds of pods spreads across the pool instead of loading one agent.

Keep the series count sane#

Drop what you will not query before it reaches storage, with metricRelabelings on the monitor. The Application Alloy honors them, so this is still the right place for the fix, and it lives next to the metric's owner:

		endpoints:
  - port: metrics
    interval: 30s
    metricRelabelings:
      - sourceLabels: [__name__]
        regex: "myapp_debug_.*"
        action: drop
	

Prefer dropping a label to dropping a metric when the metric itself is useful. A histogram is worth keeping; the same histogram split by request id is not.

Verify and scope#

Check that the target appears and is up. Alloy's UI lists the targets each component resolved, which tells you whether the monitor was read at all:

		$ kubectl -n monitoring port-forward svc/app-alloy 12345:12345
	

Two things explain a missing target almost every time: the release label does not match, or the workload's Service has no port with the name the monitor references. If the target is there but the series arrive at multiples of the real rate, clustering is not sharding, which Troubleshoot the collectors covers.

For metrics you can trace back to a specific request, add exemplars so a spike on a graph links to the trace behind it. See Correlate traces, metrics, and logs .