Correlate traces, metrics, and logs

Traces, metrics, and logs only pay off together when they are linked. The goal is that a spike on a metrics panel jumps to an example trace, the trace jumps to the log lines for that request, and a log line jumps back to its trace, all without copy-pasting an id between screens. Grafana ties Prometheus, Tempo, and Loki together to do exactly this.

flowchart LR
  M["Metric spike<br/>(Prometheus)"] -->|exemplar| T["Slow trace<br/>(Tempo)"]
  T -->|trace-to-logs| L["Log lines<br/>(Loki)"]
  L -->|derived field| T

Metrics to traces, with exemplars

An exemplar is a trace id attached to a metric sample. Prometheus stores exemplars only when its exemplar-storage feature is on: add enableFeatures: ["exemplar-storage"] to the Prometheus object from Set up Prometheus . When your app records exemplars (Instrument a workload ) and that feature is enabled, a Grafana latency panel shows markers you can click to open the trace behind that data point. That turns "latency spiked here" into "here is a request that was slow at that moment."

Traces to logs, with shared context

Configure the Tempo data source's trace-to-logs link in Grafana so a span opens the matching logs in Loki. It matches on shared labels and the trace id, so both sides must agree: your logs must carry the trace id, and both must share labels like namespace and pod. Then, from a slow span, one click shows the log lines that service wrote while handling that request.

Logs to traces, with derived fields

The reverse link is a Loki derived field: a regex that finds the trace id in a log line and turns it into a link to Tempo. When you are reading logs and spot an error, its trace is one click away. This only works if the trace id is in the log line, which is the next section.

Inject the trace id into logs

Both log links, traces to logs and logs to traces, depend on one thing: every log line must carry the trace id of the request it belongs to. Configure your logging library to pull the current trace id from the OpenTelemetry context and add it as a structured field (trace_id). Do this once per service and both directions work: traces find logs, and logs find traces.

A unified troubleshooting dashboard

Build one dashboard per service that shows all three signals (metrics, logs and traces). It needs latency and error-rate panels (with exemplars), a logs panel filtered to the service, and a link to its traces. During an incident you stay on one page: see the error rate climb, click an exemplar to the slow trace, click the span to the logs, read the error. That path from symptom to cause in a few clicks is the reason to run all three. Keep the dashboard in Git; see Dashboards as code .