Correlate traces, metrics, and logs

Traces, metrics, and logs only pay off together when they are linked. The goal is that a spike on a metrics panel jumps to an example trace, the trace jumps to the log lines for that request, and a log line jumps back to its trace, all without copy-pasting an id between screens. Grafana ties Prometheus, Tempo, and Loki together to do exactly this.

flowchart LR
  M["Metric spike<br/>(Prometheus)"] -->|exemplar| T["Slow trace<br/>(Tempo)"]
  T -->|trace-to-logs| L["Log lines<br/>(Loki)"]
  L -->|derived field| T

Metrics to traces, with exemplars#

An exemplar is a trace id attached to a metric sample. When your app records exemplars (Instrument a workload ) and Prometheus stores them, a Grafana latency panel shows markers you can click to open the trace behind that data point. That turns "latency spiked here" into "here is a request that was slow at that moment."

Traces to logs, with shared context#

Configure Tempo's trace-to-logs link so a span opens the matching logs in Loki. It matches on shared labels and the trace id, so both sides must agree: your logs must carry the trace id, and both must share labels like namespace and pod. Then, from a slow span, one click shows the log lines that service wrote while handling that request.

Logs to traces, with derived fields#

The reverse link is a Loki derived field: a regex that finds the trace id in a log line and turns it into a link to Tempo. When you are reading logs and spot an error, its trace is one click away. This only works if the trace id is in the log line, which is the next section.

Inject the trace id into logs#

The linchpin is that every log line carries the trace id of the request it belongs to. Configure your logging library to pull the current trace id from the OpenTelemetry context and add it as a structured field (trace_id). Do this once per service and all three links light up: metrics carry exemplars, traces find logs, logs find traces.

A unified troubleshooting dashboard#

Put the three on one dashboard for a service: latency and error-rate panels (with exemplars), a logs panel filtered to the service, and a link to its traces. During an incident you stay on one page: see the error rate climb, click an exemplar to the slow trace, click the span to the logs, read the error. That path from symptom to cause in a few clicks is the reason to run all three. Keep the dashboard in Git; see Dashboards as code .