Instrument a workload

Make your service emit traces one of two ways: add the OpenTelemetry SDK in code, or let the OpenTelemetry Operator inject auto-instrumentation with no code change. Both send spans to the Application Alloy . Either way, your services create spans and pass the trace's id from one to the next, since a trace is only as good as the instrumentation behind it.

		env:
  - {name: OTEL_EXPORTER_OTLP_ENDPOINT, value: "http://app-alloy.monitoring.svc.cluster.local:4318"}
  - {name: OTEL_SERVICE_NAME, value: "checkout"}
	

		metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-java: "true"
	

Context propagation is the whole point#

Context propagation is passing the trace id from one service to the next. Distributed tracing works only if each service does this, in a standard header (W3C traceparent). The instrumentation libraries do this across HTTP and gRPC, but it breaks at any hop that strips or does not forward headers: a queue, a custom protocol, a proxy that drops unknown headers. When a trace stops at a service boundary, a dropped traceparent is the first thing to check.

Set service.name and resource attributes#

service.name separates one service's spans from another's, so set it deliberately (checkout, not app). Add resource attributes for version and environment (service.version, deployment.environment) so you can tell a canary's traces from the stable ones. A trace with a blank or duplicated service.name is of little use.

Decide how much to keep#

There are two places to drop traces you do not need: the app can sample (send only a fraction), or send everything and let the collector sample, which lets it keep all error traces with tail sampling. For most services, sample low in the app and do tail sampling at the collector so you never lose a trace of a failed request.

Exemplars tie a metric to a trace#

An exemplar is a link from a metric to an example trace. Have the app attach the trace id to its Prometheus metrics as an exemplar. Then a spike on a latency graph carries a link straight to an example slow trace, the start of the connected troubleshooting in Correlate traces, metrics, and logs .

When instrumentation goes wrong#

• No traces at all: the app is exporting to the backend directly, or to the wrong Collector address.
• Traces that stop mid-flow: a hop is not propagating traceparent.
• Every span under one service name: service.name is unset, so everything collapses together.