Make your service emit traces one of two ways: add the OpenTelemetry SDK in code, or let the OpenTelemetry Operator inject auto-instrumentation with no code change. Both send spans to the [Application Alloy](/docs/hetzner/apalla/observability/traces/ingest-otlp-traces). Either way, your services create spans and pass the trace's id from one to the next, since a trace is only as good as the instrumentation behind it.

<Tabs id="otel-instrumentation" items={['SDK', 'Auto-instrumentation']}>
<TabItem value="SDK">

Add the OpenTelemetry SDK for your language and its instrumentation libraries for your web framework and clients. The libraries create spans for incoming requests and outgoing calls automatically; you add spans by hand only for the business logic you care about. Point the SDK at the collector with the standard environment variables:

```yaml
env:
  - {name: OTEL_EXPORTER_OTLP_ENDPOINT, value: "http://app-alloy.monitoring.svc.cluster.local:4318"}
  - {name: OTEL_SERVICE_NAME, value: "checkout"}
```

</TabItem>
<TabItem value="Auto-instrumentation">

The OpenTelemetry Operator injects an instrumentation agent into a pod from an annotation, so a service starts emitting traces with no code change:

```yaml
metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-java: "true"
```

It fits languages with a runtime agent (Java, Node, Python, .NET) and is the fastest way to get coverage across many services. Add SDK spans later where you want detail the agent cannot see.

</TabItem>
</Tabs>

## Context propagation is the whole point

Context propagation is passing the trace id from one service to the next. Distributed tracing works only if each service does this, in a standard header (W3C `traceparent`). The instrumentation libraries do this across HTTP and gRPC, but it breaks at any hop that strips or does not forward headers: a queue, a custom protocol, a proxy that drops unknown headers. When a trace stops at a service boundary, a dropped `traceparent` is the first thing to check.

## Set service.name and resource attributes

`service.name` separates one service's spans from another's, so set it deliberately (`checkout`, not `app`). Add resource attributes for version and environment (`service.version`, `deployment.environment`) so you can tell a canary's traces from the stable ones. A trace with a blank or duplicated `service.name` is of little use.

## Decide how much to keep

There are two places to drop traces you do not need: the app can sample (send only a fraction), or send everything and let the [collector](/docs/hetzner/apalla/observability/traces/ingest-otlp-traces) sample, which lets it keep all error traces with tail sampling. For most services, sample low in the app and do tail sampling at the collector so you never lose a trace of a failed request.

## Exemplars tie a metric to a trace

An exemplar is a link from a metric to an example trace. Have the app attach the trace id to its Prometheus metrics as an exemplar. Then a spike on a latency graph carries a link straight to an example slow trace, the start of the connected troubleshooting in [Correlate traces, metrics, and logs](/docs/hetzner/apalla/observability/traces/correlate-traces-metrics-and-logs).

## When instrumentation goes wrong

- No traces at all: the app is exporting to the backend directly, or to the wrong Collector address.
- Traces that stop mid-flow: a hop is not propagating `traceparent`.
- Every span under one service name: `service.name` is unset, so everything collapses together.