A trace is the record of one request as it moves across your services, and each step in it is a span. Your services do not send those spans to the trace store directly. They send them to a collector you run, in one standard protocol (OTLP), and it forwards them onward. That indirection is what lets you change trace backends without touching a single service, and what gives you one place to sample, batch, and enrich.

On this platform that collector is the [Application Alloy](/docs/hetzner/apalla/observability/collection/deploy-the-application-alloy), the same clustered agent that scrapes application metrics. Its values file already includes the OTLP receivers; this page is about what to do with the spans once they arrive.

## Where your services send them

The endpoint is the Application Alloy's Service, on `4317` for gRPC and `4318` for HTTP:

```text
http://app-alloy.monitoring.svc.cluster.local:4318
```

Point the standard OpenTelemetry environment variables at it and your services need to know nothing else. [Instrument a workload](/docs/hetzner/apalla/observability/traces/instrument-a-workload) covers the application side, in code or through the operator's auto-instrumentation.

## Why traces and application metrics share one agent

Span metrics are the reason. A connector reads the spans passing through and emits request-rate and latency series from them, so a service gets its golden signals without being instrumented for metrics separately. That only works cheaply when the traces and the metrics pipeline are in the same process, and it means one agent, one scaling decision, and one set of replicas that shard both the scraping and the trace load.

The span metrics land in your Prometheus alongside everything else, under whatever namespace you set on the connector, so you can alert on latency derived from traces with the same `PrometheusRule` objects you use for scraped metrics.

## Decide what you keep

Tracing every request is expensive, so you sample. There are two places to do it, and they answer different questions.

**Head sampling, in the application:** keep a fixed fraction of traces, decided when the trace starts. Cheap, and it reduces what your services emit in the first place, but it is blind to what the trace turns out to contain. A 5% head sample discards 95% of your errors.

**Tail sampling, in the collector:** decide after the trace finishes, when its duration and status are known. That is what lets you keep every failed and every slow trace while discarding the ordinary ones. It costs memory in the collector, because spans have to be held until the trace is complete.

For most services, sample modestly in the application and add tail sampling here for the traces that matter:

```terraform
otelcol.processor.tail_sampling "default" {
  policy {
    name = "errors"
    type = "status_code"
    status_code {
      status_codes = ["ERROR"]
    }
  }

  policy {
    name = "slow"
    type = "latency"
    latency {
      threshold_ms = 500
    }
  }

  policy {
    name = "sample-the-rest"
    type = "probabilistic"
    probabilistic {
      sampling_percentage = 5
    }
  }

  output {
    traces = [otelcol.exporter.otlp.tempo.input]
  }
}
```

With tail sampling in the pipeline, send the receiver's output through it on the way to the exporter, and keep the span metrics connector reading the unsampled stream so your rates stay accurate. Sampled traces make a good archive and a bad denominator.

## Enrich once, at the collector

Stamping attributes here rather than in every service keeps them consistent and means one change covers the fleet. The cluster name is the one that matters most: without it, traces from two clusters are indistinguishable in a shared backend, which is the same problem external labels solve for metrics ([Multi-cluster observability](/docs/hetzner/apalla/observability/multi-cluster/multi-cluster-observability)).

```terraform
otelcol.processor.resourcedetection "default" {
  detectors = ["env", "system"]

  output {
    traces = [otelcol.processor.batch.default.input]
  }
}
```

## Confirm a span arrives

Send one by hand rather than waiting on a service. Port-forward the receiver and post a minimal trace:

```console
$ kubectl -n monitoring port-forward svc/app-alloy 4318:4318
```

A `200` with `{"partialSuccess":{}}` means the receiver accepted it. Then check the agent counted it, which separates "accepted" from "forwarded":

```console
$ kubectl -n monitoring port-forward svc/app-alloy 12345:12345
$ curl -s localhost:12345/metrics | grep -E 'otelcol_receiver_accepted_spans|otelcol_exporter_send_failed_spans'
```

Accepted spans climbing with no send failures means the pipeline is complete. Failures on the exporter side point at the backend rather than the collector: check the endpoint and whether the backend is up ([Install Tempo or Jaeger](/docs/hetzner/apalla/observability/traces/install-tempo-or-jaeger)).

If nothing arrives at all, the usual cause is a service exporting to the backend directly, or to an address that predates this agent.