Most collection problems here are one of a handful, and several of them are quiet: no error in the log, just data that never arrives or arrives twice. Start from Alloy's own UI, which lists every component, its health, and the targets it resolved:

```console
$ kubectl -n monitoring port-forward daemonset/system-alloy 12345:12345
```

The agent also publishes metrics about itself on the same port, which is how you tell "scraping and forwarding" from "running but collecting nothing":

```console
$ curl -s localhost:12345/metrics | grep prometheus_remote_write_wal_samples_appended_total
```

A number that climbs means scrapes are landing. A flat zero means nothing is being collected, whatever the component health says.

## A target is down

Check the component's target list in the UI first, then decide which half is wrong: the component resolved no target at all, or it resolved one that will not answer.

If it resolved a target and the scrape fails, confirm the component is actually listening on the node before touching the config. From a node shell or a `hostNetwork` debug pod:

```console
$ ss -tlnp | grep -E '2381|10257|10259|10250|9962|9965|9189|20257'
```

A `401` means the API server rejected the ServiceAccount token. A `403` means the token authenticated but lacks permission, which for the kubelet and cadvisor jobs is the `nodes/metrics` resource rather than the `/metrics` URL. A TLS error on kube-controller-manager, kube-scheduler, or the kubelet means `insecure_skip_verify` is missing: those certificates cannot be verified from loopback, and the connection never leaves the node.

## `connection refused`, but only on some nodes

This is the role-gating failure, and it is the most common one on this platform. etcd, kube-controller-manager, kube-scheduler, KubeGate, and `syself-tunnel-server` exist only on control planes; `syself-proxy` only on workers; the CSI node plugin only on cloud workers. A job with a static target list runs everywhere, so on every node that lacks the component it fails forever and holds a `TargetDown` alert open, which trains everyone to ignore it.

The fix is not a longer `for:` on the alert. Gate the job on the node's role, as [Deploy the System Alloy](/docs/hetzner/apalla/observability/collection/deploy-the-system-alloy) does. Two details matter when you write that gate: control-plane nodes carry `node-role.kubernetes.io/control-plane` with an **empty value**, so test whether the label is present rather than comparing its value, and workers carry `node.kubernetes.io/worker=true`. A gate that matches on value alone silently keeps every node, because an absent label and an empty label look the same to a relabel rule.

## The same series arrives twice

These are two different causes with the same symptom: doubled rates and a counter that appears to jump.

**A pod that declares more than one port.** Kubernetes pod discovery emits one target per container port, so `cilium-operator` (`health` and `prometheus`), `hubble-relay` (`grpc` and `prometheus`), and the CSI controller (`healthz` and `metrics`) each produce two targets. If the relabel rules then rewrite the address to a fixed metrics port, both targets become the same endpoint and it is scraped twice. Keep the metrics port explicitly:

```terraform
rule {
  source_labels = ["__meta_kubernetes_pod_container_port_name"]
  regex         = "prometheus"
  action        = "keep"
}
```

**Clustering that is not on.** On the Application Alloy, every replica scrapes every target unless the discovering component has a `clustering` block. Enabling `alloy.clustering.enabled` in the chart forms the cluster but does not shard anything by itself. Confirm target distribution across replicas in the Alloy UI (`http://localhost:12345`) or under `prometheus.operator.servicemonitors.apps`. Note that counting target instances (`count(count by (instance) (up{job="my-app"}))`) measures pod targets, not collector replica counts; when sharding is broken, all replicas scrape identical targets, causing sample drops and out-of-order errors at the receiver.

## Samples rejected as out of order

Every System Alloy pod dials the same `127.0.0.1:<port>`, so without a per-node identity every node writes the same series. The receiver interleaves samples from many agents into one stream and rejects the overlap: `err-mimir-sample-out-of-order` on Grafana Mimir, `out of order sample` on Prometheus. It looks like a backend problem and is a collection problem.

Stamp the node name onto every target, from the Downward API, as `instance` and `node`. Then confirm each node has its own series:

```text
count(up{job="etcd"}) by (instance)
```

One row per control-plane node is right. A single row is the bug.

## Container logs never arrive

Metrics work, the audit files stream, and `/var/log/pods` produces nothing. The cause is almost always that a glob was handed to something that does not expand globs: `loki.source.file` needs concrete paths, and `local.file_match` is what turns `/var/log/pods/*<uid>/<container>/*.log` into real files. Chain them in that order. Check which files the agent has actually opened:

```console
$ curl -s localhost:12345/metrics | grep loki_source_file_file_bytes_total
```

If journald is the missing half instead, check whether systemd on that node writes to `/run/log/journal` rather than `/var/log/journal`, and mount both.

## Audit logs are empty but the files exist

The audit files are mode `0600` and owned by root, so a collector that drops to a non-root user starts cleanly, finds the files, and reads nothing. The System Alloy runs as root for this reason. On worker nodes the two API-side audit paths are genuinely absent, and an agent finding nothing there is correct, not a fault.

## The write-ahead log grows and memory climbs

The agent is collecting faster than the backend accepts, so samples pile up on disk and in memory. Watch the queue rather than guessing:

```console
$ curl -s localhost:12345/metrics | grep -E 'prometheus_remote_storage_samples_pending|prometheus_remote_write_wal_storage_active_series'
```

A backend that is slow or refusing writes is the usual cause, and the WAL is doing its job by absorbing it. Sustained growth is different: it means the ingest rate exceeds what the backend will ever take, and the answer is fewer series, not a bigger disk.

## A cardinality spike

Active series climb steeply, the agent's memory follows, and eventually it is killed and restarted. Find the source before changing any limits:

```text
topk(10, count by (__name__) ({__name__=~".+"}))
```

A metric labelled with something high-churn (a request id, a pod name, a user id) is nearly always what you find. Drop the label at the source with `metricRelabelings` on the workload's `ServiceMonitor`, which is where the fix belongs and where the team that owns the metric will look.

This is the failure the [two-Alloy split](/docs/hetzner/apalla/observability/collection/using-alloy-for-observability) exists to contain. When it happens on the Application Alloy, node telemetry keeps flowing and you still have the kubelet, etcd, and log streams you need to investigate. That property only holds while application scrape jobs stay off the DaemonSet, so when the clustered agent is under pressure, add a replica rather than moving jobs onto the nodes.