Long-term storage and remote-write

A single Prometheus keeps a few weeks of data on its own volume and then drops it. That is fine for live troubleshooting, but not for a capacity trend or a post-incident review from last month. To keep history, and to query many clusters at once, remote-write to a store built for it: Thanos, Grafana Mimir, or Victoria Metrics. Remote-write means shipping your metrics to an external store as they are collected, so the durable copy lives outside the cluster.

Raising retention on the store is not a substitute. The disk grows without bound, queries slow down, and the only copy of your history stays inside the cluster it describes. On a local storage class that copy is pinned to one node besides. Long-term storage moves the durable copy onto object storage, where it survives and scales independently.

remote-write

Both Alloy collectors write their samples out as they scrape, so pointing them at a long-term store is a change to one block in each config rather than a new component. Change the endpoint and the durable copy moves:

		prometheus.remote_write "central" {
  endpoint {
    url = "https://mimir.example.com/api/v1/push"
 
    basic_auth {
      username = "tenant-id"
      password_file = "/etc/alloy/secrets/remote-write-token"
    }
  }
}
	

Keep the credential in a Secret mounted into the agent, never inline in the values file. Because both agents already buffer to a write-ahead log, a store that goes away briefly costs latency rather than data.

Pick a backend

All three store their history in an S3-compatible object-storage bucket. The bucket grows with your retention window, so retention is what drives the storage cost.

Downsampling and retention tiers

You rarely need per-second resolution from six months ago. Keep tiers: raw samples for days, then downsampled (5-minute, then 1-hour) rollups for months or years. Thanos builds those rollups automatically, and you set the retention per tier so recent data stays sharp and old data stays cheap. Mimir does not downsample: there, use recording rules to keep cheaper aggregate series for the long window.

Query the whole fleet

For a fleet, the result is one query surface. Each cluster remote-writes to the central store with an external label identifying it (cluster="prod-eu"), and one Grafana queries all of them. That external label keeps each cluster's series separate and lets you compare clusters side by side. Set it on both agents in every cluster, otherwise half the telemetry arrives without a cluster name. See Multi-cluster observability .