Long-term storage and remote-write

A single Prometheus keeps a few weeks of data on its own volume and then drops it. That is fine for live troubleshooting and useless for a quarter-over-quarter capacity trend or a post-incident review from last month. To keep history, and to query many clusters at once, remote-write to a store built for it: Thanos, Grafana Mimir, or Victoria Metrics. Remote-write means shipping your metrics to an external store as Prometheus collects them, so the durable copy lives outside the cluster.

Why local retention is not enough#

Prometheus stores samples on one volume, sized for a retention window measured in days to weeks. Push it longer and the disk grows without bound, queries slow down, and the only copy of your history stays inside the cluster it describes. A local class pins that volume to its node as well, so the data stays behind when the pod moves. Long-term storage moves the durable copy off the store and onto object storage, where it survives and scales independently.

remote-write#

Both Alloy collectors write their samples out as they scrape, so pointing them at a long-term store is a change to one block in each config rather than a new component. Change the endpoint and the durable copy moves:

		prometheus.remote_write "central" {
  endpoint {
    url = "https://mimir.example.com/api/v1/push"
 
    basic_auth {
      username = "tenant-id"
      password_file = "/etc/alloy/secrets/remote-write-token"
    }
  }
}
	

Keep the credential in a Secret mounted into the agent, never inline in the values file. Because both agents already buffer to a write-ahead log, a store that goes away briefly costs latency rather than data.

Pick a backend#

Each of these solutions persists historical data to object storage. Syself Autopilot supports this using either a native object-storage bucket or JuiceFS over object storage . Be sure to provision enough bucket capacity to accommodate your required data retention period.

Downsampling and retention tiers#

You rarely need per-second resolution from six months ago. Keep tiers: raw samples for days, then downsampled (5-minute, then 1-hour) rollups for months or years. Thanos and Mimir both downsample automatically; set the retention per tier so recent data stays sharp and old data stays cheap.

Query the whole fleet#

For a fleet, the result is one query surface. Each cluster remote-writes to the central store with an external label identifying it (cluster="prod-eu"), and one Grafana queries all of them. That external label is what keeps a client's series separate and lets you compare clusters side by side, and it has to be set on both agents in every cluster or half the telemetry arrives unattributed. See Multi-cluster observability .