Decide three things before you install anything: where the store lives, how long you keep data, and how big it will get. Changing them later, once data is flowing, is much harder than choosing up front. The how-to guides come after these decisions.

In-cluster or central

Two shapes work, and the choice is mostly about how many clusters you run.

Most teams start in-cluster on their first cluster and move the store central as soon as there is a second.

Retention and cardinality budgets

Two numbers drive metric storage size: how long you keep data (retention), and how many distinct time series you collect (cardinality). Cardinality is the one that surprises people. A metric labelled with something high-churn, a pod name or a request id, multiplies into thousands of series.

Set a retention window per tier: raw data for days, downsampled data for months. Drop the labels you will not query on before they reach the store. Logs have the same tradeoff by volume, so filter noisy journald units and export only the flow verdicts you need.

Where the store's data lives

The store's own data sits on your cluster storage, so size it against what Syself Autopilot offers:

• Local NVMe through TopoLVM is fast and cheap, and pinned to one node. It fits a Prometheus or Loki ingester, the part that takes in incoming data, which can be rebuilt when its node is replaced.
• Object storage (native, or behind JuiceFS) fits the long-term tiers, Thanos, Mimir, Loki chunks, where durability matters more than latency.

See the storage model for how each behaves through node replacement, and size the volumes for your retention window plus headroom.

Self-run versus managed SaaS

Running the stack yourself costs you the operating effort and nothing per metric or per GB. For a single small cluster a managed SaaS can still be cheaper once you count your time. As the fleet grows, self-run on your own hardware pulls ahead, and the telemetry stays in your account.

A reference stack for a fleet

For an agency running many client clusters, a common shape has three layers:

• In each client cluster: two Alloy collectors gather telemetry and remote-write it to one central store, tagged with that client's tenant label.
• In the central store: Mimir holds the metrics, Loki the logs, and Tempo the traces. The tenant label keeps each client's data separate.
• On top of it: one Grafana behind SSO gives you a view across every client, and a central Alertmanager routes each client's alerts.

Each client stays isolated, and you get one view across the whole fleet. Multi-cluster observability puts the pieces together.

The collection tier is the part to get right before the store, because it is what decides whether a single misbehaving workload can cost you a node's telemetry. Using Alloy for observability covers that split.