Decide where the store lives, how long you keep data, and how big it will get before you install anything, because these are cheaper to choose up front than to change once data is flowing. A few decisions shape the whole stack, and the how-to guides come after you make them.

In-cluster or central#

Two shapes work, and the choice is mostly about how many clusters you run.

Most teams start in-cluster on their first cluster and move the store central as soon as there is a second.

Retention and cardinality budgets#

Two numbers drive metric storage size: how long you keep data (retention), and how many distinct time series you collect (cardinality). Cardinality is the one that surprises people: a metric labelled with something high-churn, a pod name or a request id, multiplies into thousands of series. Set a retention window per tier (raw for days, downsampled, meaning averaged into coarser points, for months) and drop the labels you will not query on before they reach the store. Logs have the same tradeoff by volume: filter noisy journald units and export only the flow verdicts you need.

Where the store's data lives#

The store's own data sits on your cluster storage, so size it against what Syself Autopilot offers:

• Local NVMe through TopoLVM is fast and cheap and pinned to one node, which fits a Prometheus or Loki ingester, the part that takes in incoming data, that is happy to be rebuilt if its node is replaced.
• Object storage (native, or behind JuiceFS) fits the long-term tiers, Thanos, Mimir, Loki chunks, where durability matters more than latency.

See the storage model for how each behaves through node replacement, and size the volumes for your retention window plus headroom.

Self-run versus managed SaaS#

The reason to run this yourself is the reason you run your own services: the same Prometheus, Grafana, and Loki a SaaS bills per-metric and per-GB for run on servers you already pay for. The cost you take on is operating them. For a single small cluster a managed SaaS can still be cheaper once you count your time; as the fleet grows, self-run on your own hardware pulls ahead and keeps your telemetry in your account.

A reference stack for a fleet#

For an agency running many client clusters, a common shape is: the two Alloy collectors in each client cluster, remote-writing to one multi-tenant store (Mimir for metrics, Loki for logs, Tempo for traces) with a per-client tenant label, one Grafana over all of it behind SSO, and central Alertmanager routing per client. Each client gets isolation; you get one pane over the fleet. Multi-cluster observability puts the pieces together.

The collection tier is the part to get right before the store, because it is what decides whether a single misbehaving workload can cost you a node's telemetry. Using Alloy for observability covers that split.