Running one cluster's observability in that cluster works. Running ten that way means ten Grafanas and no single view. The fleet shape is the opposite: each cluster collects locally and ships to one central store, and you run the fleet from one Grafana, one long-term store, and one alerting plane. This is how an agency operates many client clusters without logging into each one.

Ship from each cluster to the center#

Each cluster runs its own two Alloy collectors , but instead of storing everything locally, they forward:

• Metrics via remote-write , which ships samples to a central store instead of only keeping them locally.
• Logs to a central Loki, from the System Alloy that already tails them.
• Traces to a central Tempo, from the Application Alloy that receives them.

Only the destinations differ per cluster, so the collection config is the same everywhere except for the label that says which cluster it is.

The clusters stay independent (a central outage does not stop local self-healing), and the durable copy lives in one place that outlives any single cluster.

A multi-tenant central store#

A tenant is one cluster's (or one client's) slice of the store, kept separate from every other. Use stores built to hold many tenants side by side:

• Grafana Mimir for metrics: horizontally scalable, with a tenant per cluster or per client.
• Loki for logs: the same tenant model (X-Scope-OrgID) keeps each client's logs separate.

One Grafana queries both, so the whole fleet is visible from one view, with a cluster variable to focus on one at a time.

External labels are what keep clusters apart#

The linchpin is a unique external label on every cluster's data. Set it on both collectors in each cluster (cluster="client-acme"), along with the matching tenant on their log and trace writes, so a series, a log line, or a span always says which cluster it came from. That label is what lets one Grafana separate clusters, one Alertmanager route per client, and one query compare them. If two clusters share a label, their data merges and cannot be separated afterward, so set it when you build the cluster.

Central alert routing#

Point every cluster's alerts at one Alertmanager and route on the cluster label, so cluster="client-acme" pages Acme's channel and cluster="client-beta" pages Beta's, from one config. See Alert routing and receivers .

GitOps the stack per cluster#

A fleet's observability config has to be identical across clusters, or the data does not line up. Keep both collectors' values files, the external labels, and the dashboards in Git and roll them to every cluster with Argo CD , templated so each cluster gets its own cluster label and otherwise the same config. One reviewed change rolls out everywhere, which is also how you avoid the fleet drifting into per-cluster collection quirks that make a cross-cluster query meaningless.

Tenant isolation for clients#

For an agency, isolation is essential: each client sees their own cluster's metrics and logs, and never another client's. Enforce it end to end, a tenant per client in Mimir and Loki, per-client dashboards behind SSO, and per-client alert routing, so one client's view can never reach another's data. Clusters separated per client (rather than namespaces in a shared cluster) make that isolation the default.