Set up Prometheus

On this page you install Prometheus as the store for your metrics. It keeps the samples and answers queries, but it does not scrape anything itself. The two Alloy agents from Collection do the collecting and push their data in over remote-write.

The install goes through the Prometheus Operator. You apply objects like Prometheus and PrometheusRule, and the operator turns them into a running Prometheus and loaded rules. Everything else in the stack has its own page, so you add the pieces you want and skip the ones you do not.

flowchart LR
  subgraph objects["Objects you apply"]
    pobj["Prometheus"]
    rules["PrometheusRule"]
    sm["ServiceMonitor and PodMonitor<br/>written next to each application"]
  end
  pobj --> op["Prometheus Operator"]
  rules --> op
  op -->|"renders the config, reloads the rules"| prom["Prometheus<br/>remote-write receiver and query API"]
  sm -.->|"read from the API, no operator in the path"| alloy["System Alloy and Application Alloy"]
  alloy -->|"remote-write"| prom
  prom -->|"firing alerts"| am["Alertmanager"]
  graf["Grafana"] -->|"queries"| prom

Objects on the left, workloads on the right, and no arrow from Prometheus to a scrape target: the agents collect, this Prometheus receives. That is why its targets page stays empty on a working install. The dotted line is the one path the operator sits outside of, because both agents read ServiceMonitor and PodMonitor objects from the Kubernetes API themselves.

What each piece does

• The operator reconciles objects into workloads: a Prometheus becomes a StatefulSet, an Alertmanager becomes a running Alertmanager, a PrometheusRule becomes a loaded rule file. It scrapes nothing itself.
• Prometheus stores what both agents write, evaluates your alert rules, and answers queries from Grafana.
• The rest, each on its own page: Alertmanager for notifications, Grafana for dashboards, kube-state-metrics for object state, node-exporter for per-node hardware.

This setup differs from a plain Prometheus install. There, every service that wants monitoring means an edit to a central config file and a reload, and the StatefulSet, the volume, and the reload plumbing stay yours to maintain. The operator takes over that maintenance and replaces the central file with objects: teams ship a ServiceMonitor or PodMonitor next to their application, and both Alloy agents pick it up from the Kubernetes API with no central edit (Deploy the Application Alloy ).

Install the operator

The project ships a single manifest, bundle.yaml, holding the CRDs, the controller, and the RBAC it needs. It targets the default namespace, so use kustomize to put it in monitoring:

		$ kubectl create namespace monitoring
$ curl -sLO https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/refs/tags/<operator-version>/bundle.yaml
$ curl -sLO https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/refs/tags/<operator-version>/kustomization.yaml
$ kustomize edit set namespace monitoring
$ kubectl create -k .
	

Pin <operator-version> to a release tag rather than tracking the newest one, so an update does not change behavior under you. Use kubectl create -k . or kubectl apply --server-side -k . rather than client-side kubectl apply, because CRD definitions in the operator bundle exceed the 256KB Kubernetes annotation limit.

		$ kubectl -n monitoring get deploy prometheus-operator
NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
prometheus-operator   1/1     1            1           40s
	

Deploy the store

		$ kubectl apply -f prometheus-rbac.yaml -f prometheus.yaml
$ kubectl -n monitoring rollout status statefulset/prometheus-main
	

		$ kubectl -n monitoring get prometheus main \
  -o jsonpath='{.status.conditions[?(@.type=="Reconciled")].message}'
No ServiceMonitor, PodMonitor, Probe, ScrapeConfig, and PrometheusRule have been selected.
	

		$ kubectl -n monitoring port-forward svc/prometheus-operated 9090:9090
	

		count(up{job="etcd"}) by (instance)
	

What this setup covers

What you have now is a working store, not a hardened one. The receiver takes plain HTTP with no authentication or TLS, and the query API is open to every pod in the cluster, so anything that reaches the pod on port 9090 can write to or read from your store. Add a default-deny policy for the namespace (Segment with network policies ) and keep Grafana as the query surface that carries the login (Set up Grafana ).

The other limits have their own pages: the database is one volume holding one retention window (Long-term storage and remote-write ), and it stays one store per cluster (Multi-cluster observability ).

Why only one replica

The prometheus.yaml in step 2 sets replicas: 1, and that is deliberate. Raising the number does not add a backup: each replica keeps its own database on its own volume, and the pushes from the agents get split between the replicas. Instead of one complete store plus a replica as a spare, you get multiple replicas with the data divided among them, so only all of them together hold the complete set.

For real redundancy, point the agents' remote-write at a store built to replicate, like Mimir or Victoria Metrics, and keep this Prometheus for rules and dashboards (Long-term storage and remote-write ).

Sizing it

How much memory and disk this Prometheus needs depends on your workload: memory grows with the number of active series, and disk grows with how fast samples arrive and how long you keep them. Prometheus publishes these numbers about itself, so measure them instead of guessing:

		prometheus_tsdb_head_series
sum(rate(prometheus_tsdb_head_samples_appended_total[5m]))
prometheus_tsdb_storage_blocks_bytes
	

The first is how many series are live right now, the second how fast samples arrive and the third what the blocks on disk cost so far. Let it run a day, then multiply the disk number by your retention in days for the volume size, and watch the series count for memory.

The storage: 20Gi and retention: 15d in prometheus.yaml are a starting point, not a sized figure. They are enough only for the metrics the two Alloy agents collect on a small cluster. Plan your observability stack covers the retention and cardinality budgets behind those numbers, and Long-term storage and remote-write covers history beyond what one volume holds.

Before you call it production

This store is fine for live troubleshooting and for alerting on one cluster. Before you depend on it during an incident, work through the list below. None of it is part of the prometheus.yaml from step 2.

Bound its resources. The prometheus.yaml from step 2 sets no resource requests or limits. Set the requests to what the process actually uses, taken from the numbers you measured in the Sizing it section, and add a memory limit:

		spec:
  resources:
    requests:
      cpu: <measured-cpu>
      memory: <measured-memory>
    limits:
      memory: <measured-memory-plus-headroom>
	

Leave CPU unlimited: a CPU limit throttles queries exactly when you need them during an incident, while a memory limit turns a cardinality spike into one restarted pod instead of a starving node.

Cap retention by size as well as time. With retention: 15d alone, a spike in series can fill the volume before 15 days pass, and a full disk stops the store from accepting writes. Add a size cap below the volume size so the oldest data drops first:

		spec:
  retention: 15d
  retentionSize: 15GB
	

Watch the store itself. Nothing in this setup alerts when the store stops receiving. Set up Alertmanager lists the series to alert on, remote-write failures first.

Expect it to move. A node replacement takes the single replica down until the pod reschedules. Nothing is lost: the volume follows the pod and the agents buffer the gap.

With the store up, put the data on a dashboard (Set up Grafana ) and alert on the node integrity conditions (Platform alert rules ).