Set up Prometheus

Prometheus is the store: it holds the samples and answers the queries. It does not scrape anything here. The two Alloy agents in Collection do the collecting and write into it over remote-write, so what you install on this page is a Prometheus that receives.

The Prometheus Operator gets you there. It watches the cluster for objects like Prometheus and PrometheusRule and turns them into running processes and loaded rules. The Prometheus object is your store. Everything else in the stack is a separate install with its own page, so you add the pieces you want and skip the ones you do not.

flowchart LR
  subgraph objects["Objects you apply"]
    pobj["Prometheus"]
    rules["PrometheusRule"]
    sm["ServiceMonitor and PodMonitor<br/>written next to each application"]
  end
  pobj --> op["Prometheus Operator"]
  rules --> op
  op -->|"renders the config, reloads the rules"| prom["Prometheus<br/>remote-write receiver and query API"]
  sm -.->|"read from the API, no operator in the path"| alloy["System Alloy and Application Alloy"]
  alloy -->|"remote-write"| prom
  prom -->|"firing alerts"| am["Alertmanager"]
  graf["Grafana"] -->|"queries"| prom

Objects on the left, workloads on the right, and no arrow from Prometheus to a scrape target: the agents collect, this Prometheus receives. That is why its targets page stays empty on a working install. The dotted line is the one path the operator sits outside of, because both agents read ServiceMonitor and PodMonitor objects from the Kubernetes API themselves.

What each piece does#

• The operator reconciles objects into workloads: a Prometheus becomes a StatefulSet, an Alertmanager becomes a running Alertmanager, a PrometheusRule becomes a loaded rule file. It scrapes nothing itself.
• Prometheus stores what both agents write, evaluates your alert rules, and answers queries from Grafana.
• The rest, each on its own page: Alertmanager for notifications, Grafana for dashboards, kube-state-metrics for object state, node-exporter for per-node hardware.

Why the operator, not a config file#

The alternative is a plain StatefulSet with a prometheus.yml in a ConfigMap. It works, and it keeps everything in one file you can open. The price is that every service which wants monitoring becomes an edit to that file, a review of that edit, and a reload, while the StatefulSet, the volume claim, the Service, and the reload plumbing stay yours to maintain.

The operator moves that into objects. Apply a Prometheus and it renders the config and builds the StatefulSet to match; apply a PrometheusRule and the rule lands as a file and reloads with no restart. Changing retention or storage becomes an edit to one field instead of an edit to a config file plus the workload that mounts it.

The CRDs are worth more here than the controller. Both Alloy agents read ServiceMonitor and PodMonitor objects straight from the Kubernetes API, so a team ships one next to its application and collection starts, with no central edit and no operator in that path (Deploy the Application Alloy ). Those objects are the interface your teams write against, and they keep working unchanged if you later move the store to Mimir.

The cost is a controller and its CRDs to keep upgraded, and one step of indirection when something surprises you: the config Prometheus runs is the one the operator rendered, not one you wrote. Read it back from the Secret the operator writes:

		$ kubectl -n monitoring get secret prometheus-main \
  -o jsonpath='{.data.prometheus\.yaml\.gz}' | base64 -d | gunzip
global:
  scrape_interval: 30s
  external_labels:
    cluster: <cluster-name>
    prometheus: monitoring/main
    prometheus_replica: $(POD_NAME)
  evaluation_interval: 30s
rule_files:
- /etc/prometheus/rules/prometheus-main-rulefiles-0/*.yaml
scrape_configs: []
...
	

The empty scrape_configs in that output is the design working: the selectors in the object below match nothing, so the store gets no targets of its own.

Install the operator#

The project ships a single manifest, bundle.yaml, holding the CRDs, the controller, and the RBAC it needs. It targets the default namespace, so use kustomize to put it in monitoring:

		$ kubectl create namespace monitoring
$ curl -sLO https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/refs/tags/<operator-version>/bundle.yaml
$ curl -sLO https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/refs/tags/<operator-version>/kustomization.yaml
$ kustomize edit set namespace monitoring
$ kubectl create -k .
	

Pin <operator-version> to a release tag rather than tracking the newest one, so an update does not change behavior under you. Use kubectl create -k . or kubectl apply --server-side -k . rather than client-side kubectl apply, because CRD definitions in the operator bundle exceed the 256KB Kubernetes annotation limit.

		$ kubectl -n monitoring get deploy prometheus-operator
NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
prometheus-operator   1/1     1            1           40s
	

Deploy the store#

		apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  namespace: monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: prometheus
  namespace: monitoring
rules:
  - apiGroups: [""]
    resources: ["services", "endpoints", "pods"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: prometheus
  namespace: monitoring
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: prometheus
subjects:
  - kind: ServiceAccount
    name: prometheus
    namespace: monitoring
	

		apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: main
  namespace: monitoring
spec:
  serviceAccountName: prometheus
  replicas: 1
  retention: 15d
  # Both Alloy agents push their samples in, so accept remote-write.
  enableRemoteWriteReceiver: true
  # Point monitor selectors at a non-matching label so this Prometheus
  # scrapes nothing and never competes with the agents over targets, while
  # ensuring the operator renders a full config and loads rule files.
  serviceMonitorSelector:
    matchLabels:
      scrape-by-prometheus: "true"
  podMonitorSelector:
    matchLabels:
      scrape-by-prometheus: "true"
  probeSelector:
    matchLabels:
      scrape-by-prometheus: "true"
  scrapeConfigSelector:
    matchLabels:
      scrape-by-prometheus: "true"
  # Rules are the exception: load every PrometheusRule labelled
  # prometheus: main, in any namespace.
  ruleSelector:
    matchLabels:
      prometheus: main
  ruleNamespaceSelector: {}
  # Stamp every series with the cluster it came from, so one store can tell
  # your clusters apart.
  externalLabels:
    cluster: <cluster-name>
  # Where firing alerts go. The operator creates this Service along with the
  # Alertmanager on the alerting page.
  alerting:
    alertmanagers:
      - namespace: monitoring
        name: alertmanager-operated
        port: web
  # standard volumes provision only on cloud nodes, so keep the store there.
  # change this if you point storageClassName at a local class instead.
  nodeSelector:
    autopilot.syself.com/machine-type: hcloud
  # The operator sets no pod securityContext of its own, and a fresh volume
  # belongs to root, so without fsGroup Prometheus cannot write its database.
  securityContext:
    runAsUser: 1000
    runAsGroup: 2000
    fsGroup: 2000
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  storage:
    volumeClaimTemplate:
      spec:
        # standard = Hetzner Cloud volumes, cloud nodes only. The nodeSelector
        # above keeps this on a cloud node. To run on a bare-metal node instead,
        # drop it and name the local class matching that node's disks.
        # (local-nvme, local-ssd, or local-hdd through TopoLVM).
        storageClassName: standard
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 20Gi
	

		$ kubectl apply -f prometheus-rbac.yaml -f prometheus.yaml
$ kubectl -n monitoring rollout status statefulset/prometheus-main
	

		$ kubectl -n monitoring get prometheus main \
  -o jsonpath='{.status.conditions[?(@.type=="Reconciled")].message}'
No ServiceMonitor, PodMonitor, Probe, ScrapeConfig, and PrometheusRule have been selected.
	

		$ kubectl -n monitoring port-forward svc/prometheus-operated 9090:9090
	

		count(up{job="etcd"}) by (instance)
	

What this setup covers#

What you have now is a working store, not a hardened one. Here is what it gives you and where each part stops.

Making it highly available#

Raising replicas on the object does not do it. Both agents write to prometheus-operated, a headless Service, so each write connection lands on one pod and neither replica ends up with the whole set. A second replica gives you two partial stores rather than a spare.

Availability on a receiving store means a store built to replicate. Point the agents' remote-write at Mimir or Victoria Metrics, keep this Prometheus for rules and dashboards, and the durable copy lives somewhere that survives losing a pod (Long-term storage and remote-write ). Alertmanager is the opposite case: its replicas gossip and de-duplicate, so running more than one there does buy redundancy (Set up Alertmanager ).

Sizing it#

Memory tracks active series, and disk is your retention window multiplied by the rate samples arrive. Both inputs are series this Prometheus keeps about itself, so measure them on your own workload instead of guessing:

		prometheus_tsdb_head_series
sum(rate(prometheus_tsdb_head_samples_appended_total[5m]))
prometheus_tsdb_storage_blocks_bytes
	

The first is how many series are live right now, the second how fast samples arrive, the third what the compacted blocks on disk cost so far. Let it run a day, then multiply that third number by your retention in days and add headroom for compaction. That is your volume size, measured rather than assumed. Watch the series count for memory, because it is the one that grows on you: a single label with high churn multiplies into thousands of series.

The storage: 20Gi and retention: 15d in the object above are a starting point, not a sized figure. They hold the scrape set these guides set up on a small cluster and nothing more. Plan your observability stack covers the retention and cardinality budgets behind those numbers, and Long-term storage and remote-write covers history beyond what one volume holds.

Before you call it production#

This store is fit for live troubleshooting and for alerting on one cluster. What stands between it and something you lean on during an incident is the list below, and none of it is in the object above.

Bound its resources. The store's container requests nothing and has no limit, so the scheduler places it without accounting for the memory it will use, and a container with no request is among the first reclaimed when the node comes under memory pressure. Take the series count you measured above, then set requests to match what the process actually uses:

		spec:
  resources:
    requests:
      cpu: <measured-cpu>
      memory: <measured-memory>
    limits:
      memory: <measured-memory-plus-headroom>
	

Set the memory limit and leave CPU unlimited. A memory limit turns a cardinality spike into one restarted pod instead of a node that runs out of memory under every workload on it, while a CPU limit throttles the store exactly when someone is querying it during an incident.

Cap retention by size as well as time. retention: 15d alone means a cardinality spike fills the volume before the window expires, and a full disk stops the store from accepting writes. Add a size cap below the volume size so the oldest blocks drop instead:

		spec:
  retention: 15d
  retentionSize: 15GB
	

Keep the gap. The cap governs the persistent blocks, while the write-ahead log and compaction need room on the same volume.

Watch the store itself. Nothing here alerts on it. The PrometheusRule objects covering the stack's own health arrived with kube-prometheus-stack, and installing the operator on its own leaves them out, so a store that stops receiving is silent rather than loud. Set up Alertmanager lists the series to alert on, remote-write failures first.

Expect it to move. With one replica, a node replacement takes the store down for as long as a reschedule takes. Nothing is lost, because the volume follows the pod and the agents' write-ahead logs cover the gap. Resist pinning it up with a PodDisruptionBudget that keeps the single replica running: on a platform that replaces nodes, that blocks the drain rather than protecting your data.

With the store up, put the data on a dashboard (Set up Grafana ) and alert on the node integrity conditions (Platform alert rules ).