Prometheus is the store: it holds the samples and answers the queries. It does not scrape anything here. The two Alloy agents in [Collection](/docs/hetzner/apalla/observability/collection/using-alloy-for-observability) do the collecting and write into it over remote-write, so what you install on this page is a Prometheus that receives. The Prometheus Operator gets you there. It watches the cluster for objects like `Prometheus` and `PrometheusRule` and turns them into running processes and loaded rules. The `Prometheus` object is your store. Everything else in the stack is a separate install with its own page, so you add the pieces you want and skip the ones you do not. ```mermaid flowchart LR subgraph objects["Objects you apply"] pobj["Prometheus"] rules["PrometheusRule"] sm["ServiceMonitor and PodMonitor
written next to each application"] end pobj --> op["Prometheus Operator"] rules --> op op -->|"renders the config, reloads the rules"| prom["Prometheus
remote-write receiver and query API"] sm -.->|"read from the API, no operator in the path"| alloy["System Alloy and Application Alloy"] alloy -->|"remote-write"| prom prom -->|"firing alerts"| am["Alertmanager"] graf["Grafana"] -->|"queries"| prom ``` Objects on the left, workloads on the right, and no arrow from Prometheus to a scrape target: the agents collect, this Prometheus receives. That is why its targets page stays empty on a working install. The dotted line is the one path the operator sits outside of, because both agents read `ServiceMonitor` and `PodMonitor` objects from the Kubernetes API themselves. ## What each piece does - **The operator** reconciles objects into workloads: a `Prometheus` becomes a StatefulSet, an `Alertmanager` becomes a running Alertmanager, a `PrometheusRule` becomes a loaded rule file. It scrapes nothing itself. - **Prometheus** stores what both agents write, evaluates your alert rules, and answers queries from Grafana. - **The rest, each on its own page:** [Alertmanager](/docs/hetzner/apalla/observability/alerting/set-up-alertmanager) for notifications, [Grafana](/docs/hetzner/apalla/observability/dashboards/set-up-grafana) for dashboards, [kube-state-metrics](/docs/hetzner/apalla/observability/metrics/kube-state-metrics) for object state, [node-exporter](/docs/hetzner/apalla/observability/metrics/node-and-hardware-metrics) for per-node hardware. ## Why the operator, not a config file The alternative is a plain StatefulSet with a `prometheus.yml` in a ConfigMap. It works, and it keeps everything in one file you can open. The price is that every service which wants monitoring becomes an edit to that file, a review of that edit, and a reload, while the StatefulSet, the volume claim, the Service, and the reload plumbing stay yours to maintain. The operator moves that into objects. Apply a `Prometheus` and it renders the config and builds the StatefulSet to match; apply a `PrometheusRule` and the rule lands as a file and reloads with no restart. Changing retention or storage becomes an edit to one field instead of an edit to a config file plus the workload that mounts it. The CRDs are worth more here than the controller. Both Alloy agents read `ServiceMonitor` and `PodMonitor` objects straight from the Kubernetes API, so a team ships one next to its application and collection starts, with no central edit and no operator in that path ([Deploy the Application Alloy](/docs/hetzner/apalla/observability/collection/deploy-the-application-alloy)). Those objects are the interface your teams write against, and they keep working unchanged if you later move the store to Mimir. The cost is a controller and its CRDs to keep upgraded, and one step of indirection when something surprises you: the config Prometheus runs is the one the operator rendered, not one you wrote. Read it back from the Secret the operator writes: ```console $ kubectl -n monitoring get secret prometheus-main \ -o jsonpath='{.data.prometheus\.yaml\.gz}' | base64 -d | gunzip global: scrape_interval: 30s external_labels: cluster: prometheus: monitoring/main prometheus_replica: $(POD_NAME) evaluation_interval: 30s rule_files: - /etc/prometheus/rules/prometheus-main-rulefiles-0/*.yaml scrape_configs: [] ... ``` The empty `scrape_configs` in that output is the design working: the selectors in the object below match nothing, so the store gets no targets of its own. ## Install the operator The project ships a single manifest, `bundle.yaml`, holding the CRDs, the controller, and the RBAC it needs. It targets the `default` namespace, so use kustomize to put it in `monitoring`: ```console $ kubectl create namespace monitoring $ curl -sLO https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/refs/tags//bundle.yaml $ curl -sLO https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/refs/tags//kustomization.yaml $ kustomize edit set namespace monitoring $ kubectl create -k . ``` Pin `` to a release tag rather than tracking the newest one, so an update does not change behavior under you. Use `kubectl create -k .` or `kubectl apply --server-side -k .` rather than client-side `kubectl apply`, because CRD definitions in the operator bundle exceed the 256KB Kubernetes annotation limit. ```console $ kubectl -n monitoring get deploy prometheus-operator NAME READY UP-TO-DATE AVAILABLE AGE prometheus-operator 1/1 1 1 40s ``` ## Deploy the store Give it a ServiceAccount and namespace read access A store that scrapes nothing needs no cluster-wide read access. It does do one piece of discovery, finding the Alertmanager pods to send alerts to, and that needs services, endpoints, and pods in its own namespace: ```yaml title="prometheus-rbac.yaml" apiVersion: v1 kind: ServiceAccount metadata: name: prometheus namespace: monitoring --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: prometheus namespace: monitoring rules: - apiGroups: [""] resources: ["services", "endpoints", "pods"] verbs: ["get", "list", "watch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: prometheus namespace: monitoring roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: prometheus subjects: - kind: ServiceAccount name: prometheus namespace: monitoring ``` Leave the Role out and Prometheus still starts, but its log fills with `endpoints is forbidden` and it never finds Alertmanager. Write the Prometheus object ```yaml title="prometheus.yaml" apiVersion: monitoring.coreos.com/v1 kind: Prometheus metadata: name: main namespace: monitoring spec: serviceAccountName: prometheus replicas: 1 retention: 15d # Both Alloy agents push their samples in, so accept remote-write. enableRemoteWriteReceiver: true # Point monitor selectors at a non-matching label so this Prometheus # scrapes nothing and never competes with the agents over targets, while # ensuring the operator renders a full config and loads rule files. serviceMonitorSelector: matchLabels: scrape-by-prometheus: "true" podMonitorSelector: matchLabels: scrape-by-prometheus: "true" probeSelector: matchLabels: scrape-by-prometheus: "true" scrapeConfigSelector: matchLabels: scrape-by-prometheus: "true" # Rules are the exception: load every PrometheusRule labelled # prometheus: main, in any namespace. ruleSelector: matchLabels: prometheus: main ruleNamespaceSelector: {} # Stamp every series with the cluster it came from, so one store can tell # your clusters apart. externalLabels: cluster: # Where firing alerts go. The operator creates this Service along with the # Alertmanager on the alerting page. alerting: alertmanagers: - namespace: monitoring name: alertmanager-operated port: web # standard volumes provision only on cloud nodes, so keep the store there. # change this if you point storageClassName at a local class instead. nodeSelector: autopilot.syself.com/machine-type: hcloud # The operator sets no pod securityContext of its own, and a fresh volume # belongs to root, so without fsGroup Prometheus cannot write its database. securityContext: runAsUser: 1000 runAsGroup: 2000 fsGroup: 2000 runAsNonRoot: true seccompProfile: type: RuntimeDefault storage: volumeClaimTemplate: spec: # standard = Hetzner Cloud volumes, cloud nodes only. The nodeSelector # above keeps this on a cloud node. To run on a bare-metal node instead, # drop it and name the local class matching that node's disks. # (local-nvme, local-ssd, or local-hdd through TopoLVM). storageClassName: standard accessModes: ["ReadWriteOnce"] resources: requests: storage: 20Gi ``` Apply it and confirm it accepts writes ```console $ kubectl apply -f prometheus-rbac.yaml -f prometheus.yaml $ kubectl -n monitoring rollout status statefulset/prometheus-main ``` The operator reports back what it selected, and selecting nothing is the goal here: ```console $ kubectl -n monitoring get prometheus main \ -o jsonpath='{.status.conditions[?(@.type=="Reconciled")].message}' No ServiceMonitor, PodMonitor, Probe, ScrapeConfig, and PrometheusRule have been selected. ``` That message is there while nothing is selected at all. It goes empty once you add rules on the alerting page, so the check that keeps holding is the empty targets page below. Deploy the collectors ([Deploy the System Alloy](/docs/hetzner/apalla/observability/collection/deploy-the-system-alloy)), then check that samples are arriving from every node rather than from one: ```console $ kubectl -n monitoring port-forward svc/prometheus-operated 9090:9090 ``` At `http://localhost:9090`, query for a series each agent produces: ```text count(up{job="etcd"}) by (instance) ``` One row per control-plane node means the pipeline works end to end, including the per-node identity that keeps each agent's series distinct. The targets page stays empty, which is expected: this Prometheus receives, it does not scrape. ## What this setup covers What you have now is a working store, not a hardened one. Here is what it gives you and where each part stops. | Aspect | What you get here | Where it stops | | ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Availability** | One replica, and both agents buffer to a write-ahead log, so a restart or a rollout costs you latency rather than samples. | Queries and rule evaluation stop while that replica restarts. | | **Write path** | Plain HTTP from the agents, inside the cluster. | No authentication and no TLS on the receiver, so anything that reaches the pod on `9090` can write samples into your store. | | **Read path** | Grafana is the query surface and carries the login ([Set up Grafana](/docs/hetzner/apalla/observability/dashboards/set-up-grafana)). | The Prometheus API itself is open to everything on the pod network. | | **Isolation** | A namespace-scoped Role with no cluster-wide read, and a non-root pod with `seccompProfile` set to `RuntimeDefault`. | Nothing restricts which pods can reach it. Add a default-deny policy ([Segment with network policies](/docs/hetzner/apalla/security/segment-with-network-policies)). | | **Durability** | `standard` is a network-attached, Ceph-replicated volume, so the database follows the pod through a node replacement ([the storage model](/docs/hetzner/apalla/concepts/internals/storage)). | One volume, in one cluster, holding one retention window, with no copy anywhere else. | | **Fleet** | The `externalLabels` entry stamps every series with its cluster, so one store can tell your clusters apart. | Still one store per cluster ([Multi-cluster observability](/docs/hetzner/apalla/observability/multi-cluster/multi-cluster-observability)). | ### Making it highly available Raising `replicas` on the object does not do it. Both agents write to `prometheus-operated`, a headless Service, so each write connection lands on one pod and neither replica ends up with the whole set. A second replica gives you two partial stores rather than a spare. Availability on a receiving store means a store built to replicate. Point the agents' remote-write at Mimir or Victoria Metrics, keep this Prometheus for rules and dashboards, and the durable copy lives somewhere that survives losing a pod ([Long-term storage and remote-write](/docs/hetzner/apalla/observability/metrics/long-term-storage-and-remote-write)). Alertmanager is the opposite case: its replicas gossip and de-duplicate, so running more than one there does buy redundancy ([Set up Alertmanager](/docs/hetzner/apalla/observability/alerting/set-up-alertmanager)). ### Sizing it Memory tracks active series, and disk is your retention window multiplied by the rate samples arrive. Both inputs are series this Prometheus keeps about itself, so measure them on your own workload instead of guessing: ```text prometheus_tsdb_head_series sum(rate(prometheus_tsdb_head_samples_appended_total[5m])) prometheus_tsdb_storage_blocks_bytes ``` The first is how many series are live right now, the second how fast samples arrive, the third what the compacted blocks on disk cost so far. Let it run a day, then multiply that third number by your retention in days and add headroom for compaction. That is your volume size, measured rather than assumed. Watch the series count for memory, because it is the one that grows on you: a single label with high churn multiplies into thousands of series. The `storage: 20Gi` and `retention: 15d` in the object above are a starting point, not a sized figure. They hold the scrape set these guides set up on a small cluster and nothing more. [Plan your observability stack](/docs/hetzner/apalla/observability/plan-your-observability-stack) covers the retention and cardinality budgets behind those numbers, and [Long-term storage and remote-write](/docs/hetzner/apalla/observability/metrics/long-term-storage-and-remote-write) covers history beyond what one volume holds. ### Before you call it production This store is fit for live troubleshooting and for alerting on one cluster. What stands between it and something you lean on during an incident is the list below, and none of it is in the object above. **Bound its resources.** The store's container requests nothing and has no limit, so the scheduler places it without accounting for the memory it will use, and a container with no request is among the first reclaimed when the node comes under memory pressure. Take the series count you measured above, then set requests to match what the process actually uses: ```yaml title="prometheus.yaml" spec: resources: requests: cpu: memory: limits: memory: ``` Set the memory limit and leave CPU unlimited. A memory limit turns a cardinality spike into one restarted pod instead of a node that runs out of memory under every workload on it, while a CPU limit throttles the store exactly when someone is querying it during an incident. **Cap retention by size as well as time.** `retention: 15d` alone means a cardinality spike fills the volume before the window expires, and a full disk stops the store from accepting writes. Add a size cap below the volume size so the oldest blocks drop instead: ```yaml title="prometheus.yaml" spec: retention: 15d retentionSize: 15GB ``` Keep the gap. The cap governs the persistent blocks, while the write-ahead log and compaction need room on the same volume. **Watch the store itself.** Nothing here alerts on it. The `PrometheusRule` objects covering the stack's own health arrived with kube-prometheus-stack, and installing the operator on its own leaves them out, so a store that stops receiving is silent rather than loud. [Set up Alertmanager](/docs/hetzner/apalla/observability/alerting/set-up-alertmanager) lists the series to alert on, remote-write failures first. **Expect it to move.** With one replica, a node replacement takes the store down for as long as a reschedule takes. Nothing is lost, because the volume follows the pod and the agents' write-ahead logs cover the gap. Resist pinning it up with a PodDisruptionBudget that keeps the single replica running: on a platform that replaces nodes, that blocks the drain rather than protecting your data. With the store up, put the data on a dashboard ([Set up Grafana](/docs/hetzner/apalla/observability/dashboards/set-up-grafana)) and alert on the node integrity conditions ([Platform alert rules](/docs/hetzner/apalla/observability/alerting/platform-alert-rules)). > [!TIP] > Running an observability stack is operations work, and you do not have to take it on alone. Syself's team can build this with you and operate it afterwards, from a one-time setup to a fully managed stack. See [Run it yourself, or with help](/docs/hetzner/apalla/concepts/ownership/run-it-yourself-or-with-help), or write to [contact@syself.com](mailto:contact@syself.com).