On this page you install Prometheus as the store for your metrics. It keeps the samples and answers queries, but it does not scrape anything itself. The two Alloy agents from [Collection](/docs/hetzner/apalla/observability/collection/using-alloy-for-observability) do the collecting and push their data in over remote-write.
The install goes through the Prometheus Operator. You apply objects like `Prometheus` and `PrometheusRule`, and the operator turns them into a running Prometheus and loaded rules. Everything else in the stack has its own page, so you add the pieces you want and skip the ones you do not.
```mermaid
flowchart LR
subgraph objects["Objects you apply"]
pobj["Prometheus"]:::app
rules["PrometheusRule"]:::app
sm["ServiceMonitor and PodMonitor
written next to each application"]:::app
end
pobj --> op["Prometheus Operator"]:::app
rules --> op
op -->|"renders the config, reloads the rules"| prom["Prometheus
remote-write receiver and query API"]:::data
sm -.->|"read from the API, no operator in the path"| alloy["System Alloy and Application Alloy"]:::app
alloy -->|"remote-write"| prom
prom -->|"firing alerts"| am["Alertmanager"]:::app
graf["Grafana"]:::app -->|"queries"| prom
```
Objects on the left, workloads on the right, and no arrow from Prometheus to a scrape target: the agents collect, this Prometheus receives. That is why its targets page stays empty on a working install. The dotted line is the one path the operator sits outside of, because both agents read `ServiceMonitor` and `PodMonitor` objects from the Kubernetes API themselves.
## What each piece does
- **The operator** reconciles objects into workloads: a `Prometheus` becomes a StatefulSet, an `Alertmanager` becomes a running Alertmanager, a `PrometheusRule` becomes a loaded rule file. It scrapes nothing itself.
- **Prometheus** stores what both agents write, evaluates your alert rules, and answers queries from Grafana.
- **The rest, each on its own page:** [Alertmanager](/docs/hetzner/apalla/observability/alerting/set-up-alertmanager) for notifications, [Grafana](/docs/hetzner/apalla/observability/dashboards/set-up-grafana) for dashboards, [kube-state-metrics](/docs/hetzner/apalla/observability/metrics/kube-state-metrics) for object state, [node-exporter](/docs/hetzner/apalla/observability/metrics/node-and-hardware-metrics) for per-node hardware.
This setup differs from a plain Prometheus install. There, every service that wants monitoring means an edit to a central config file and a reload, and the StatefulSet, the volume, and the reload plumbing stay yours to maintain. The operator takes over that maintenance and replaces the central file with objects: teams ship a `ServiceMonitor` or `PodMonitor` next to their application, and both Alloy agents pick it up from the Kubernetes API with no central edit ([Deploy the Application Alloy](/docs/hetzner/apalla/observability/collection/deploy-the-application-alloy)).
## Install the operator
The project ships a single manifest, `bundle.yaml`, holding the CRDs, the controller, and the RBAC it needs. It targets the `default` namespace, so use kustomize to put it in `monitoring`:
```console
$ kubectl create namespace monitoring
$ curl -sLO https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/refs/tags//bundle.yaml
$ curl -sLO https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/refs/tags//kustomization.yaml
$ kustomize edit set namespace monitoring
$ kubectl create -k .
```
Pin `` to a release tag rather than tracking the newest one, so an update does not change behavior under you. Use `kubectl create -k .` or `kubectl apply --server-side -k .` rather than client-side `kubectl apply`, because CRD definitions in the operator bundle exceed the 256KB Kubernetes annotation limit.
```console
$ kubectl -n monitoring get deploy prometheus-operator
NAME READY UP-TO-DATE AVAILABLE AGE
prometheus-operator 1/1 1 1 40s
```
## Deploy the store
Give it a ServiceAccount and namespace read access
A store that scrapes nothing needs no cluster-wide read access. It does do one piece of discovery, finding the Alertmanager pods to send alerts to, and that needs services, endpoints, and pods in its own namespace:
```yaml title="prometheus-rbac.yaml"
apiVersion: v1
kind: ServiceAccount
metadata:
name: prometheus
namespace: monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: prometheus
namespace: monitoring
rules:
- apiGroups: [""]
resources: ["services", "endpoints", "pods"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: prometheus
namespace: monitoring
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: prometheus
subjects:
- kind: ServiceAccount
name: prometheus
namespace: monitoring
```
Leave the Role out and Prometheus still starts, but its log fills with `endpoints is forbidden` and it never finds Alertmanager.
Write the Prometheus object
```yaml title="prometheus.yaml"
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
name: main
namespace: monitoring
spec:
serviceAccountName: prometheus
replicas: 1
retention: 15d
# Both Alloy agents push their samples in, so accept remote-write.
enableRemoteWriteReceiver: true
# Point monitor selectors at a non-matching label so this Prometheus
# scrapes nothing and never competes with the agents over targets, while
# ensuring the operator renders a full config and loads rule files.
serviceMonitorSelector:
matchLabels:
scrape-by-prometheus: "true"
podMonitorSelector:
matchLabels:
scrape-by-prometheus: "true"
probeSelector:
matchLabels:
scrape-by-prometheus: "true"
scrapeConfigSelector:
matchLabels:
scrape-by-prometheus: "true"
# Rules are the exception: load every PrometheusRule labelled
# prometheus: main, in any namespace.
ruleSelector:
matchLabels:
prometheus: main
ruleNamespaceSelector: {}
# Stamp every series with the cluster it came from, so one store can tell
# your clusters apart.
externalLabels:
cluster:
# Where firing alerts go. The operator creates this Service along with the
# Alertmanager on the alerting page.
alerting:
alertmanagers:
- namespace: monitoring
name: alertmanager-operated
port: web
# standard volumes provision only on cloud nodes, so keep the store there.
# change this if you point storageClassName at a local class instead.
nodeSelector:
autopilot.syself.com/machine-type: hcloud
# The operator sets no pod securityContext of its own, and a fresh volume
# belongs to root, so without fsGroup Prometheus cannot write its database.
securityContext:
runAsUser: 1000
runAsGroup: 2000
fsGroup: 2000
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
storage:
volumeClaimTemplate:
spec:
# standard = Hetzner Cloud volumes, cloud nodes only. The nodeSelector
# above keeps this on a cloud node. To run on a bare-metal node instead,
# drop it and name the local class matching that node's disks.
# (local-nvme, local-ssd, or local-hdd through TopoLVM).
storageClassName: standard
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 20Gi
```
Apply it and confirm it accepts writes
```console
$ kubectl apply -f prometheus-rbac.yaml -f prometheus.yaml
$ kubectl -n monitoring rollout status statefulset/prometheus-main
```
The operator reports back what it selected, and selecting nothing is the goal here:
```console
$ kubectl -n monitoring get prometheus main \
-o jsonpath='{.status.conditions[?(@.type=="Reconciled")].message}'
No ServiceMonitor, PodMonitor, Probe, ScrapeConfig, and PrometheusRule have been selected.
```
The message goes away once you add rules on the alerting page. The check that stays valid is the empty targets page below.
Deploy the collectors ([Deploy the System Alloy](/docs/hetzner/apalla/observability/collection/deploy-the-system-alloy)), then check that samples are arriving from every node rather than from one:
```console
$ kubectl -n monitoring port-forward svc/prometheus-operated 9090:9090
```
At `http://localhost:9090`, query for a series each agent produces:
```text
count(up{job="etcd"}) by (instance)
```
One row per control-plane node means the pipeline works end to end, including the per-node identity that keeps each agent's series distinct. The targets page in the UI stays empty. That is expected, because this Prometheus does not scrape anything itself.
## What this setup covers
What you have now is a working store, not a hardened one. The receiver takes plain HTTP with no authentication or TLS, and the query API is open to every pod in the cluster, so anything that reaches the pod on port `9090` can write to or read from your store. Add a default-deny policy for the namespace ([Segment with network policies](/docs/hetzner/apalla/security/segment-with-network-policies)) and keep Grafana as the query surface that carries the login ([Set up Grafana](/docs/hetzner/apalla/observability/dashboards/set-up-grafana)).
The other limits have their own pages: the database is one volume holding one retention window ([Long-term storage and remote-write](/docs/hetzner/apalla/observability/metrics/long-term-storage-and-remote-write)), and it stays one store per cluster ([Multi-cluster observability](/docs/hetzner/apalla/observability/multi-cluster/multi-cluster-observability)).
### Why only one replica
The `prometheus.yaml` in step 2 sets `replicas: 1`, and that is deliberate. Raising the number does not add a backup: each replica keeps its own database on its own volume, and the pushes from the agents get split between the replicas. Instead of one complete store plus a replica as a spare, you get multiple replicas with the data divided among them, so only all of them together hold the complete set.
For real redundancy, point the agents' remote-write at a store built to replicate, like Mimir or Victoria Metrics, and keep this Prometheus for rules and dashboards ([Long-term storage and remote-write](/docs/hetzner/apalla/observability/metrics/long-term-storage-and-remote-write)).
### Sizing it
How much memory and disk this Prometheus needs depends on your workload: memory grows with the number of active series, and disk grows with how fast samples arrive and how long you keep them. Prometheus publishes these numbers about itself, so measure them instead of guessing:
```text
prometheus_tsdb_head_series
sum(rate(prometheus_tsdb_head_samples_appended_total[5m]))
prometheus_tsdb_storage_blocks_bytes
```
The first is how many series are live right now, the second how fast samples arrive and the third what the blocks on disk cost so far. Let it run a day, then multiply the disk number by your retention in days for the volume size, and watch the series count for memory.
The `storage: 20Gi` and `retention: 15d` in `prometheus.yaml` are a starting point, not a sized figure. They are enough only for the metrics the two Alloy agents collect on a small cluster. [Plan your observability stack](/docs/hetzner/apalla/observability/plan-your-observability-stack) covers the retention and cardinality budgets behind those numbers, and [Long-term storage and remote-write](/docs/hetzner/apalla/observability/metrics/long-term-storage-and-remote-write) covers history beyond what one volume holds.
### Before you call it production
This store is fine for live troubleshooting and for alerting on one cluster. Before you depend on it during an incident, work through the list below. None of it is part of the `prometheus.yaml` from step 2.
**Bound its resources.** The `prometheus.yaml` from step 2 sets no resource requests or limits. Set the requests to what the process actually uses, taken from the numbers you measured in the Sizing it section, and add a memory limit:
```yaml title="prometheus.yaml"
spec:
resources:
requests:
cpu:
memory:
limits:
memory:
```
Leave CPU unlimited: a CPU limit throttles queries exactly when you need them during an incident, while a memory limit turns a cardinality spike into one restarted pod instead of a starving node.
**Cap retention by size as well as time.** With `retention: 15d` alone, a spike in series can fill the volume before 15 days pass, and a full disk stops the store from accepting writes. Add a size cap below the volume size so the oldest data drops first:
```yaml title="prometheus.yaml"
spec:
retention: 15d
retentionSize: 15GB
```
**Watch the store itself.** Nothing in this setup alerts when the store stops receiving. [Set up Alertmanager](/docs/hetzner/apalla/observability/alerting/set-up-alertmanager) lists the series to alert on, remote-write failures first.
**Expect it to move.** A node replacement takes the single replica down until the pod reschedules. Nothing is lost: the volume follows the pod and the agents buffer the gap.
With the store up, put the data on a dashboard ([Set up Grafana](/docs/hetzner/apalla/observability/dashboards/set-up-grafana)) and alert on the node integrity conditions ([Platform alert rules](/docs/hetzner/apalla/observability/alerting/platform-alert-rules)).
> [!TIP]
> Running an observability stack is operations work, and you do not have to take it on alone. Syself's team can build this with you and operate it afterwards, from a one-time setup to a fully managed stack. See [Run it yourself, or with help](/docs/hetzner/apalla/concepts/ownership/run-it-yourself-or-with-help), or write to [contact@syself.com](mailto:contact@syself.com).