Alertmanager turns a Prometheus alert into a notification. Nothing installs it for you: you create an `Alertmanager` object and the operator from [Set up Prometheus](/docs/hetzner/apalla/observability/metrics/set-up-prometheus) runs it. This page gets one alert flowing end to end; [Alert routing and receivers](/docs/hetzner/apalla/observability/alerting/alert-routing-and-receivers) sends it to real channels. Prometheus evaluates your rules and sends firing alerts to Alertmanager, which groups them, silences what it should, and forwards the rest to receivers (the destinations, like Slack, email, or a pager). Two objects drive it. A `PrometheusRule` holds your alert expressions; the operator loads it only when its labels match the store's `ruleSelector`, the filter that decides which rules it loads. The config, holding the routing tree and the receivers, is a Secret. Create the Alertmanager and its config By default, the config lives in a Secret named `alertmanager-` for an Alertmanager called ``, and the operator mounts it into the pods. If your Secret uses a different name, set `configSecret` under `spec` in the `Alertmanager` object. It is a Secret rather than a ConfigMap because receiver credentials end up in it, so never move it to a ConfigMap: ```yaml title="alertmanager.yaml" apiVersion: v1 kind: Secret metadata: name: alertmanager-main # alertmanager-, default name the operator looks for namespace: monitoring stringData: alertmanager.yaml: | route: receiver: default group_by: ["alertname", "cluster", "namespace"] receivers: - name: default --- apiVersion: monitoring.coreos.com/v1 kind: Alertmanager metadata: name: main namespace: monitoring spec: # configSecret: alertmanager-main # Optional: set if secret name differs from alertmanager- # Two replicas gossip, so they de-duplicate notifications and losing one # node does not silence your paging. replicas: 2 # standard volumes provision only on cloud nodes, so keep the replicas there. # Change this if you point storageClassName at a local class instead. nodeSelector: autopilot.syself.com/machine-type: hcloud securityContext: runAsUser: 1000 runAsGroup: 2000 fsGroup: 2000 runAsNonRoot: true seccompProfile: type: RuntimeDefault storage: volumeClaimTemplate: spec: # standard = Hetzner Cloud volumes, cloud nodes only. The nodeSelector # above keeps this on a cloud node. To run on a bare-metal node instead, # drop it and name the local class matching that node's disks. # (local-nvme, local-ssd, or local-hdd through TopoLVM). storageClassName: standard accessModes: ["ReadWriteOnce"] resources: requests: storage: 10Gi ``` ```console $ kubectl apply -f alertmanager.yaml $ kubectl -n monitoring rollout status statefulset/alertmanager-main ``` The operator puts the replicas behind a Service called `alertmanager-operated`, on port `web`. That is the Service the store's `alerting.alertmanagers` block already points at, so Prometheus finds both replicas with nothing further to configure: ```console $ kubectl -n monitoring port-forward svc/prometheus-operated 9090:9090 ``` `http://localhost:9090/api/v1/alertmanagers` lists them; an empty list means the store cannot see them, which is usually the missing Role from [Set up Prometheus](/docs/hetzner/apalla/observability/metrics/set-up-prometheus). Write a rule ```yaml title="prometheusrule.yaml" apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: example-rules namespace: monitoring labels: prometheus: main # matched by the store's ruleSelector spec: groups: - name: example rules: - alert: TargetDown expr: up == 0 for: 5m labels: severity: warning annotations: summary: "{{ $labels.job }} target is down on {{ $labels.instance }}" ``` The `for: 5m` means the condition must hold for five minutes before the alert fires, which keeps a brief spike from paging anyone. Confirm the store picked the rule up on `http://localhost:9090/rules`; if the page is empty, the label does not match the `ruleSelector`. Test firing an alert Confirm the path works before you rely on it. Port-forward Alertmanager and watch the alert arrive: ```console $ kubectl -n monitoring port-forward svc/alertmanager-operated 9093:9093 ``` Open `http://localhost:9093`. To force a test, apply a rule whose `expr` is `vector(1)` (always fires), confirm it shows up, then remove it. ## Grouping and inhibition Two features keep noise down, covered in full in [Silences and grouping](/docs/hetzner/apalla/observability/alerting/silences-and-grouping): **grouping** collapses many related alerts (a whole pool going down) into one notification, and **inhibition** suppresses a downstream alert when an upstream one is already firing (do not page for "pod not ready" on a node you already know is down). ## Keep the config in Git Credentials do not belong in the manifest. Keep the routing tree in Git, load the receiver secrets from their own Secret, and reference them by file path in the config, which is what the `*_file` receiver options are for ([Alert routing and receivers](/docs/hetzner/apalla/observability/alerting/alert-routing-and-receivers)). Keep the rules in Git too, so a rebuilt cluster comes back alerting from the start. ## Meta monitoring Your monitoring cannot alert you about its own outage. If a collector stops writing or Alertmanager stops delivering, no alert fires and the cluster looks quiet. Monitoring the monitoring itself closes that gap. Nothing ships a dead-man's-switch alert for you, so write one. An alert that always fires is the whole trick: ```yaml title="watchdog-rule.yaml" apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: watchdog namespace: monitoring labels: prometheus: main spec: groups: - name: watchdog rules: - alert: Watchdog expr: vector(1) labels: {severity: none} annotations: summary: "This alert always fires. Route it off-cluster; silence means the pipeline is dead." ``` Route it to a small outside service (Dead Man's Snitch, healthchecks.io, or a PagerDuty heartbeat) that expects a regular ping and pages you when the ping stops. A healthy pipeline stays quiet, and a dead one raises the alarm on its own. ### Collect the stack's own metrics first None of those alerts can fire on a stack that is not scraping itself, and by default this one is not. Prometheus scrapes nothing, and the Application Alloy only reads `ServiceMonitor` objects, so unless you create them the monitoring components are the one part of the cluster with no telemetry. Three pieces close it: - **The System Alloy** scrapes its own `127.0.0.1:12345` on every node, as the `alloy-system` job in [its config](/docs/hetzner/apalla/observability/collection/deploy-the-system-alloy). - **The Application Alloy** sets `serviceMonitor.enabled: true`, and because it reads every `ServiceMonitor` it picks up its own. - **Prometheus, Alertmanager, Grafana, Loki, and Tempo** need a `ServiceMonitor` each. The operator creates none for the first two, and the charts leave theirs off by default: ```yaml title="stack-servicemonitors.yaml" apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: prometheus namespace: monitoring spec: selector: matchLabels: operated-prometheus: "true" endpoints: - port: web interval: 30s --- apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: alertmanager namespace: monitoring spec: selector: matchLabels: operated-alertmanager: "true" endpoints: - port: web interval: 30s ``` Grafana, Loki, and Tempo each expose a chart flag for the same thing (`serviceMonitor.enabled=true` on the Grafana and Tempo charts, `monitoring.serviceMonitor.enabled=true` on Loki). Turn them on when you install, then confirm every component answers: ```text count(up{job=~"alloy-system|app-alloy|prometheus-operated|alertmanager-operated|grafana|tempo|monitoring/loki"}) by (job) ``` Expect one row per job. Note that the `job` label comes from the Service each component's chart or operator creates, not from the `ServiceMonitor` name, so it is `prometheus-operated` and `alertmanager-operated` for the operator's Services and `monitoring/loki` for the Loki chart. If you are unsure what a component ended up as, list everything with `count(up) by (job)` and look for the one you just enabled. A component missing from that list is a blind spot, not a passing check. ### Then alert on it With those series arriving, alert on the stack's own health: `up == 0` for the monitoring targets, remote-write failures (`prometheus_remote_storage_samples_failed_total` climbing, with `prometheus_remote_storage_samples_pending` as the earlier signal), dropped log lines (`loki_write_dropped_entries_total`, which also catches an audit event rejected for exceeding Loki's entry limit), and Alertmanager delivery failures (`alertmanager_notifications_failed_total`). Count the agents against the node count rather than waiting for `up == 0`, because a target that disappears stops producing `up` at all: ```text count(up{job="alloy-system"}) < count(kube_node_info) ``` Where you can, run these checks from outside the cluster, for example on the central store in [Multi-cluster observability](/docs/hetzner/apalla/observability/multi-cluster/multi-cluster-observability), so a whole-cluster outage still reaches you. > [!NOTE] > For self-monitoring metrics (`prometheus_remote_storage_samples_failed_total`, `alertmanager_notifications_failed_total`) to arrive in your store, enable self-scraping on your Alloy instances (`localhost:12345/metrics`) and ensure `ServiceMonitor` resources are enabled on the Prometheus, Alertmanager, Loki, and Tempo chart installations. Without active scrapes on the monitoring stack itself, these metrics will not be present.