Set up Alertmanager

Alertmanager turns a Prometheus alert into a notification. Nothing installs it for you: you create an Alertmanager object and the operator from Set up Prometheus runs it. This page gets one alert flowing end to end; Alert routing and receivers sends it to real channels.

Prometheus evaluates your rules and sends firing alerts to Alertmanager, which groups them, silences what it should, and forwards the rest to receivers (the destinations, like Slack, email, or a pager). Two objects drive it. A PrometheusRule holds your alert expressions; the operator loads it only when its labels match the store's ruleSelector, the filter that decides which rules it loads. The config, holding the routing tree and the receivers, is a Secret.

		apiVersion: v1
kind: Secret
metadata:
  name: alertmanager-main # alertmanager-<name>, default name the operator looks for
  namespace: monitoring
stringData:
  alertmanager.yaml: |
    route:
      receiver: default
      group_by: ["alertname", "cluster", "namespace"]
    receivers:
      - name: default
---
apiVersion: monitoring.coreos.com/v1
kind: Alertmanager
metadata:
  name: main
  namespace: monitoring
spec:
  # configSecret: alertmanager-main # Optional: set if secret name differs from alertmanager-<name>
  # Two replicas gossip, so they de-duplicate notifications and losing one
  # node does not silence your paging.
  replicas: 2
  # standard volumes provision only on cloud nodes, so keep the replicas there.
  # Change this if you point storageClassName at a local class instead.
  nodeSelector:
    autopilot.syself.com/machine-type: hcloud
  securityContext:
    runAsUser: 1000
    runAsGroup: 2000
    fsGroup: 2000
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  storage:
    volumeClaimTemplate:
      spec:
        # standard = Hetzner Cloud volumes, cloud nodes only. The nodeSelector
        # above keeps this on a cloud node. To run on a bare-metal node instead,
        # drop it and name the local class matching that node's disks.
        # (local-nvme, local-ssd, or local-hdd through TopoLVM).
        storageClassName: standard
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 10Gi
	

		$ kubectl apply -f alertmanager.yaml
$ kubectl -n monitoring rollout status statefulset/alertmanager-main
	

		$ kubectl -n monitoring port-forward svc/prometheus-operated 9090:9090
	

		apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: example-rules
  namespace: monitoring
  labels:
    prometheus: main # matched by the store's ruleSelector
spec:
  groups:
    - name: example
      rules:
        - alert: TargetDown
          expr: up == 0
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "{{ $labels.job }} target is down on {{ $labels.instance }}"
	

		$ kubectl -n monitoring port-forward svc/alertmanager-operated 9093:9093
	

Grouping and inhibition#

Two features keep noise down, covered in full in Silences and grouping : grouping collapses many related alerts (a whole pool going down) into one notification, and inhibition suppresses a downstream alert when an upstream one is already firing (do not page for "pod not ready" on a node you already know is down).

Keep the config in Git#

Credentials do not belong in the manifest. Keep the routing tree in Git, load the receiver secrets from their own Secret, and reference them by file path in the config, which is what the *_file receiver options are for (Alert routing and receivers ). Keep the rules in Git too, so a rebuilt cluster comes back alerting from the start.

Meta monitoring#

Your monitoring cannot alert you about its own outage. If a collector stops writing or Alertmanager stops delivering, no alert fires and the cluster looks quiet. Monitoring the monitoring itself closes that gap.

Nothing ships a dead-man's-switch alert for you, so write one. An alert that always fires is the whole trick:

		apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: watchdog
  namespace: monitoring
  labels:
    prometheus: main
spec:
  groups:
    - name: watchdog
      rules:
        - alert: Watchdog
          expr: vector(1)
          labels: {severity: none}
          annotations:
            summary: "This alert always fires. Route it off-cluster; silence means the pipeline is dead."
	

Route it to a small outside service (Dead Man's Snitch, healthchecks.io, or a PagerDuty heartbeat) that expects a regular ping and pages you when the ping stops. A healthy pipeline stays quiet, and a dead one raises the alarm on its own.

Collect the stack's own metrics first#

None of those alerts can fire on a stack that is not scraping itself, and by default this one is not. Prometheus scrapes nothing, and the Application Alloy only reads ServiceMonitor objects, so unless you create them the monitoring components are the one part of the cluster with no telemetry. Three pieces close it:

• The System Alloy scrapes its own 127.0.0.1:12345 on every node, as the alloy-system job in its config .
• The Application Alloy sets serviceMonitor.enabled: true, and because it reads every ServiceMonitor it picks up its own.
• Prometheus, Alertmanager, Grafana, Loki, and Tempo need a ServiceMonitor each. The operator creates none for the first two, and the charts leave theirs off by default:

		apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: prometheus
  namespace: monitoring
spec:
  selector:
    matchLabels:
      operated-prometheus: "true"
  endpoints:
    - port: web
      interval: 30s
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: alertmanager
  namespace: monitoring
spec:
  selector:
    matchLabels:
      operated-alertmanager: "true"
  endpoints:
    - port: web
      interval: 30s
	

Grafana, Loki, and Tempo each expose a chart flag for the same thing (serviceMonitor.enabled=true on the Grafana and Tempo charts, monitoring.serviceMonitor.enabled=true on Loki). Turn them on when you install, then confirm every component answers:

		count(up{job=~"alloy-system|app-alloy|prometheus-operated|alertmanager-operated|grafana|tempo|monitoring/loki"}) by (job)
	

Expect one row per job. Note that the job label comes from the Service each component's chart or operator creates, not from the ServiceMonitor name, so it is prometheus-operated and alertmanager-operated for the operator's Services and monitoring/loki for the Loki chart. If you are unsure what a component ended up as, list everything with count(up) by (job) and look for the one you just enabled. A component missing from that list is a blind spot, not a passing check.

Then alert on it#

With those series arriving, alert on the stack's own health: up == 0 for the monitoring targets, remote-write failures (prometheus_remote_storage_samples_failed_total climbing, with prometheus_remote_storage_samples_pending as the earlier signal), dropped log lines (loki_write_dropped_entries_total, which also catches an audit event rejected for exceeding Loki's entry limit), and Alertmanager delivery failures (alertmanager_notifications_failed_total). Count the agents against the node count rather than waiting for up == 0, because a target that disappears stops producing up at all:

		count(up{job="alloy-system"}) < count(kube_node_info)
	

Where you can, run these checks from outside the cluster, for example on the central store in Multi-cluster observability , so a whole-cluster outage still reaches you.