Alert routing and receivers

Alertmanager's routing tree decides which alert goes to whom. You match on the labels an alert carries, severity, a team label, a cluster label, and send each branch to the right receiver. Done well, a page reaches the on-call for that service, and one client's alerts never land in another client's channel.

The routing tree#

Routes are a tree: an alert enters at the root and walks down until it matches a branch. Match on labels, and put the most specific routes first.

		route:
  receiver: default-email
  group_by: ["alertname", "cluster", "namespace"]
  routes:
    - matchers: ["severity = critical"]
      receiver: pagerduty
    - matchers: ["team = payments"]
      receiver: payments-slack
    - matchers: ["cluster = client-acme"]
      receiver: acme-slack
	

Receivers#

Each receiver is a destination. Add these receiver configurations to the same alertmanager-config.yaml file alongside your routes. Keep every credential in a Secret and reference it, never inline.

• Slack: an incoming-webhook URL per channel.

  		receivers:
    - name: payments-slack
      slack_configs:
        - api_url_file: /etc/alertmanager/secrets/slack-payments/url
          channel: "#payments-alerts"
  	

• PagerDuty / Opsgenie: a routing/integration key; use these for the critical branch that must page a human.

• Email (SMTP): a smtp_* block for a mailbox; fine for warnings and digests.

• Generic webhook: a webhook_configs URL for anything else, a ticketing system, a custom bot, a chatops handler.

Per-client routing for agencies#

For an agency, the cluster (or a client) label on every alert is what keeps tenants apart. Each cluster remote-writes with its own external label (see Multi-cluster observability ), so central Alertmanager can route cluster = client-acme to the Acme channel and cluster = client-beta to Beta's, from one config. A client only ever sees their own alerts, and you see all of them.

Test each route#

A routing tree is easy to get subtly wrong: a matcher that never matches, or an alert that quietly falls through to default-email. Test it before you trust it.

amtool is Alertmanager's command-line tool, and it ships in the same release, so it is already in the Alertmanager container and there is nothing to install in the cluster. To check a config before you apply it, put the same binary on your own machine from the Alertmanager release⁠ tarball, where amtool sits next to alertmanager itself.

Start with check-config, which catches a typo or a receiver you referenced but never defined:

		$ amtool check-config alertmanager-config.yaml
Checking 'alertmanager-config.yaml'  SUCCESS
Found:
 - global config
 - route
 - 0 inhibit rules
 - 4 receivers
 - 0 templates
	

config routes show then draws the branches in the order Alertmanager reads them:

		$ amtool config routes show --config.file=alertmanager-config.yaml
Routing tree:
.
└── default-route  receiver: default-email
    ├── {severity="critical"}  receiver: pagerduty
    ├── {team="payments"}  receiver: payments-slack
    └── {cluster="client-acme"}  receiver: acme-slack
	

config routes test takes the labels an alert would carry and prints the receiver it reaches:

		$ amtool config routes test --config.file=alertmanager-config.yaml \
  severity=critical cluster=client-acme
pagerduty
	

That result is the reason to test. A critical alert from Acme pages pagerduty; it does not reach acme-slack, because the first matching branch wins and severity = critical sits above the cluster route. Drop the severity and the same alert lands where you would expect:

		$ amtool config routes test --config.file=alertmanager-config.yaml \
  severity=warning cluster=client-acme
acme-slack
	

Neither answer is wrong, but only one of them is what you meant. Decide which, then order the branches to match.

To test what is running rather than what you wrote, run the same command inside the pod. The operator renders your config into the container, so that copy is the live one:

		$ kubectl -n monitoring exec alertmanager-main-0 \
  -c alertmanager -- amtool config routes test \
  --config.file=/etc/alertmanager/config_out/alertmanager.env.yaml \
  severity=critical cluster=client-acme
	

Test the critical path especially, so you find out a page would not fire in a drill, not during an incident.