Alertmanager's routing tree decides which alert goes to whom. You match on the labels an alert carries, `severity`, a team label, a `cluster` label, and send each branch to the right receiver. Done well, a page reaches the on-call for that service, and one client's alerts never land in another client's channel.

## The routing tree

Routes are a tree: an alert enters at the root and walks down until it matches a branch. Match on labels, and put the most specific routes first.

```yaml title="alertmanager-config.yaml"
route:
  receiver: default-email
  group_by: ["alertname", "cluster", "namespace"]
  routes:
    - matchers: ["severity = critical"]
      receiver: pagerduty
    - matchers: ["team = payments"]
      receiver: payments-slack
    - matchers: ["cluster = client-acme"]
      receiver: acme-slack
```

## Receivers

Each receiver is a destination. Add these receiver configurations to the same `alertmanager-config.yaml` file alongside your routes. Keep every credential in a Secret and reference it, never inline.

- **Slack:** an incoming-webhook URL per channel.

  ```yaml title="alertmanager-config.yaml"
  receivers:
    - name: payments-slack
      slack_configs:
        - api_url_file: /etc/alertmanager/secrets/slack-payments/url
          channel: "#payments-alerts"
  ```

- **PagerDuty / Opsgenie:** a routing/integration key; use these for the `critical` branch that must page a human.
- **Email (SMTP):** a `smtp_*` block for a mailbox; fine for warnings and digests.
- **Generic webhook:** a `webhook_configs` URL for anything else, a ticketing system, a custom bot, a chatops handler.

## Per-client routing for agencies

For an agency, the `cluster` (or a `client`) label on every alert is what keeps tenants apart. Each cluster remote-writes with its own external label (see [Multi-cluster observability](/docs/hetzner/apalla/observability/multi-cluster/multi-cluster-observability)), so central Alertmanager can route `cluster = client-acme` to the Acme channel and `cluster = client-beta` to Beta's, from one config. A client only ever sees their own alerts, and you see all of them.

## Test each route

A routing tree is easy to get subtly wrong: a matcher that never matches, or an alert that quietly falls through to `default-email`. Test it before you trust it.

`amtool` is Alertmanager's command-line tool, and it ships in the same release, so it is already in the Alertmanager container and there is nothing to install in the cluster. To check a config before you apply it, put the same binary on your own machine from the [Alertmanager release](https://github.com/prometheus/alertmanager/releases) tarball, where `amtool` sits next to `alertmanager` itself.

Start with `check-config`, which catches a typo or a receiver you referenced but never defined:

```console
$ amtool check-config alertmanager-config.yaml
Checking 'alertmanager-config.yaml'  SUCCESS
Found:
 - global config
 - route
 - 0 inhibit rules
 - 4 receivers
 - 0 templates
```

`config routes show` then draws the branches in the order Alertmanager reads them:

```console
$ amtool config routes show --config.file=alertmanager-config.yaml
Routing tree:
.
└── default-route  receiver: default-email
    ├── {severity="critical"}  receiver: pagerduty
    ├── {team="payments"}  receiver: payments-slack
    └── {cluster="client-acme"}  receiver: acme-slack
```

`config routes test` takes the labels an alert would carry and prints the receiver it reaches:

```console
$ amtool config routes test --config.file=alertmanager-config.yaml \
  severity=critical cluster=client-acme
pagerduty
```

That result is the reason to test. A critical alert from Acme pages `pagerduty`; it does not reach `acme-slack`, because the first matching branch wins and `severity = critical` sits above the cluster route. Drop the severity and the same alert lands where you would expect:

```console
$ amtool config routes test --config.file=alertmanager-config.yaml \
  severity=warning cluster=client-acme
acme-slack
```

Neither answer is wrong, but only one of them is what you meant. Decide which, then order the branches to match.

To test what is running rather than what you wrote, run the same command inside the pod. The operator renders your config into the container, so that copy is the live one:

```console
$ kubectl -n monitoring exec alertmanager-main-0 \
  -c alertmanager -- amtool config routes test \
  --config.file=/etc/alertmanager/config_out/alertmanager.env.yaml \
  severity=critical cluster=client-acme
```

Test the critical path especially, so you find out a page would not fire in a drill, not during an incident.