Silences and grouping

On a platform that drains and replaces nodes, one upgrade can fire dozens of alerts at once, none of which need a human. Grouping collapses related alerts into one notification, and silences mute expected noise during maintenance. Get both right, or the on-call rotation learns to ignore the pager.

Group related alerts#

Alertmanager groups alerts that share the labels you list in group_by, and sends one notification per group instead of one per alert:

		route:
  group_by: ["alertname", "cluster", "namespace"]
  group_wait: 30s # wait this long to collect the first batch of a new group
  group_interval: 5m # wait this long before sending an update for a group
  repeat_interval: 4h # re-notify an unresolved group this often
	

group_wait gives a burst a moment to gather, so a whole pool going NotReady arrives as one notification, not twenty. group_interval paces updates to a group, and repeat_interval re-pages if it is still firing.

Silence maintenance#

A node replacement or a cluster upgrade rolls nodes on purpose, and the alerts that follow (a node briefly NotReady, a pod rescheduling) are expected. Silence them for the window rather than paging on your own maintenance. A silence matches alert labels for a set duration, and amtool, the CLI that ships with Alertmanager (see Test each route ), adds one:

		$ amtool silence add \
  --alertmanager.url=http://localhost:9093 \
  --duration=2h --comment="cluster upgrade, ticket OPS-1234" \
  alertname=~"KubeNodeNotReady|KubePodNotReady" cluster="prod-eu"
	

Scope the silence tightly (the specific alerts and the specific cluster) so you do not blind yourself to a real problem elsewhere while maintenance runs. Set the duration to a little longer than the planned window, and let it expire on its own.

Inhibition#

Inhibition suppresses one alert while another is firing, which cuts the cascade where one root cause triggers ten symptoms. Do not page for "pod not ready" and "service has no endpoints" on a node whose "node down" alert is already firing:

		inhibit_rules:
  - source_matchers: ["alertname = KubeNodeNotReady"]
    target_matchers: ["severity = warning"]
    equal: ["node"]
	

Audit who silenced what#

A silence hides alerts, so it is worth an audit trail. Require a --comment with a ticket reference on every silence (as above), and periodically list active silences (amtool silence query) so a forgotten silence does not quietly hide a real alert for weeks.