An alert on "CPU over 80%" pages you for something that may not matter. An alert on "we are burning through our error budget" pages you for something a customer experiences. Service-level objectives move alerting from raw thresholds to whether the service is actually meeting its promise, which is the difference between noise and an alert worth acting on immediately.

## SLI, SLO, and error budget

- An **SLI** (service-level indicator) is a measurement of the service, from your own metrics: the fraction of requests served under 300 ms, or the fraction that returned success.
- An **SLO** (service-level objective) is the target for that indicator over a window: 99.9% of requests succeed over 30 days.
- The **error budget** is what the SLO allows you to miss: at 99.9%, 0.1% of requests can fail before you are out of budget. As long as budget remains, you are within target.

## Build SLIs from app metrics

You need the app's own request metrics for this, so instrument it first ([Custom application metrics](/docs/hetzner/apalla/observability/metrics/custom-application-metrics)). Two common SLIs:

- **Availability:** `sum(rate(http_requests_total{code!~"5.."}[5m])) / sum(rate(http_requests_total[5m]))`.
- **Latency:** the fraction of requests under your threshold, from a histogram: `histogram_quantile` or a bucket ratio.

Precompute these with **recording rules** so the dashboards and alerts read a cheap pre-aggregated series instead of recomputing a heavy query every evaluation.

## Alert on burn rate, over multiple windows

The good alert is multi-window burn-rate. Burn rate is how fast you are spending the budget: a burn rate of 1 spends the whole budget exactly over the SLO window; a burn rate of 14 spends it in about two days. Alert when a fast burn holds over a short window **and** a slower burn holds over a longer one, so a brief spike does not page and a real, sustained problem does. This catches both a sudden outage and a slow, sustained drain, without the false pages a single threshold gives you.

Writing these rules by hand is error-prone. **Sloth** generates the recording and multi-window burn-rate alerting rules from a short SLO spec, so you declare the objective and it emits the PrometheusRule.

## Dashboards and fatigue

Put the budget on a dashboard: how much is left, and the burn rate over time. A team that can see the budget spends it deliberately (release faster while there is room, and slow down when it runs low) instead of arguing about every alert. Keep the discipline: only page on things tied to an SLO a customer cares about. Everything else is a ticket or a dashboard, not a page. That is how you avoid the alert fatigue that makes people ignore the pager.