Most engineering teams maintain hundreds of legacy alerts configured around static infrastructure thresholds: CPU utilization above 85%, disk usage over 90%, or average response time exceeding 200ms. These alerts inevitably fire during harmless background cron jobs or momentary network jitter, conditioning on-call engineers to ignore pager notifications until a catastrophic outage occurs.
The Shift to Service Level Indicators (SLIs)
Service Level Objectives (SLOs) align alerting directly with user experience. If a user can successfully authenticate and checkout within 500 milliseconds, internal container CPU usage is irrelevant. An SLI measures the exact percentage of valid requests that meet defined availability and latency contracts over a rolling 30-day window.
Multi-Window, Multi-Burn-Rate PromQL Formulas
To eliminate false alarms while guaranteeing rapid response during critical failures, Google SRE practices define multi-window burn rate alerts:
- 14.4x Burn Rate (2% budget consumed in 1 hour): Triggers an immediate emergency pager notification because the entire monthly error budget will be depleted in 50 hours if left unaddressed. Requires both a 1-hour short window and a 5-minute confirmation window to prevent transient spikes from paging.
- 6x Burn Rate (5% budget consumed in 6 hours): Pages on-call during working hours or files a high-priority ticket for morning triage.
- 1x Burn Rate (Budget consumed at normal allowable pace): No notification, surfaced on weekly operational review dashboards.
Implementing this alerting model requires careful PromQL recording rules to calculate rolling error ratios efficiently without overloading Prometheus queriers. In our consulting work, we partner with team leads to author, test, and validate these exact alert pipelines before cutover.