CPU is not a symptom
High CPU is normal for a busy service and meaningless to a customer. Alerting on it produces pages that resolve themselves and teach responders that alerts are noise. Alert on what the customer experiences: elevated error rate, latency beyond the threshold users notice, and requests failing to complete. Resource metrics belong on a dashboard you consult during an investigation, not in a pager rule.
Error budgets convert reliability into a decision
Define the target — say 99.9% of checkout requests succeeding within 800ms over 30 days — and the permitted failure becomes a budget. Burning it slowly is a ticket. Burning it fast enough to exhaust the month in hours is a page. This single reframing removes most arbitrary threshold arguments, because the question shifts from "is 500ms too slow" to "are we spending the budget faster than we can afford".
Multi-window burn-rate alerting
A single threshold either pages on brief harmless spikes or misses slow sustained degradation. Two windows fix it: a short window catching fast burn for immediate response, and a long window catching slow burn as a ticket. It is a small amount of configuration and it eliminates a large share of false pages.
Every alert needs a runbook and an owner
If an alert fires and the responder does not know what to do, the alert has failed regardless of its accuracy. Our standard is that no alert reaches production without a linked runbook stating what it means, what to check first, what the likely causes are, and what to do if it is a false positive. Alerts without owners get deleted at the quarterly review.
Traces answer the question dashboards raise
Dashboards tell you something is wrong. Distributed traces tell you where. Instrument service boundaries with OpenTelemetry, propagate context properly, and sample intelligently — keep all errors and slow requests, sample the healthy majority. Without traces, a microservice incident becomes a conference call where six teams each confirm their service looks fine.
The metric worth tracking about your alerting
Count pages per on-call shift, and the proportion that resulted in action. If more than a small minority of pages are actionable, alerting is healthy. If most are not, you do not have an on-call rota — you have a rota of people learning to ignore alarms, which is how genuine incidents get missed.