How do you avoid alert fatigue?

Alert on sustained conditions rather than on single events, so a rule fires after several consecutive failures instead of on the first blip. Give every rule a cooldown so a two-hour outage produces one notification rather than twenty-four, and suppress alerts during planned work. Then keep the set of alerts that reach a phone small enough that each one is worth the interruption.

How many consecutive failures should trigger an alert?

Three is the common default and it is a reasonable starting point, because a single failed check is often a network hiccup rather than an outage. The number interacts with your check interval, since three failures at a five-minute interval means a quarter of an hour before anyone hears. On a payment path, shorten the interval rather than dropping to a single failure.

What is a cooldown in monitoring alerts?

A cooldown is the minimum time between repeat notifications for the same rule. Without one, a sustained outage sends a notification on every failed check, which is how a channel gets muted during exactly the incident it existed for. Thirty minutes is a common default, and the right value is roughly how long it takes somebody to act on the first message.

Should contract drift page someone at night?

Usually not, because a drifting endpoint is still answering and nothing is down for anyone yet. Drift belongs in a channel that gets read during working hours, with an owner assigned the same day. The exception is a field that a paying integration depends on, and that decision belongs to whoever knows the consumers rather than to a default setting.

API Alerting Without the Noise: Thresholds, Windows and Cooldowns

An alerting setup has one failure mode that matters more than all the others combined, and it is that people stop reading it. Everything in this chapter is in service of that one problem.

Alert on conditions, not on events

A single failed check is weak evidence. Networks drop packets, load balancers recycle connections, and a deploy rolls one instance at a time.

A rule that fires on the first failure reacts to a blip that recovers on its own, while a rule that waits for three consecutive failures stays quiet through it and still catches the outage two checks later.

The cost of waiting is small and the cost of not waiting is large. Waiting two extra checks delays the real alert by two intervals, while firing on every blip teaches the team that the channel is usually wrong. The first cost is measured in minutes, and the second one is permanent.

The four conditions worth having

Most useful alerting reduces to four rule shapes, and each one keys on a different failure.

Rule	Fires on	Tune it with
Consecutive failures	A run of failed checks	The count, against your interval
Latency threshold	Average response time over a window exceeding a limit	The limit and the window length
Error rate	The share of failing checks in a window crossing a percentage	The percentage and the window
Contract deviation	A response that no longer matches the contract	The minimum severity

The latency and error-rate rules exist because of a failure the consecutive rule misses entirely. An endpoint that fails one check in three never accumulates three in a row, and is unmistakably broken. A window-based rule catches that pattern, so it is worth having both.

The one number to think hardest about is the window length. A window of five checks at a one-minute interval reacts in five minutes and is jumpy, while twenty checks at the same interval is calm and slow. Pick the reaction time first, then divide by the interval.

Cooldowns

A rule without a cooldown notifies on every evaluation for as long as the condition holds. A two-hour outage under a five-minute interval then produces twenty-four messages that all say the same thing.

Set the cooldown to roughly how long it takes somebody to pick the alert up and respond. Thirty minutes is a sensible default, and anything under five minutes is repetition rather than information. The recovery notification is the one that should always come through, because "it is back" is the message people are actually waiting for.

Route by consequence, not by severity label

Most alerting setups have one channel and a severity field nobody trusts. A better model asks what should happen when this fires.

It should	Send it to	Example
Wake somebody up	A phone	Checkout returns 500 for three checks running
Be seen within the hour	A chat channel people watch	p95 on a core read path doubled
Be picked up today	A chat channel or a ticket	An endpoint started returning an undocumented field
Be reviewed weekly	A digest or a dashboard	Coverage dropped because six endpoints shipped without monitors

Keeping the top row small is the entire discipline. Every alert promoted into it dilutes the ones already there, so the honest test for a new one is whether you would accept being woken up by it twice in a month.

Three alerts that should not page

Three common conditions look like outages and are not, and each one has a fix that is cheaper than a night shift.

An expired monitoring credential. The service is fine and your check cannot authenticate. Treat an authentication failure as its own category, because sending it as downtime turns a token rotation into an incident. Authentication for monitors covers how to keep them from expiring at all.

Planned work. A deployment window that pages the person running the deployment teaches everyone to mute the channel, and the mute outlives the deployment. Suppress during maintenance instead, which incidents and maintenance windows covers.

A monitor for a service that does not exist yet. In design-first work, monitors are often created before the endpoints ship. A monitor whose target has never answered should wait rather than report an outage, which design-first monitoring covers.

Rules belong to groups, not to monitors

Configuring alerts on each monitor individually works until you have forty of them, at which point nobody knows what is configured where and a new monitor silently arrives with nothing.

Defining a set of rules once and assigning it to a whole environment or a whole specification scales better and answers the auditing question. The detail worth knowing about any such system is how overlapping assignments resolve. The usual rule is that the most specific assignment wins outright rather than merging with the broader one. A narrow policy carrying only a drift rule can therefore remove downtime alerting from everything it covers, silently.

In Routebase

Alert policies are reusable sets of rules, assigned to an environment, a specification or a single monitor, with the most specific assignment winning. A policy holds at most one rule per type, and the four types are downtime, latency threshold, error rate and schema drift.

The New Alert Policy dialog on the alert policies page, with a name and description, a downtime rule set to two consecutive failures with a fifteen-minute cooldown, and a latency threshold rule set to 800 milliseconds over a window of five checks with a thirty-minute cooldown. — Each rule carries its own condition and its own cooldown, so the rule that pages on downtime can react faster than the one that reports a slow week.

The defaults are three consecutive failures for downtime, 2000 milliseconds over five checks for latency, 50 percent over ten checks for error rate, and error severity for schema drift. Cooldowns default to 30 minutes and run from 5 to 120. A more specific policy replaces the broader one rather than merging with it. The policy Routebase creates for a drift watch therefore brings a downtime rule along, so that the specification does not lose its downtime alerting.

Alerts reach the in-app notification centre, email per category, and Slack or Microsoft Teams as first-class targets with their own categories for contract drift and for incidents and recovery. See Alert Policies and Messaging.