The page at 03:12 is the seventh of the night. The first six were a check flapping on a single region, a disk crossing 80% on a host that has sat at 79% for a year, and four notifications about the same failed deploy. By the seventh, acknowledging has stopped being a decision — it's a reflex performed half-asleep, and the person doing it could not tell you what the alert said.
That's alert fatigue, and it is usually described as a human problem: people are tired, people should care more. It isn't. It's a measurement problem. Every one of those six alerts is countable, attributable to a specific check, and deletable. Teams stay fatigued not because the noise is mysterious but because nobody owns the job of measuring it.
What alert fatigue actually costs
A study by PagerDuty found that 49% of on-call engineers experience alert fatigue, leading to slower response times and, ironically, more missed critical incidents.
The mechanism is worth being precise about, because it explains why "just be more careful" fails. Noise doesn't make people miss alerts by making them careless. It trains a rational response: when 90% of pages require no action, the fastest correct strategy for the other 10% is to acknowledge first and read later. That habit is invisible in your metrics until the one page that mattered gets the same treatment.
It shows up in three places you can already see:
- Acknowledgement time drifts upward, because the alert has to compete for attention. This is the MTTA and MTTR damage, and it's the easiest to quantify.
- Escalations stop meaning anything. Once the secondary is paged on routine noise, a genuine step-2 escalation carries no information.
- People leave the rotation. Off-hours interruption load is one of the most cited reasons engineers refuse on-call, and it costs more than any monitoring tool.
The fix isn't fewer monitors. Monitor everything you can afford to — the cost of alert fatigue comes from what you notify on, and those are separate decisions. Detection should be broad; notification should be narrow.
Diagnose the noise before you tune anything
Most alert-noise projects start with someone raising thresholds by instinct. Measure first — one week of alert history is enough to build this table for your own team:
| Signal | How to compute it | Healthy range | What a bad number means |
|---|---|---|---|
| Alerts per on-call shift | Total notifications ÷ shifts | Under 5 | The rotation is a queue, not an exception path |
| Actionability rate | Alerts that led to a human action ÷ total | Above 80% | Most pages are informational and shouldn't be pages |
| Self-resolving rate | Alerts that cleared before anyone responded ÷ total | Under 10% | You're alerting on transients; the system healed itself |
| Top-3 concentration | Share of volume from your three noisiest checks | Under 30% | A handful of checks is producing most of your fatigue |
| Off-hours pages per engineer per week | Pages outside business hours ÷ engineers | Under 2 | The rotation is unsustainable regardless of anything else |
| Repeat rate | Alerts firing on the same cause more than 3× in a week | Near zero | You have a known problem being re-announced instead of fixed |
The ranges are starting points to argue with, not laws. The number that matters most is top-3 concentration, because it's the one that makes the work tractable: in most teams a small number of checks generates the majority of the noise, so the first week of cleanup is a short list rather than a re-architecture.
The actionability test
Every alert has to answer one question:
Does this require a human to make a decision in the next N minutes?
There are only three honest answers, and they map to the three things you can actually do:
| Answer | What it is | Where it goes |
|---|---|---|
| Yes, now | An alert | Paging channel, on-call |
| Yes, but not tonight | A notification | Team chat, business hours |
| No | Information | Dashboard or ticket — not a notification |
The third row is where most fatigue is manufactured. "Deploy completed", "backup finished", "certificate renews in 30 days", "CPU at 75%" — all useful, none of them requiring a decision. They belong on a dashboard you look at deliberately, or in a weekly digest. Routing them to the same channel as a paging alert doesn't just add volume; it teaches people that the channel is low-stakes.
If an alert can't produce a specific answer to "what would you do about this right now?", it isn't an alert. Which severity it gets and which channel it goes to is a separate decision — that's the severity and routing question, and it only applies to the rows that survive this test.
The five sources of noise, and the fix for each
1. Transients treated as incidents
A single failed check could be a network hiccup between your probe and your server. Alerting on it is alerting on the internet's weather.
The fix is confirmation logic, not a higher threshold:
- Duration: a CPU spike lasting 10 seconds is normal; one lasting 10 minutes is a problem. Require the condition to persist.
- Consecutive failures: require 2–3 failed checks before notifying.
- Multi-region confirmation: for external checks, require failure from more than one probing node before declaring an outage. A single node failing is more likely to be about that node.
This alone usually removes most of the self-resolving rate above, and it costs you very little detection time — three 60-second checks is a three-minute delay on a real outage, against dozens of false pages avoided.
2. One cause, many alerts
A database primary goes down and you get eleven notifications: the database check, the six services that depend on it, the API latency alert, and two synthetic checks. Eleven pages, one incident, one action.
The fix is dependency awareness and grouping — suppress dependent alerts when their upstream is already failing, and collapse alerts sharing a cause into a single notification thread. Where your tooling can't express the dependency, alert on the user-facing symptom rather than on every component beneath it. One "checkout is failing" page is worth more than eleven component pages, and it's also the classification that severity levels are meant to be based on.
3. Alerts nobody owns
An alert with no owner is never tuned and never deleted. It fires, someone acknowledges it, and the cycle repeats for two years.
Every notifying check needs a named owner — a person or a team, recorded where the check is configured, not in a wiki nobody opens. The owner's job isn't to respond to every firing; it's to answer "should this still exist?" at review time. Unowned alerts are the ones that survive every cleanup, because deleting something nobody understands feels riskier than muting it.
4. Thresholds at the resource limit instead of at the risk of impact
"Disk at 90%" is a threshold set where the resource runs out, not where you need to act. On a slow-filling volume it's a month of warning; on a log partition during an incident it's four minutes — the same number, two completely different alerts.
Alert on the trend, not the level: projected time to exhaustion is the actionable form ("this volume fills in under 12 hours"). The same logic applies to error rates and latency — a static threshold either fires constantly during normal peaks or misses a slow degradation entirely. This is what anomaly detection is for: comparing behaviour against the pattern for that host at that hour rather than a number you guessed once.
5. Planned work
Deployments, migrations, and reboots generate exactly the signals you built alerts to catch. If planned work reaches the paging channel, you're training people to assume any alert during business hours is probably a deploy.
Maintenance windows suppress notifications for specific services during a defined period. The discipline that matters is scheduling them before the work, which means the deploy process has to include it — a maintenance window created after the pages go out has already done the damage.
Make the surviving alerts carry a diagnosis
An alert that says "Server is down" is barely useful. One that says "Server is down: disk /var/log is 100% full, causing MySQL to crash" tells you exactly what to fix.
This matters for fatigue and not just for speed. A vague alert forces the responder to open a laptop and investigate before they can even judge whether it's real, which makes every page expensive regardless of severity. An alert carrying the failing condition, the recent values, and a link to the check's history can often be triaged from a phone screen in fifteen seconds — and a cheap page is a page people don't resent.
Three things every notification should include: what failed (specific condition and value), what it affects (service, not just hostname), and where to look next (a link to the check, dashboard, or runbook). Root cause analysis turns alerts from symptoms into diagnoses.
The review ritual that keeps it from coming back
Cleanup is not a project you finish. Noise regrows every time someone adds a check during an incident and never revisits it. What keeps volume down is a recurring review with teeth:
- Every two weeks, list the top 10 alerts by volume. Not by severity — by count.
- Each one gets a verdict: fix, downgrade, or delete. "Leave it" is not on the list. If it fired ten times and produced no action, it isn't earning its place.
- Default to delete. An alert you deleted and later miss is one commit to restore. An alert that trains people to ignore their phone costs you the next real incident.
- Track alerts per shift as a number the team owns, the way you'd track error budget. When it rises, cleanup gets scheduled — not discussed.
Ten minutes every two weeks is enough. The reason to make it a ritual rather than a heroic quarterly effort is that the heroic version always loses to whatever is on fire that week.
A worked example
One team's alert volume over six weeks of doing exactly the above:
Week 0 Baseline
214 notifications / week, 38 to the paging channel
Actionability rate 21%. Top 3 checks = 61% of volume.
Week 1 Deleted 9 informational alerts (deploy + backup notices)
Moved certificate expiry to a weekly digest
214 -> 121 / week
Week 2 Added 3-consecutive-failure + 2-region confirmation to
external HTTP checks
Self-resolving rate 34% -> 6%
121 -> 74 / week
Week 3 Grouped 11 dependent service checks behind the database check
Assigned an owner to every remaining notifying check
74 -> 52 / week
Week 5 Replaced 4 static disk thresholds with time-to-full projection
Maintenance windows added to the deploy pipeline
52 -> 31 / week, 9 to the paging channel
Actionability rate 21% -> 84%. MTTA 14 min -> 4 min.
Zero incidents missed during the six weeks.
The MTTA improvement is the part worth noticing. Nothing about detection changed — the checks run at the same interval and catch the same failures. Acknowledgement got four times faster because a page became rare enough to be worth reading.
Mistakes that quietly make it worse
Muting instead of deleting. A muted alert is still configured, still owned by nobody, and still there for the next person to un-mute during an unrelated incident. Mute is a tool for the next six hours, not a decision.
Raising thresholds until nothing fires. This is the overcorrection, and it produces a monitoring setup that is quiet because it is blind. The goal is a high actionability rate, not a low alert count — one page a week that always matters beats zero pages and an outage your customers report first.
Adding a channel instead of removing an alert. When the paging channel gets noisy, the instinct is to route some alerts to a new Slack channel. Now you have two channels, the same volume, and one that nobody has agreed to watch. Fewer alerts, not more destinations. Which channel each severity uses is a routing decision, and it doesn't solve volume.
Alerting per host instead of per service. Twenty web servers behind a load balancer should not produce twenty alerts when a bad deploy hits all of them. Alert on the service-level symptom and let the per-host data live in the dashboard.
Treating acknowledgement as resolution. If acknowledging silences an alert permanently, people will acknowledge to make the noise stop and the underlying condition will persist. Acknowledgement should mean "I'm on it", with re-alerting if the condition is still true later.
Wiring it up in Xitoring
The controls that matter for noise are all configuration, not engineering. Triggers support duration and consecutive-failure conditions, so transients never reach a human. Checks confirm from multiple probing nodes before declaring an external failure. Escalation policies define the ladder and its wait times, so an unacknowledged alert climbs on its own instead of being broadcast to everyone at once. Maintenance windows keep planned work out of the alert stream, and anomaly detection covers the degradation that static thresholds either miss or over-report.
Channel assignment is where the routing table lives: over twenty notification channels mean informational alerts can go to chat or email while the genuinely urgent ones use something that interrupts. Plain-English root cause analysis puts the failing condition in the notification itself, which is what makes a page cheap to triage.
The free plan covers 8 checks without a credit card — enough to instrument the handful of user-facing flows that deserve to wake someone, which is also the fastest way to find out how much of your current alert volume was never about them.
Related reading
- Incident Severity Levels: SEV1–SEV5 Explained — the severity scale, escalation ladder, and channel matrix for the alerts that survive
- MTTR, MTTD, MTBF: Incident Metrics Explained — how alert noise shows up in your acknowledgement times
- Error Budgets Explained — deciding which alerts are worth waking someone for
- A Beginner's Guide to Real-Time Server Alerts — the fundamentals of threshold and alert setup
- Best Practices for Setting Up Server Monitoring — building the monitoring layer these alerts sit on
- What Is Uptime Monitoring? The 2026 Guide — where external detection fits
- Notification Channels — the delivery options available in Xitoring
Frequently Asked Questions
What is alert fatigue?
Alert fatigue is the desensitization that follows sustained exposure to alerts that don't require action. It isn't carelessness — it's an adaptation. When most pages need no response, acknowledging quickly and investigating later becomes the efficient strategy, and that habit then applies to the page that did matter. The practical definition is measurable rather than emotional: if your actionability rate is low and acknowledgement times are drifting up, you have alert fatigue regardless of how anyone feels about it.
How many alerts per week is too many?
Volume matters less than the share that need action. Under five notifications per on-call shift with an actionability rate above 80% is a healthy shape; twenty per shift where most need no action is not, even if none of them wake anyone. Two numbers to watch are off-hours pages per engineer per week — more than two is a retention problem — and the share of volume coming from your three noisiest checks, which tells you whether cleanup is a short list or a real project.
How do I reduce alert noise without missing incidents?
Cut non-actionable notifications first, since deleting a deployment notice cannot cause a missed incident. Then add confirmation logic — consecutive failures and multi-region agreement — which trades a couple of minutes of detection time for most of your false positives. Then group alerts that share a cause. Only after that should you touch thresholds on real conditions, and prefer trend-based conditions like projected time-to-full over static levels. Done in that order, the risky changes come last and against a much quieter baseline.
Should I delete an alert that caught a real incident once?
Usually yes, if it has produced dozens of non-actionable firings since. The relevant comparison isn't "did this ever help" but "does it help more than it costs" — and a noisy alert's cost is paid on every other alert, by making all of them less likely to be read. If the underlying condition genuinely matters, the answer is a better-formed alert on the user-facing symptom, not the preservation of a bad one that happened to fire on the right day.
Does alert fatigue affect MTTR?
It affects the acknowledgement stage most directly: when pages are routinely noise, they compete for attention and time-to-acknowledge rises. Because acknowledgement sits inside mean time to resolve, that delay lands in your MTTR without any change to how fast the fix itself happens — which is why cutting noise often improves response metrics more than any tooling change. Teams that measure the stages separately can see it clearly; teams that only track MTTR see a number that got worse for no visible reason.