Alerts & Notifications2 min read

    Creating Alert Escalation Policies

    By AmirReliability & Network Engineering
    Share

    What Are Escalation Policies?

    Escalation policies define a chain of actions when an alert isn't acknowledged within a specified time. They ensure that critical issues never go unnoticed.

    Creating a Policy

    Step 1: Define Levels

    Navigate to Settings → Escalation Policies → Create Policy.

    Each level specifies:

    • Who to notify (user or channel group)
    • How to notify (which notification channels)
    • Wait time before escalating to the next level

    Example Policy

    For a critical monitor — one where a failed check means user-facing impact — the first level has to use a channel that interrupts:

    Level Action Wait
    1 Call + SMS the on-call engineer 8 min
    2 Call + SMS the secondary on-call 7 min
    3 Notify the team lead via all channels 15 min
    4 Page the engineering manager

    Chat-first levels (Slack, Teams, Discord) belong on policies for monitors that don't warrant waking anyone — a background job, or a warning threshold with headroom left. Overnight, a chat message has a read latency measured in hours, so a critical alert that starts there has effectively not been delivered.

    Set each wait time above your realistic acknowledgement time. If it takes four minutes to surface from sleep, find a laptop, and open the alert, a three-minute wait escalates on every single incident — and within a month the secondary has learned that a level-2 page carries no information. For per-severity ladders and how to pick these numbers from your own acknowledgement data, see Incident Severity Levels: SEV1–SEV5 Explained.

    Step 2: Assign to Monitors

    Apply the policy to specific monitors or monitor groups. Different services can use different policies based on their criticality.

    Step 3: Set Acknowledgment Rules

    • Auto-resolve: Automatically close if the issue self-heals
    • Require acknowledgment: Someone must explicitly acknowledge the alert
    • Re-alert: Re-notify if the issue persists after acknowledgment

    On-Call Schedules

    Pair escalation policies with on-call schedules to route alerts to the right person based on the time of day and day of week.