Auto-Triggers

Setting triggers manually is time-consuming and often results in poorly tuned thresholds. Too sensitive triggers cause alert fatigue; too lenient triggers miss real problems. Auto-Triggers solves this by analyzing your actual metrics and recommending data-driven alert thresholds tailored to your infrastructure.

What Auto-Triggers Does

Automatic Trigger Creation

When you add a server, Auto-Triggers:

1. Learns your baseline - Collects 24 hours of metric data for accurate understanding
2. Analyzes patterns - Understands "normal" for YOUR infrastructure across different times of day
3. Calculates thresholds - Recommends alert values based on baselines (typically 2-3x normal)
4. Creates default triggers - Generates triggers for all key server metrics automatically
5. Assigns notifications - Connects to default Notification Role
6. Presents recommendations - Shows suggested triggers for review and adjustment

Data-driven, not guesswork - Thresholds based on what's actually happening on your systems, not generic industry averages.

Timeline

Be patient: Auto-Triggers needs 24 hours of baseline data to understand your infrastructure's natural patterns. Accepting recommendations too early (before 24h) results in poorly tuned triggers.

How Auto-Triggers Works

Baseline Learning

Auto-Triggers doesn't use generic thresholds—it learns from YOUR infrastructure:

24-hour observation period: Auto-Triggers collects full-day data to capture:

• Peak usage times (business hours typically higher CPU/memory)
• Low-usage periods (nights and weekends)
• Natural variation patterns (not just single moment in time)
• Weekly patterns (Mondays vs Fridays may differ)

Example - Server CPU Usage:

1. Observation period (first 24 hours):
   • Morning peak (8 AM): 70% CPU
   • Afternoon normal (2 PM): 50% CPU
   • Evening (6 PM): 45% CPU
   • Overnight low (2 AM): 15% CPU
   • Average across day: 45% CPU
2. Baseline established:
   • Normal CPU: ~45% (accounting for natural daily variation)
   • Peak acceptable: ~70% (observed peak during normal hours)
3. Auto-Trigger recommendations:
   • Alert if CPU sustains > 85% (genuinely anomalous, above peak)
   • Avoids false-positives during normal business-hour peaks

Why 24 hours matters: A 15-minute baseline might observe only business-hours peak. If you accept triggers too early, normal off-peak usage looks like an anomaly, or normal peaks get too-lenient thresholds.

Threshold Calculation Strategy

Auto-Triggers uses different multipliers for different metric types:

Conservative by default - Auto-Triggers errs on the side of fewer false positives. You can always tighten thresholds later.

What Gets Auto-Triggered

Server Monitoring Triggers (Currently Supported)

When you add a server, Auto-Triggers creates recommendations for:

System Metrics:

• CPU Usage - Alert if > baseline + 20-30%
• Memory Usage - Alert if > baseline + 30-40%
• Disk Usage - Alert if > 85% (critical at 95%)
• Disk I/O - Alert if read/write latency > 3x baseline
• Network Traffic - Alert if bandwidth > 5x baseline (possible attack or leak)

Process Monitoring:

• Process Count - Alert if critical process stops
• Service Status - Alert if systemd/Windows service down

See Server Monitoring for complete metrics list.

Uptime Check Triggers

When you create an uptime check, Auto-Triggers creates:

HTTP/HTTPS Checks:

• Alert if response time > 5000ms (5 seconds)
• Alert if status code ≠ 200 (or expected code)
• Alert if response doesn't contain expected text
• Alert if SSL certificate expires within 14 days

DNS Checks:

• Alert if DNS resolution fails
• Alert if resolved IP changes unexpectedly
• Alert if resolution time > 2000ms

TCP/UDP/Ping Checks:

• Alert if connection fails
• Alert if response time > baseline + 200%

Heartbeat Checks:

• Alert if expected ping not received within interval + grace period

See Uptime Monitoring for all check types.

Viewing Auto-Trigger Recommendations

Accessing Recommendations

For Servers:

1. Dashboard → Servers → [Your Server] → Triggers
2. Look for label "Auto-Recommended" or "Suggested"
3. Click "Review Recommendations"

For Uptime Checks:

1. Dashboard → Uptime → [Your Check] → Triggers
2. Auto-created trigger already exists (created immediately)
3. Recommendations for additional triggers shown if applicable

Recommendation States

Auto-Created & Active ✅

• Trigger automatically created when server/check added
• Already monitoring with default thresholds
• Edit anytime to customize

Recommended & Pending 🔵

• System suggests this trigger based on detected metrics
• Not yet active
• Click "Accept" to enable

Custom (User-Created) ⚙️

• You created this trigger manually
• Not part of Auto-Trigger system
• Full manual control

Accepting or Customizing Recommendations

One-Click Accept

To quickly enable recommended triggers:

1. Review the suggested trigger details (metric, threshold, notification role)
2. Click "Accept" or "Enable Recommended Trigger"
3. Trigger activates immediately
4. Starts monitoring for violations

When to accept as-is:

• First-time setup (optimize later after seeing real incidents)
• Standard infrastructure (baselines likely accurate)
• You trust the system's learning

Customize Before Accepting

To adjust thresholds before enabling:

1. Click "Edit" on recommended trigger
2. Modify threshold value (increase for less sensitivity, decrease for more)
3. Adjust Fault Tolerance (buffer time before alerting)
4. Change Notification Role (who gets alerted)
5. Save customized trigger

When to customize:

• You know your infrastructure's quirks (database always spikes at midnight during backups)
• Testing/dev servers (higher thresholds, fewer alerts)
• Critical production (tighter thresholds, faster alerting)

Bulk Accept

To enable all recommendations at once:

1. Triggers page → "Review All Recommendations"
2. Select multiple suggested triggers (checkbox)
3. Click "Accept Selected" or "Enable All"
4. All selected triggers activate simultaneously

When to bulk accept:

• New infrastructure deployment (enable everything, refine later)
• Standard stack across servers (recommendations consistent)
• Quick setup (optimize during operation)

Customizing Auto-Trigger Behavior

Adjusting Defaults

Change what Auto-Triggers creates by default:

1. Dashboard → Account Settings → Auto-Trigger Preferences
2. Configure:
   • Enable/Disable Auto-Triggers - Turn feature on/off globally
   • Default Notification Role - Which role receives auto-created trigger alerts
   • Sensitivity Level - Conservative (fewer alerts) vs Aggressive (more alerts)
   • Metric Selection - Which metrics get auto-triggered (enable/disable)

Per-Server Configuration

When adding a new server:

1. "Add Server" form → Options section
2. Auto-Trigger checkbox - Enable or disable for this server
3. If disabled, no automatic triggers created (manual setup required)

When to disable:

• Test/dev servers (don't need alerts)
• Decommissioned servers (about to remove)
• Custom monitoring setup (you'll create triggers manually)

Per-Check Configuration

When creating uptime check:

1. Check creation form → Advanced Options
2. Auto-Trigger - Enabled by default, can disable
3. If disabled, you create triggers manually after check creation

Understanding Trigger Components

What's in an Auto-Triggered Alert

Every auto-created trigger includes:

1. Condition - What to monitor

• Example: "CPU Usage"

2. Threshold - When to alert

• Example: "> 85%"

3. Fault Tolerance - Buffer before alerting (default: 5 minutes)

• Why: Prevents alerts for brief spikes
• See FAQ: Understanding Fault Tolerance

4. Notification Role - Who gets alerted

• Default: Account default notification role
• Customize per trigger

5. Severity - Priority level

• Critical, Warning, Info
• Auto-assigned based on metric type

Editing Auto-Created Triggers

All auto-created triggers are fully editable:

1. Navigate to trigger (Servers → [Server] → Triggers → [Trigger Name])
2. Click "Edit"
3. Modify any parameter:
   • Change threshold value
   • Adjust Fault Tolerance
   • Change notification role
   • Add/remove conditions
   • Enable/disable trigger
4. Save changes

No restrictions - Once created, auto-triggers behave identically to manual triggers. Full control.

Best Practices

After Auto-Triggers Created (after ~24 hours)

1. Review all auto-created triggers - Verify they make sense for your use case
2. Test one trigger - Manually violate threshold to confirm alerts work
3. Adjust Fault Tolerance - If you get false positives, increase FT; if missing incidents, decrease FT
4. Customize critical servers - Tighten thresholds for production, loosen for dev/test
5. Document exceptions - Note why you disabled/customized specific triggers
6. Manual triggers for integrations - Create custom triggers for integration metrics (MySQL, Nginx, etc.) based on Auto-Triggers recommendations as starting points

For Large Deployments

When monitoring many servers:

• Let Auto-Triggers run on first server
• Review and refine triggers for 1-2 days
• Document your customizations (thresholds, FT, roles)
• Apply same customizations to subsequent servers via API or Ansible
• Or accept all Auto-Triggers and refine after observing incident patterns

For Different Server Types

Customize Auto-Trigger strategy by server role:

When Auto-Triggers Helps vs Manual Setup

Perfect for Auto-Triggers

✅ Standard infrastructure - Web servers, databases, common stacks
✅ Unknown baselines - You don't know what "normal" looks like yet
✅ Quick deployment - Need monitoring operational fast
✅ Many servers - Don't have time to configure 50 servers manually
✅ Learning mode - Use Auto-Triggers as starting point, refine over time

Consider Manual Triggers When

⚠️ Highly custom applications - Proprietary metrics Auto-Triggers doesn't understand
⚠️ Known thresholds - You already know exact values (like SLA requirements: "response time must be < 200ms")
⚠️ Complex logic - Need composite conditions (Alert if CPU > 80% AND memory > 90% AND disk I/O high)
⚠️ External requirements - Compliance mandates specific threshold values

Best approach: Use Auto-Triggers for standard metrics, add custom manual triggers for edge cases.

Troubleshooting

No Auto-Trigger Recommendations Appearing

Symptoms: Server added, but no recommended triggers shown after 20+ minutes

Solutions:

1. Verify Auto-Triggers enabled:
   • Account Settings → Auto-Trigger Preferences → Ensure enabled globally
   • Server settings → Ensure Auto-Trigger not disabled for this specific server
2. Check baseline data:
   • Servers → [Your Server] → Metrics
   • Verify graphs showing data (if no data, Auto-Triggers can't recommend thresholds)
3. Wait longer:
   • Auto-Triggers needs 15+ minutes of data
   • Refresh page, recommendations may have just appeared
4. Manually create triggers:
   • If recommendations not appearing, create triggers manually
   • System may not have enough data variation to establish baseline

Auto-Created Triggers Too Sensitive (False Positives)

Symptoms: Getting alerts for normal behavior, not real incidents

Solutions:

1. Increase Fault Tolerance:
   • Edit trigger → Change FT from 5 min to 10 or 15 min
   • Gives more buffer before alerting
2. Raise threshold:
   • If trigger alerts at CPU > 80%, change to > 90%
   • Find balance between sensitivity and usefulness
3. Disable specific trigger:
   • If trigger consistently false positive, disable it
   • Not every metric needs active monitoring
4. Wait for re-tuning:
   • Auto-Triggers continuously learns
   • After more data collected, may auto-adjust recommendations

See FAQ: How can I reduce unnecessary notifications for more strategies.

Auto-Created Triggers Missing Real Incidents

Symptoms: Problems occurred but no alerts sent

Solutions:

1. Lower threshold:
   • If CPU spiked to 95% but trigger set at "> 95%", lower to "> 85%"
   • More aggressive = catch issues earlier
2. Decrease Fault Tolerance:
   • Change FT from 10 min to 3-5 min
   • Alert faster when issues occur
3. Add additional triggers:
   • Auto-Triggers creates common triggers, not exhaustive list
   • Manually add triggers for edge cases
4. Verify notification delivery:
   • Check notification role configured correctly
   • Test alert delivery: FAQ: Why are my notifications NOT arriving?

Disabling Auto-Triggers

Global Disable

Turn off Auto-Triggers for all future servers/checks:

1. Account Settings → Auto-Trigger Preferences
2. Toggle "Enable Auto-Triggers" OFF
3. Save settings

Impact:

• New servers: No automatic triggers created
• New uptime checks: No automatic triggers created
• Existing triggers: Remain active (not deleted)
• You must create all triggers manually going forward

Per-Resource Disable

Disable for specific server or check:

When adding server:

• "Add Server" form → Options → Uncheck "Auto-Trigger"

For existing server:

• Servers → [Server] → Settings → Auto-Trigger → Disable
• Prevents future auto-recommendations (doesn't delete existing triggers)

When creating uptime check:

• Check creation → Advanced Options → Uncheck "Auto-Trigger"

Common Scenarios

Scenario 1: First Production Server

Timeline:

• 0:00 - Install Xitogent, server registered
• 0:02 - First metrics arrive
• 0:15 - Auto-Trigger recommendations appear (8 triggers suggested)
• 0:16 - Review recommendations: all look reasonable
• 0:16 - Bulk accept all 8 triggers
• 0:17 - Test one trigger (manually spike CPU to verify alerting works)
• 0:18 - Alert received successfully
• Total: 18 minutes from zero to fully monitored with active alerts

Scenario 2: High-Traffic Web Server

Challenge: Web server normally uses 60-70% CPU during business hours

Auto-Trigger recommendation: Alert if CPU > 85% (based on learning 70% baseline)

Your decision:

• Accept: 85% threshold gives good buffer above normal 70%
• Leave Fault Tolerance at 5 minutes (prevents alerts during brief request spikes)
• Assign to "Production-Critical" notification role (alerts ops team)

Result: Alert only when CPU sustains > 85% for 5+ minutes—real problems, not normal traffic peaks.

Scenario 3: Database with Backup Spike

Challenge: MySQL server spikes to 90% CPU every night at 2 AM during backups

Auto-Trigger recommendation: Alert if CPU > 80%

Your decision:

• Problem: Will alert every night during known maintenance
• Solution Option 1: Increase threshold to > 95% (only alert for true anomalies)
• Solution Option 2: Create Maintenance Schedule for 2-3 AM (silences alerts during backup window)
• Solution Option 3: Disable CPU trigger, create custom trigger with time-based conditions

Result: No false positives during scheduled maintenance, still catch unexpected CPU spikes.

Scenario 4: Dev/Test Environment

Challenge: Test servers have erratic behavior (load testing, experiments, deployments)

Auto-Trigger recommendation: 15 triggers for various metrics

Your decision:

• Disable most triggers (too many false positives in test environment)
• Keep only: - Disk space trigger (prevent fill-up)
  • Service down trigger (catch accidental service stops)
• Increase FT to 30 minutes (only alert for sustained issues)

Result: Minimal alert noise from test environment, still catch true problems.

See Also

• Auto-Discovery - Automatic service detection that feeds Auto-Triggers
• Notification Roles - Configure who receives Auto-Trigger alerts
• Fault Tolerance - Understanding the alert buffer
• FAQ: Understanding Triggers - Trigger fundamentals
• FAQ: What are Auto-Triggers? - Quick overview
• Server Monitoring - All metrics that can be auto-triggered
• Glossary: Auto-Trigger - Term definition

Next Step: After Auto-Triggers are active, use Auto Fault Tolerance to dynamically reduce false alerts during instability!