Auto-Triggers
Setting triggers manually is time-consuming and often results in poorly tuned thresholds. Too sensitive triggers cause alert fatigue; too lenient triggers miss real problems. Auto-Triggers solves this by analyzing your actual metrics and recommending data-driven alert thresholds tailored to your infrastructure.
Time Savings
Manual trigger setup: 2-3 hours (guessing thresholds, tuning after false positives)
Auto-Triggers: 10 minutes (review and accept recommendations)
Result: Spend time responding to real incidents, not configuring triggers.
What Auto-Triggers Does
Automatic Trigger Creation
When you add a server, Auto-Triggers:
- Learns your baseline - Collects 24 hours of metric data for accurate understanding
- Analyzes patterns - Understands "normal" for YOUR infrastructure across different times of day
- Calculates thresholds - Recommends alert values based on baselines (typically 2-3x normal)
- Creates default triggers - Generates triggers for all key server metrics automatically
- Assigns notifications - Connects to default Notification Role
- Presents recommendations - Shows suggested triggers for review and adjustment
Data-driven, not guesswork - Thresholds based on what's actually happening on your systems, not generic industry averages.
Timeline
| Milestone | Timing | What's Happening |
|---|---|---|
| Server added | Instant | Xitogent starts collecting metrics |
| First data arrives | 2-5 minutes | Initial metrics flow to dashboard |
| Baseline established | 24 hours | System observes full day of normal behavior including peak/low periods |
| Auto-Trigger recommendations appear | ~24 hours | Suggested triggers ready for review after sufficient baseline data |
| Accept recommendations | < 1 minute | One-click activation |
| Triggers active | Immediately | Monitoring and alerting operational |
Be patient: Auto-Triggers needs 24 hours of baseline data to understand your infrastructure's natural patterns. Accepting recommendations too early (before 24h) results in poorly tuned triggers.
How Auto-Triggers Works
Baseline Learning
Auto-Triggers doesn't use generic thresholds—it learns from YOUR infrastructure:
24-hour observation period: Auto-Triggers collects full-day data to capture:
- Peak usage times (business hours typically higher CPU/memory)
- Low-usage periods (nights and weekends)
- Natural variation patterns (not just single moment in time)
- Weekly patterns (Mondays vs Fridays may differ)
Example - Server CPU Usage:
- Observation period (first 24 hours):
- Morning peak (8 AM): 70% CPU
- Afternoon normal (2 PM): 50% CPU
- Evening (6 PM): 45% CPU
- Overnight low (2 AM): 15% CPU
- Average across day: 45% CPU
- Baseline established:
- Normal CPU: ~45% (accounting for natural daily variation)
- Peak acceptable: ~70% (observed peak during normal hours)
- Auto-Trigger recommendations:
- Alert if CPU sustains > 85% (genuinely anomalous, above peak)
- Avoids false-positives during normal business-hour peaks
Why 24 hours matters: A 15-minute baseline might observe only business-hours peak. If you accept triggers too early, normal off-peak usage looks like an anomaly, or normal peaks get too-lenient thresholds.
Threshold Calculation Strategy
Auto-Triggers uses different multipliers for different metric types:
| Metric Type | Baseline Multiplier | Reasoning |
|---|---|---|
| CPU Usage | 1.2-1.5x | Small increases significant (80% → 95% critical) |
| Memory Usage | 1.3-1.7x | Gradual growth expected, but sustained spikes matter |
| Disk Space | 1.05-1.1x | Filling disk dangerous, alert early |
| Database Connections | 2-3x | Connection spikes normal under load |
| HTTP Response Time | 3-5x | Temporary slowness expected, sustained slow is problem |
| Error Rates | 5-10x | Even small error rate increases worth investigating |
Conservative by default - Auto-Triggers errs on the side of fewer false positives. You can always tighten thresholds later.
What Gets Auto-Triggered
Server Monitoring Triggers (Currently Supported)
When you add a server, Auto-Triggers creates recommendations for:
System Metrics:
- CPU Usage - Alert if > baseline + 20-30%
- Memory Usage - Alert if > baseline + 30-40%
- Disk Usage - Alert if > 85% (critical at 95%)
- Disk I/O - Alert if read/write latency > 3x baseline
- Network Traffic - Alert if bandwidth > 5x baseline (possible attack or leak)
Process Monitoring:
- Process Count - Alert if critical process stops
- Service Status - Alert if systemd/Windows service down
Current Limitation
Auto-Triggers currently supports server metrics only. Integration metrics (MySQL, Nginx, Redis, etc.) are not yet included in automatic trigger creation. Create integration triggers manually or use recommended thresholds as a starting point for custom configuration.
Roadmap: Integration support coming in future releases.
See Server Monitoring for complete metrics list.
Uptime Check Triggers
When you create an uptime check, Auto-Triggers creates:
HTTP/HTTPS Checks:
- Alert if response time > 5000ms (5 seconds)
- Alert if status code ≠ 200 (or expected code)
- Alert if response doesn't contain expected text
- Alert if SSL certificate expires within 14 days
DNS Checks:
- Alert if DNS resolution fails
- Alert if resolved IP changes unexpectedly
- Alert if resolution time > 2000ms
TCP/UDP/Ping Checks:
- Alert if connection fails
- Alert if response time > baseline + 200%
Heartbeat Checks:
- Alert if expected ping not received within interval + grace period
See Uptime Monitoring for all check types.
Viewing Auto-Trigger Recommendations
Accessing Recommendations
For Servers:
- Dashboard → Servers → [Your Server] → Triggers
- Look for label "Auto-Recommended" or "Suggested"
- Click "Review Recommendations"
For Uptime Checks:
- Dashboard → Uptime → [Your Check] → Triggers
- Auto-created trigger already exists (created immediately)
- Recommendations for additional triggers shown if applicable
Recommendation States
Auto-Created & Active ✅
- Trigger automatically created when server/check added
- Already monitoring with default thresholds
- Edit anytime to customize
Recommended & Pending 🔵
- System suggests this trigger based on detected metrics
- Not yet active
- Click "Accept" to enable
Custom (User-Created) ⚙️
- You created this trigger manually
- Not part of Auto-Trigger system
- Full manual control
Accepting or Customizing Recommendations
One-Click Accept
To quickly enable recommended triggers:
- Review the suggested trigger details (metric, threshold, notification role)
- Click "Accept" or "Enable Recommended Trigger"
- Trigger activates immediately
- Starts monitoring for violations
When to accept as-is:
- First-time setup (optimize later after seeing real incidents)
- Standard infrastructure (baselines likely accurate)
- You trust the system's learning
Customize Before Accepting
To adjust thresholds before enabling:
- Click "Edit" on recommended trigger
- Modify threshold value (increase for less sensitivity, decrease for more)
- Adjust Fault Tolerance (buffer time before alerting)
- Change Notification Role (who gets alerted)
- Save customized trigger
When to customize:
- You know your infrastructure's quirks (database always spikes at midnight during backups)
- Testing/dev servers (higher thresholds, fewer alerts)
- Critical production (tighter thresholds, faster alerting)
Bulk Accept
To enable all recommendations at once:
- Triggers page → "Review All Recommendations"
- Select multiple suggested triggers (checkbox)
- Click "Accept Selected" or "Enable All"
- All selected triggers activate simultaneously
When to bulk accept:
- New infrastructure deployment (enable everything, refine later)
- Standard stack across servers (recommendations consistent)
- Quick setup (optimize during operation)
Customizing Auto-Trigger Behavior
Adjusting Defaults
Change what Auto-Triggers creates by default:
- Dashboard → Account Settings → Auto-Trigger Preferences
- Configure:
- Enable/Disable Auto-Triggers - Turn feature on/off globally
- Default Notification Role - Which role receives auto-created trigger alerts
- Sensitivity Level - Conservative (fewer alerts) vs Aggressive (more alerts)
- Metric Selection - Which metrics get auto-triggered (enable/disable)
Per-Server Configuration
When adding a new server:
- "Add Server" form → Options section
- Auto-Trigger checkbox - Enable or disable for this server
- If disabled, no automatic triggers created (manual setup required)
When to disable:
- Test/dev servers (don't need alerts)
- Decommissioned servers (about to remove)
- Custom monitoring setup (you'll create triggers manually)
Per-Check Configuration
When creating uptime check:
- Check creation form → Advanced Options
- Auto-Trigger - Enabled by default, can disable
- If disabled, you create triggers manually after check creation
Understanding Trigger Components
What's in an Auto-Triggered Alert
Every auto-created trigger includes:
1. Condition - What to monitor
- Example: "CPU Usage"
2. Threshold - When to alert
- Example: "> 85%"
3. Fault Tolerance - Buffer before alerting (default: 5 minutes)
- Why: Prevents alerts for brief spikes
- See FAQ: Understanding Fault Tolerance
4. Notification Role - Who gets alerted
- Default: Account default notification role
- Customize per trigger
5. Severity - Priority level
- Critical, Warning, Info
- Auto-assigned based on metric type
Editing Auto-Created Triggers
All auto-created triggers are fully editable:
- Navigate to trigger (Servers → [Server] → Triggers → [Trigger Name])
- Click "Edit"
- Modify any parameter:
- Change threshold value
- Adjust Fault Tolerance
- Change notification role
- Add/remove conditions
- Enable/disable trigger
- Save changes
No restrictions - Once created, auto-triggers behave identically to manual triggers. Full control.
Best Practices
After Auto-Triggers Created (after ~24 hours)
- Review all auto-created triggers - Verify they make sense for your use case
- Test one trigger - Manually violate threshold to confirm alerts work
- Adjust Fault Tolerance - If you get false positives, increase FT; if missing incidents, decrease FT
- Customize critical servers - Tighten thresholds for production, loosen for dev/test
- Document exceptions - Note why you disabled/customized specific triggers
- Manual triggers for integrations - Create custom triggers for integration metrics (MySQL, Nginx, etc.) based on Auto-Triggers recommendations as starting points
For Large Deployments
When monitoring many servers:
- Let Auto-Triggers run on first server
- Review and refine triggers for 1-2 days
- Document your customizations (thresholds, FT, roles)
- Apply same customizations to subsequent servers via API or Ansible
- Or accept all Auto-Triggers and refine after observing incident patterns
For Different Server Types
Customize Auto-Trigger strategy by server role:
| Server Type | Strategy |
|---|---|
| Production database | Aggressive triggers (tight thresholds, FT = 1-2 min) |
| Production web | Standard triggers (accept recommendations, FT = 5 min) |
| Development | Loose triggers (2x recommended thresholds, FT = 15 min) |
| Testing | Minimal triggers (disable most, keep only critical) |
| CI/CD runners | Custom triggers (ignore CPU spikes during builds) |
When Auto-Triggers Helps vs Manual Setup
Perfect for Auto-Triggers
✅ Standard infrastructure - Web servers, databases, common stacks
✅ Unknown baselines - You don't know what "normal" looks like yet
✅ Quick deployment - Need monitoring operational fast
✅ Many servers - Don't have time to configure 50 servers manually
✅ Learning mode - Use Auto-Triggers as starting point, refine over time
Consider Manual Triggers When
⚠️ Highly custom applications - Proprietary metrics Auto-Triggers doesn't understand
⚠️ Known thresholds - You already know exact values (like SLA requirements: "response time must be < 200ms")
⚠️ Complex logic - Need composite conditions (Alert if CPU > 80% AND memory > 90% AND disk I/O high)
⚠️ External requirements - Compliance mandates specific threshold values
Best approach: Use Auto-Triggers for standard metrics, add custom manual triggers for edge cases.
Troubleshooting
No Auto-Trigger Recommendations Appearing
Symptoms: Server added, but no recommended triggers shown after 20+ minutes
Solutions:
- Verify Auto-Triggers enabled:
- Account Settings → Auto-Trigger Preferences → Ensure enabled globally
- Server settings → Ensure Auto-Trigger not disabled for this specific server
- Check baseline data:
- Servers → [Your Server] → Metrics
- Verify graphs showing data (if no data, Auto-Triggers can't recommend thresholds)
- Wait longer:
- Auto-Triggers needs 15+ minutes of data
- Refresh page, recommendations may have just appeared
- Manually create triggers:
- If recommendations not appearing, create triggers manually
- System may not have enough data variation to establish baseline
Auto-Created Triggers Too Sensitive (False Positives)
Symptoms: Getting alerts for normal behavior, not real incidents
Solutions:
- Increase Fault Tolerance:
- Edit trigger → Change FT from 5 min to 10 or 15 min
- Gives more buffer before alerting
- Raise threshold:
- If trigger alerts at CPU > 80%, change to > 90%
- Find balance between sensitivity and usefulness
- Disable specific trigger:
- If trigger consistently false positive, disable it
- Not every metric needs active monitoring
- Wait for re-tuning:
- Auto-Triggers continuously learns
- After more data collected, may auto-adjust recommendations
See FAQ: How can I reduce unnecessary notifications for more strategies.
Auto-Created Triggers Missing Real Incidents
Symptoms: Problems occurred but no alerts sent
Solutions:
- Lower threshold:
- If CPU spiked to 95% but trigger set at "> 95%", lower to "> 85%"
- More aggressive = catch issues earlier
- Decrease Fault Tolerance:
- Change FT from 10 min to 3-5 min
- Alert faster when issues occur
- Add additional triggers:
- Auto-Triggers creates common triggers, not exhaustive list
- Manually add triggers for edge cases
- Verify notification delivery:
- Check notification role configured correctly
- Test alert delivery: FAQ: Why are my notifications NOT arriving?
Disabling Auto-Triggers
Global Disable
Turn off Auto-Triggers for all future servers/checks:
- Account Settings → Auto-Trigger Preferences
- Toggle "Enable Auto-Triggers" OFF
- Save settings
Impact:
- New servers: No automatic triggers created
- New uptime checks: No automatic triggers created
- Existing triggers: Remain active (not deleted)
- You must create all triggers manually going forward
Per-Resource Disable
Disable for specific server or check:
When adding server:
- "Add Server" form → Options → Uncheck "Auto-Trigger"
For existing server:
- Servers → [Server] → Settings → Auto-Trigger → Disable
- Prevents future auto-recommendations (doesn't delete existing triggers)
When creating uptime check:
- Check creation → Advanced Options → Uncheck "Auto-Trigger"
Common Scenarios
Scenario 1: First Production Server
Timeline:
- 0:00 - Install Xitogent, server registered
- 0:02 - First metrics arrive
- 0:15 - Auto-Trigger recommendations appear (8 triggers suggested)
- 0:16 - Review recommendations: all look reasonable
- 0:16 - Bulk accept all 8 triggers
- 0:17 - Test one trigger (manually spike CPU to verify alerting works)
- 0:18 - Alert received successfully
- Total: 18 minutes from zero to fully monitored with active alerts
Scenario 2: High-Traffic Web Server
Challenge: Web server normally uses 60-70% CPU during business hours
Auto-Trigger recommendation: Alert if CPU > 85% (based on learning 70% baseline)
Your decision:
- Accept: 85% threshold gives good buffer above normal 70%
- Leave Fault Tolerance at 5 minutes (prevents alerts during brief request spikes)
- Assign to "Production-Critical" notification role (alerts ops team)
Result: Alert only when CPU sustains > 85% for 5+ minutes—real problems, not normal traffic peaks.
Scenario 3: Database with Backup Spike
Challenge: MySQL server spikes to 90% CPU every night at 2 AM during backups
Auto-Trigger recommendation: Alert if CPU > 80%
Your decision:
- Problem: Will alert every night during known maintenance
- Solution Option 1: Increase threshold to > 95% (only alert for true anomalies)
- Solution Option 2: Create Maintenance Schedule for 2-3 AM (silences alerts during backup window)
- Solution Option 3: Disable CPU trigger, create custom trigger with time-based conditions
Result: No false positives during scheduled maintenance, still catch unexpected CPU spikes.
Scenario 4: Dev/Test Environment
Challenge: Test servers have erratic behavior (load testing, experiments, deployments)
Auto-Trigger recommendation: 15 triggers for various metrics
Your decision:
- Disable most triggers (too many false positives in test environment)
- Keep only: - Disk space trigger (prevent fill-up)
- Service down trigger (catch accidental service stops)
- Increase FT to 30 minutes (only alert for sustained issues)
Result: Minimal alert noise from test environment, still catch true problems.
See Also
- Auto-Discovery - Automatic service detection that feeds Auto-Triggers
- Notification Roles - Configure who receives Auto-Trigger alerts
- Fault Tolerance - Understanding the alert buffer
- FAQ: Understanding Triggers - Trigger fundamentals
- FAQ: What are Auto-Triggers? - Quick overview
- Server Monitoring - All metrics that can be auto-triggered
- Glossary: Auto-Trigger - Term definition
Next Step: After Auto-Triggers are active, use Auto Fault Tolerance to dynamically reduce false alerts during instability!