Managing Detections
This guide covers monitoring, maintaining, and troubleshooting detections in production.
Managing Detections
This guide covers monitoring, maintaining, and troubleshooting detections in production.
Detection List View
The main detections page displays all configured detections with key information and controls.
Header Controls
Create New Detection
- Opens drawer to create a new detection
- See Creating Detections
Manage Destinations
- Opens destination management interface
- Disabled when no destinations exist
- Shows count of configured destinations
Detection Visualization
Four donut charts display detection distribution (when feature flag enabled):
Scheduled State
- Active vs Paused detections
- Click segment to filter list
Severity
- Distribution across LOW, MEDIUM, HIGH, CRITICAL
- Click segment to filter by severity
Detection Outcome
- Distribution of last run results
- Matched, Not Matched, Error, No Runs
- Click segment to filter by outcome
Destination Platform
- Distribution of alert destination platforms in use
- Click segment to filter detections using specific platform
Chart Interactions:
- Click chart segment to apply filter
- Click again to remove filter
- Multiple filters can be active simultaneously
- Active filters are visually highlighted
Search and Filter
Text Search:
- Searches detection name, description, scheduled state, severity, last outcome
- Case-insensitive partial matching
- Updates results in real-time
Chart Filters:
- Click chart segments to filter
- Combine multiple filters
- Clear individual filters by clicking segment again
Detection Grid
The grid displays detections with the following columns:
Detection
- Detection name (clickable)
- Click to open detail view
Scheduled State
- Editable dropdown: Active or Paused
- Green badge for Active
- Orange badge for Paused
- Click to change state inline
- State change triggers immediate API call
- Confirmation message displays on success
Severity
- Badge showing LOW, MEDIUM, HIGH, or CRITICAL
- Color-coded for quick identification
Recurrence
- Human-readable schedule description
- Hourly: "Hourly at :MM past the hour"
- Daily: "Daily at HH:MM UTC"
- Weekly: "Weekly on Day at HH:MM UTC"
Detection Outcome
- Badge showing result of last execution
- Matched: Detection triggered
- Not Matched: Detection ran but did not trigger
- Error: Execution failed
- No Runs: Detection has never executed
Last Run
- Timestamp of last execution
- Formatted as relative time (e.g., "2 hours ago")
- Hover for exact timestamp
Actions
- Run Now: Manually trigger detection immediately
- Overflow Menu (three dots):
- Edit: Open detection configuration
- View Details: Open read-only detail view
- Delete: Delete detection (with confirmation)
Detection States
Active
Behavior:
- Detection executes on schedule
- Alerts sent when triggered
- Appears in Active filter
Use When:
- Detection is ready for production
- Query is tested and validated
- Alert destinations are configured
Paused
Behavior:
- Detection does not execute on schedule
- Can still be run manually via "Run Now"
- Appears in Paused filter
Use When:
- Tuning detection query or threshold
- Maintenance window for data sources
- Temporarily high false positive rate
- Testing changes before re-enabling
Changing State
Inline Change:
- Click Scheduled State dropdown in grid
- Select Active or Paused
- State changes immediately
- Confirmation message displays
Bulk Operations:
- Not currently supported
- Change states individually
Detection Outcomes
Matched
Meaning: Detection found matches meeting threshold
Indicators:
- Green "Matched" badge
- Match count displayed
- Alerts sent to destinations
Actions:
- Click detection name to view details
- Click replay link to investigate matches
- Review alert destination responses
Not Matched
Meaning: Detection ran but found no matches (or below threshold)
Indicators:
- Gray "Not Matched" badge
- Match count: 0 or below threshold
Actions:
- Normal behavior for many detections
- If unexpected, review query and data sources
- Consider adjusting threshold
Error
Meaning: Detection execution failed
Indicators:
- Red "Error" badge
- Error details in detection outcome
Common Causes:
- Query syntax error
- Data source unavailable
- Timeout exceeded
- Authentication failure
Actions:
- Click detection name to view error details
- Review error message
- Test query in detection configuration
- Check data source connectivity
- Verify query syntax
- Adjust time window if timeout
No Runs
Meaning: Detection has never executed
Indicators:
- Gray "No Runs" badge
- No last run timestamp
Causes:
- Detection recently created
- Detection always paused
- Schedule not yet reached
Actions:
- Wait for scheduled execution
- Click "Run Now" to test immediately
- Verify detection is Active
Manual Execution
Run Now
Manually trigger detection execution immediately:
- Click Run Now button in Actions column
- Detection executes immediately (ignores schedule)
- Outcome recorded as MANUAL run type
- Alerts sent if detection matches
Use Cases:
- Test detection after creation
- Investigate current activity
- Verify alert destinations
- Test query changes before scheduling
Behavior:
- Runs regardless of Scheduled State (Active or Paused)
- Uses current detection configuration
- Executes with same time window as scheduled runs
- Outcome recorded in detection history
Detection Details View
View complete detection configuration and execution history:
- Click detection name in grid
- Read-only view displays:
- Detection configuration
- Schedule settings
- Threshold configuration
- Alert destinations
- Execution history
- Recent outcomes
Use For:
- Reviewing detection configuration
- Investigating execution history
- Sharing detection details with team
- Auditing detection behavior
Editing Detections
Edit Detection
- Click overflow menu (three dots) in Actions column
- Select Edit
- Detection configuration drawer opens
- Modify settings
- Click Save to apply changes
Changes Take Effect:
- Immediately for next scheduled run
- Does not affect in-progress executions
- Alert destinations update immediately
What Can Be Edited
- Detection name and description
- Scheduled state
- Severity
- FSQL query
- Trigger conditions (operator, threshold, eagerness)
- Schedule (frequency, time, day)
- Alert destinations
Best Practices
- Test Changes: Use "Run Now" to test before saving
- Pause First: Pause detection before major query changes
- Document Changes: Update description to reflect changes
- Monitor Results: Watch first few executions after changes
- Gradual Rollout: Test with lower severity first
Deleting Detections
Delete Detection
- Click overflow menu (three dots) in Actions column
- Select Delete
- Confirmation dialog displays detection name
- Click Confirm to delete
Confirmation Required: Prevents accidental deletion
What Happens:
- Detection permanently deleted
- Scheduled executions cancelled
- Historical outcomes preserved (in database)
- Alert destinations not affected
- Cannot be undone
Before Deleting:
- Verify detection is no longer needed
- Document reason for deletion
- Consider pausing instead if temporary
Monitoring Detection Health
Key Metrics
Execution Success Rate:
- Percentage of successful executions (Matched + Not Matched)
- Low rate indicates query or data source issues
Match Rate:
- Percentage of executions that matched
- High rate may indicate threshold too low
- Low rate may indicate threshold too high or rare condition
Error Rate:
- Percentage of executions that errored
- Any errors require investigation
Alert Delivery:
- Verify alerts reach destinations
- Check destination platform for delivery confirmation
Health Indicators
Healthy Detection:
- Executes on schedule
- Outcome: Matched or Not Matched (not Error)
- Alerts delivered successfully
- Match rate appropriate for detection type
Unhealthy Detection:
- Frequent errors
- Timeouts
- No executions (when Active)
- Alert delivery failures
Monitoring Workflow
- Daily Review: Check for Error outcomes
- Weekly Review: Review match rates and adjust thresholds
- Monthly Review: Evaluate detection effectiveness
- Continuous: Monitor alert destination delivery
Troubleshooting
Detection Not Executing
Symptoms: No Runs or Last Run timestamp not updating
Checks:
- Verify Scheduled State is Active
- Check schedule configuration (time, frequency)
- Verify detection was saved
- Check system status for outages
Resolution:
- Change to Active if Paused
- Correct schedule if misconfigured
- Use "Run Now" to test immediately
Detection Always Errors
Symptoms: Every execution results in Error outcome
Checks:
- Click detection name to view error details
- Test query in detection configuration
- Verify data sources are connected
- Check query syntax
- Review time window
Common Errors:
Query Syntax Error:
- Error message indicates syntax issue
- Test query and fix syntax
- Save and re-run
Data Source Unavailable:
- Error message indicates source not found
- Verify data source name
- Check data source connectivity
- Update query if source renamed
Query Timeout:
- Error message indicates timeout
- Reduce time window
- Add more specific filters
- Specify data sources in FROM clause
Authentication Failure:
- Error message indicates auth issue
- Verify API credentials
- Check data source permissions
Detection Never Matches
Symptoms: Always Not Matched, never Matched
Checks:
- Verify data exists in time window
- Test query manually
- Review filter conditions
- Check threshold configuration
Possible Causes:
No Data in Time Window:
- Data sources may not have recent data
- Adjust time window
- Verify data ingestion
Filters Too Restrictive:
- Query filters exclude all results
- Broaden filters
- Test with fewer conditions
Threshold Too High:
- Matches found but below threshold
- Review match counts in test results
- Lower threshold
Wrong Event Type:
- Query targets wrong event class
- Verify event type in data sources
- Adjust query to correct event class
Alerts Not Received
Symptoms: Detection matches but alerts not received
Checks:
- Verify destination is Active
- Verify destination is linked to detection
- Check destination configuration
- Test destination separately
- Review CloudWatch logs
Common Issues:
Destination Not Linked:
- Edit detection
- Go to Alert Configuration tab
- Select destination
- Save
Destination Inactive:
- Go to Manage Destinations
- Activate destination
- Verify status
Invalid Credentials:
- Check destination configuration
- Verify API keys, tokens, passwords
- Update secrets in Secrets Manager
- Test destination
Network Issues:
- Check destination platform status
- Verify webhook URLs
- Test connectivity
High False Positive Rate
Symptoms: Detection matches frequently with non-actionable alerts
Checks:
- Review recent matches
- Analyze common patterns
- Identify false positive causes
Solutions:
Adjust Threshold:
- Increase threshold to reduce noise
- Example: Change "greater than 0" to "greater than 5"
Add Filters:
- Exclude known benign activity
- Add user/host/IP exclusions
- Narrow time windows
Refine Query:
- Make conditions more specific
- Add additional AND conditions
- Use more precise field matching
Change Eagerness:
- Switch to Exhaustive for accurate counts
- Adjust threshold based on complete results
Best Practices
Detection Lifecycle
Development:
- Create detection in Paused state
- Test query thoroughly
- Run manually multiple times
- Verify alert destinations
- Activate when ready
Production:
- Monitor first few executions
- Adjust threshold based on results
- Document any tuning changes
- Review regularly for effectiveness
Maintenance:
- Review monthly for relevance
- Update queries as environment changes
- Adjust thresholds as needed
- Archive or delete obsolete detections
Naming Conventions
Detection Names:
- Use clear, descriptive names
- Include key indicators (e.g., "Failed SSH Logins >10")
- Keep under 25 characters
- Use consistent naming scheme
Destination Names:
- Include team or purpose (e.g., "SOC Team Slack")
- Include environment if applicable (e.g., "Prod PagerDuty")
- Use descriptive names for easy identification
Documentation
Detection Descriptions:
- Explain what detection monitors
- Document threshold rationale
- Note any exclusions or special cases
- Include runbook links if available
Change Log:
- Document significant changes
- Note threshold adjustments
- Record query modifications
- Track effectiveness over time
Alert Management
Severity Assignment:
- CRITICAL: Immediate threat, page on-call
- HIGH: Significant concern, notify team
- MEDIUM: Notable event, create ticket
- LOW: Awareness, log only
Destination Routing:
- Route by severity
- Route by detection type
- Route by team responsibility
- Avoid alert fatigue
Response Procedures:
- Document response steps
- Include investigation guides
- Link to runbooks
- Define escalation paths
Performance Optimization
Query Performance
Slow Queries:
- Specify data sources in FROM clause
- Add more restrictive filters
- Reduce time window
- Limit field selection
Expensive Queries:
- Use Eager mode for early termination
- Increase threshold to stop sooner
- Narrow time windows
- Target specific sources
Detection Scheduling
Load Distribution:
- Stagger detection schedules
- Avoid running all detections at same time
- Distribute across hour for hourly detections
Frequency Selection:
- Match frequency to threat urgency
- Don't over-schedule low-priority detections
- Consider data ingestion lag
Next Steps
- Alert Destinations - Configure alert routing
- FSQL Query Guide - Optimize detection queries
- Destination Setup Guides - Configure specific destinations
Updated 2 days ago