Managing Detections

This guide covers monitoring, maintaining, and troubleshooting detections in production.

Managing Detections

This guide covers monitoring, maintaining, and troubleshooting detections in production.

Detection List View

The main detections page displays all configured detections with key information and controls.

Header Controls

Create New Detection

Manage Destinations

  • Opens destination management interface
  • Disabled when no destinations exist
  • Shows count of configured destinations

Detection Visualization

Four donut charts display detection distribution (when feature flag enabled):

Scheduled State

  • Active vs Paused detections
  • Click segment to filter list

Severity

  • Distribution across LOW, MEDIUM, HIGH, CRITICAL
  • Click segment to filter by severity

Detection Outcome

  • Distribution of last run results
  • Matched, Not Matched, Error, No Runs
  • Click segment to filter by outcome

Destination Platform

  • Distribution of alert destination platforms in use
  • Click segment to filter detections using specific platform

Chart Interactions:

  • Click chart segment to apply filter
  • Click again to remove filter
  • Multiple filters can be active simultaneously
  • Active filters are visually highlighted

Search and Filter

Text Search:

  • Searches detection name, description, scheduled state, severity, last outcome
  • Case-insensitive partial matching
  • Updates results in real-time

Chart Filters:

  • Click chart segments to filter
  • Combine multiple filters
  • Clear individual filters by clicking segment again

Detection Grid

The grid displays detections with the following columns:

Detection

  • Detection name (clickable)
  • Click to open detail view

Scheduled State

  • Editable dropdown: Active or Paused
  • Green badge for Active
  • Orange badge for Paused
  • Click to change state inline
  • State change triggers immediate API call
  • Confirmation message displays on success

Severity

  • Badge showing LOW, MEDIUM, HIGH, or CRITICAL
  • Color-coded for quick identification

Recurrence

  • Human-readable schedule description
  • Hourly: "Hourly at :MM past the hour"
  • Daily: "Daily at HH:MM UTC"
  • Weekly: "Weekly on Day at HH:MM UTC"

Detection Outcome

  • Badge showing result of last execution
  • Matched: Detection triggered
  • Not Matched: Detection ran but did not trigger
  • Error: Execution failed
  • No Runs: Detection has never executed

Last Run

  • Timestamp of last execution
  • Formatted as relative time (e.g., "2 hours ago")
  • Hover for exact timestamp

Actions

  • Run Now: Manually trigger detection immediately
  • Overflow Menu (three dots):
    • Edit: Open detection configuration
    • View Details: Open read-only detail view
    • Delete: Delete detection (with confirmation)

Detection States

Active

Behavior:

  • Detection executes on schedule
  • Alerts sent when triggered
  • Appears in Active filter

Use When:

  • Detection is ready for production
  • Query is tested and validated
  • Alert destinations are configured

Paused

Behavior:

  • Detection does not execute on schedule
  • Can still be run manually via "Run Now"
  • Appears in Paused filter

Use When:

  • Tuning detection query or threshold
  • Maintenance window for data sources
  • Temporarily high false positive rate
  • Testing changes before re-enabling

Changing State

Inline Change:

  1. Click Scheduled State dropdown in grid
  2. Select Active or Paused
  3. State changes immediately
  4. Confirmation message displays

Bulk Operations:

  • Not currently supported
  • Change states individually

Detection Outcomes

Matched

Meaning: Detection found matches meeting threshold

Indicators:

  • Green "Matched" badge
  • Match count displayed
  • Alerts sent to destinations

Actions:

  • Click detection name to view details
  • Click replay link to investigate matches
  • Review alert destination responses

Not Matched

Meaning: Detection ran but found no matches (or below threshold)

Indicators:

  • Gray "Not Matched" badge
  • Match count: 0 or below threshold

Actions:

  • Normal behavior for many detections
  • If unexpected, review query and data sources
  • Consider adjusting threshold

Error

Meaning: Detection execution failed

Indicators:

  • Red "Error" badge
  • Error details in detection outcome

Common Causes:

  • Query syntax error
  • Data source unavailable
  • Timeout exceeded
  • Authentication failure

Actions:

  1. Click detection name to view error details
  2. Review error message
  3. Test query in detection configuration
  4. Check data source connectivity
  5. Verify query syntax
  6. Adjust time window if timeout

No Runs

Meaning: Detection has never executed

Indicators:

  • Gray "No Runs" badge
  • No last run timestamp

Causes:

  • Detection recently created
  • Detection always paused
  • Schedule not yet reached

Actions:

  • Wait for scheduled execution
  • Click "Run Now" to test immediately
  • Verify detection is Active

Manual Execution

Run Now

Manually trigger detection execution immediately:

  1. Click Run Now button in Actions column
  2. Detection executes immediately (ignores schedule)
  3. Outcome recorded as MANUAL run type
  4. Alerts sent if detection matches

Use Cases:

  • Test detection after creation
  • Investigate current activity
  • Verify alert destinations
  • Test query changes before scheduling

Behavior:

  • Runs regardless of Scheduled State (Active or Paused)
  • Uses current detection configuration
  • Executes with same time window as scheduled runs
  • Outcome recorded in detection history

Detection Details View

View complete detection configuration and execution history:

  1. Click detection name in grid
  2. Read-only view displays:
    • Detection configuration
    • Schedule settings
    • Threshold configuration
    • Alert destinations
    • Execution history
    • Recent outcomes

Use For:

  • Reviewing detection configuration
  • Investigating execution history
  • Sharing detection details with team
  • Auditing detection behavior

Editing Detections

Edit Detection

  1. Click overflow menu (three dots) in Actions column
  2. Select Edit
  3. Detection configuration drawer opens
  4. Modify settings
  5. Click Save to apply changes

Changes Take Effect:

  • Immediately for next scheduled run
  • Does not affect in-progress executions
  • Alert destinations update immediately

What Can Be Edited

  • Detection name and description
  • Scheduled state
  • Severity
  • FSQL query
  • Trigger conditions (operator, threshold, eagerness)
  • Schedule (frequency, time, day)
  • Alert destinations

Best Practices

  1. Test Changes: Use "Run Now" to test before saving
  2. Pause First: Pause detection before major query changes
  3. Document Changes: Update description to reflect changes
  4. Monitor Results: Watch first few executions after changes
  5. Gradual Rollout: Test with lower severity first

Deleting Detections

Delete Detection

  1. Click overflow menu (three dots) in Actions column
  2. Select Delete
  3. Confirmation dialog displays detection name
  4. Click Confirm to delete

Confirmation Required: Prevents accidental deletion

What Happens:

  • Detection permanently deleted
  • Scheduled executions cancelled
  • Historical outcomes preserved (in database)
  • Alert destinations not affected
  • Cannot be undone

Before Deleting:

  • Verify detection is no longer needed
  • Document reason for deletion
  • Consider pausing instead if temporary

Monitoring Detection Health

Key Metrics

Execution Success Rate:

  • Percentage of successful executions (Matched + Not Matched)
  • Low rate indicates query or data source issues

Match Rate:

  • Percentage of executions that matched
  • High rate may indicate threshold too low
  • Low rate may indicate threshold too high or rare condition

Error Rate:

  • Percentage of executions that errored
  • Any errors require investigation

Alert Delivery:

  • Verify alerts reach destinations
  • Check destination platform for delivery confirmation

Health Indicators

Healthy Detection:

  • Executes on schedule
  • Outcome: Matched or Not Matched (not Error)
  • Alerts delivered successfully
  • Match rate appropriate for detection type

Unhealthy Detection:

  • Frequent errors
  • Timeouts
  • No executions (when Active)
  • Alert delivery failures

Monitoring Workflow

  1. Daily Review: Check for Error outcomes
  2. Weekly Review: Review match rates and adjust thresholds
  3. Monthly Review: Evaluate detection effectiveness
  4. Continuous: Monitor alert destination delivery

Troubleshooting

Detection Not Executing

Symptoms: No Runs or Last Run timestamp not updating

Checks:

  1. Verify Scheduled State is Active
  2. Check schedule configuration (time, frequency)
  3. Verify detection was saved
  4. Check system status for outages

Resolution:

  • Change to Active if Paused
  • Correct schedule if misconfigured
  • Use "Run Now" to test immediately

Detection Always Errors

Symptoms: Every execution results in Error outcome

Checks:

  1. Click detection name to view error details
  2. Test query in detection configuration
  3. Verify data sources are connected
  4. Check query syntax
  5. Review time window

Common Errors:

Query Syntax Error:

  • Error message indicates syntax issue
  • Test query and fix syntax
  • Save and re-run

Data Source Unavailable:

  • Error message indicates source not found
  • Verify data source name
  • Check data source connectivity
  • Update query if source renamed

Query Timeout:

  • Error message indicates timeout
  • Reduce time window
  • Add more specific filters
  • Specify data sources in FROM clause

Authentication Failure:

  • Error message indicates auth issue
  • Verify API credentials
  • Check data source permissions

Detection Never Matches

Symptoms: Always Not Matched, never Matched

Checks:

  1. Verify data exists in time window
  2. Test query manually
  3. Review filter conditions
  4. Check threshold configuration

Possible Causes:

No Data in Time Window:

  • Data sources may not have recent data
  • Adjust time window
  • Verify data ingestion

Filters Too Restrictive:

  • Query filters exclude all results
  • Broaden filters
  • Test with fewer conditions

Threshold Too High:

  • Matches found but below threshold
  • Review match counts in test results
  • Lower threshold

Wrong Event Type:

  • Query targets wrong event class
  • Verify event type in data sources
  • Adjust query to correct event class

Alerts Not Received

Symptoms: Detection matches but alerts not received

Checks:

  1. Verify destination is Active
  2. Verify destination is linked to detection
  3. Check destination configuration
  4. Test destination separately
  5. Review CloudWatch logs

Common Issues:

Destination Not Linked:

  • Edit detection
  • Go to Alert Configuration tab
  • Select destination
  • Save

Destination Inactive:

  • Go to Manage Destinations
  • Activate destination
  • Verify status

Invalid Credentials:

  • Check destination configuration
  • Verify API keys, tokens, passwords
  • Update secrets in Secrets Manager
  • Test destination

Network Issues:

  • Check destination platform status
  • Verify webhook URLs
  • Test connectivity

High False Positive Rate

Symptoms: Detection matches frequently with non-actionable alerts

Checks:

  1. Review recent matches
  2. Analyze common patterns
  3. Identify false positive causes

Solutions:

Adjust Threshold:

  • Increase threshold to reduce noise
  • Example: Change "greater than 0" to "greater than 5"

Add Filters:

  • Exclude known benign activity
  • Add user/host/IP exclusions
  • Narrow time windows

Refine Query:

  • Make conditions more specific
  • Add additional AND conditions
  • Use more precise field matching

Change Eagerness:

  • Switch to Exhaustive for accurate counts
  • Adjust threshold based on complete results

Best Practices

Detection Lifecycle

Development:

  1. Create detection in Paused state
  2. Test query thoroughly
  3. Run manually multiple times
  4. Verify alert destinations
  5. Activate when ready

Production:

  1. Monitor first few executions
  2. Adjust threshold based on results
  3. Document any tuning changes
  4. Review regularly for effectiveness

Maintenance:

  1. Review monthly for relevance
  2. Update queries as environment changes
  3. Adjust thresholds as needed
  4. Archive or delete obsolete detections

Naming Conventions

Detection Names:

  • Use clear, descriptive names
  • Include key indicators (e.g., "Failed SSH Logins >10")
  • Keep under 25 characters
  • Use consistent naming scheme

Destination Names:

  • Include team or purpose (e.g., "SOC Team Slack")
  • Include environment if applicable (e.g., "Prod PagerDuty")
  • Use descriptive names for easy identification

Documentation

Detection Descriptions:

  • Explain what detection monitors
  • Document threshold rationale
  • Note any exclusions or special cases
  • Include runbook links if available

Change Log:

  • Document significant changes
  • Note threshold adjustments
  • Record query modifications
  • Track effectiveness over time

Alert Management

Severity Assignment:

  • CRITICAL: Immediate threat, page on-call
  • HIGH: Significant concern, notify team
  • MEDIUM: Notable event, create ticket
  • LOW: Awareness, log only

Destination Routing:

  • Route by severity
  • Route by detection type
  • Route by team responsibility
  • Avoid alert fatigue

Response Procedures:

  • Document response steps
  • Include investigation guides
  • Link to runbooks
  • Define escalation paths

Performance Optimization

Query Performance

Slow Queries:

  • Specify data sources in FROM clause
  • Add more restrictive filters
  • Reduce time window
  • Limit field selection

Expensive Queries:

  • Use Eager mode for early termination
  • Increase threshold to stop sooner
  • Narrow time windows
  • Target specific sources

Detection Scheduling

Load Distribution:

  • Stagger detection schedules
  • Avoid running all detections at same time
  • Distribute across hour for hourly detections

Frequency Selection:

  • Match frequency to threat urgency
  • Don't over-schedule low-priority detections
  • Consider data ingestion lag

Next Steps