Auto-Investigating Detections from Databricks

Launch Query Worker investigations on the detections you choose, and read the reports in Unity Catalog

📘

TL;DR

run_query_worker selects detections with a SQL predicate and starts a Query Worker investigation on each. fetch_worker_report collects the finished reports. Both are job parameters — change which detections qualify, and what the Worker is asked, without opening a notebook.

What a Query Worker Does

A federated search answers a question you already know how to ask. A Query Worker is given a detection and works out the questions for itself: which entities are worth pivoting on, which platforms can answer for them, what each result means for the next search. It runs for minutes rather than seconds, and returns a written investigation rather than rows.

That makes it the right tool for the detections you would otherwise hand to an analyst — and the wrong one for bulk enrichment, where federated search is faster and cheaper.

Two jobs cover the lifecycle: one launches investigations, the other collects the reports.

How It Works

run_query_worker
  └─ reads detections_table with your filter
     └─ starts a Query Worker per detection
        └─ records the investigation id, returns

fetch_worker_report
  └─ finds investigations with no report yet
     └─ collects the finished ones, merges them in

Separate on purpose. A Worker investigation takes minutes; the launching job does not wait.

Both write to worker_investigations, so a row appears when an investigation starts and gains its report later.

Which Detections Qualify

detection_filter is a SQL predicate against your detections table. The default is deliberately narrow:

severity = 'Critical' AND status NOT IN ('Resolved', 'Suppressed')

Every matching detection spends an agentic investigation against your Query tenant. Start narrow, widen after a week of real volume.

🚧

The filter is SQL run by the job's principal

It is checked for shapes that would turn a predicate into a second statement, but granting someone the ability to set it grants them that principal's reach.

What the Worker Is Asked

worker_question is a template over the detection's own columns. Its default is the full brief, so you can read and edit it as a parameter.

Two behaviors make a long template safe on a sparse table:

  • A line whose substitutions all come out empty is dropped — a detection with no host contributes no Host: line.
  • A column your table lacks is ignored rather than failing the run.

Listing every field worth sending therefore costs nothing on detections that carry only some.

Plan Without Launching

databricks bundle run run_query_worker --profile <profile> --params max_investigations=0

Shows which detections qualify without spending anything. Use it whenever you change the filter.

Collect the Reports

databricks bundle run fetch_worker_report --profile <profile>

This is the only scheduled job in the bundle, and it ships paused.

Match the interval to how long investigations take. Workers run for several minutes. Hourly is usually good.

Read a Full Report

The dashboard shows what an investigation concluded, not the report. AI/BI dashboards cannot display documents; a table cell truncates markdown to a few characters.

Open the view_investigation_report notebook, paste an Investigation id from the dashboard into its widget, and run. It renders the markdown as a document.

🚧

Use serverless compute, not a SQL warehouse

The compute selector offers both. A warehouse refuses the cell with Unsupported cell during execution. SQL warehouses only support executing SQL cells — a message about the cell, not the compute.

The id is copied across by hand: a dashboard link cannot tell a notebook which row was clicked.

What the Analyst Sees

The dashboard's Worker Investigations tab carries one row per investigation — the detection it came from, its state, when it launched, what it concluded — with counters for running, finished and awaiting a report.


Did this page help you?