Query for Databricks
Investigate your detections against every connected source, from inside your lakehouse
TL;DRA Databricks Asset Bundle you deploy into your own workspace. It runs federated searches and Query Worker investigations against the Query Security Data Mesh, and writes the results to Unity Catalog tables that your dashboards, notebooks and Genie spaces can already read. Nothing of Query's runs in your workspace, and none of your lakehouse data leaves it.
The Problem It Solves
Plenty of security teams now keep their detections in Databricks — as a SIEM replacement, as the tier their SIEM is too expensive to hold, or simply because that is where the data engineering already lives. A detection fires, and the row is right there in a Delta table.
What is not there is the evidence that settles it. Whether the account signed in from somewhere new lives in the identity provider. Whether the hash is known bad lives in a threat intelligence feed. Whether the host talked to that address lives in the firewall logs. Whether the user was phished lives in the mail gateway.
Getting that evidence into the lakehouse first is a pipeline project per source — and the sources worth asking are usually the ones nobody has budget to ingest. So the analyst opens another console, and another, and the detection sits in the queue while they do.
What This Changes
This integration lets your Databricks workspace ask those questions directly, at the moment it needs the answer, without ingesting anything.
A notebook can run one federated search across every connected platform and get back OCSF-normalized rows. A job can take the detections that matter and launch a Query Worker on each, so the investigation is already written when someone opens the queue. Either way the results land in Unity Catalog, where whatever you already built — a dashboard, a scheduled query, a Genie space — can read them like any other table.
The data you reach stays where it lives. Query does not retain or duplicate it, and the rows that land in your lakehouse are the answers you asked for, not a copy of someone else's platform.
When to Use It
Reach for it if:
- Your investigation work already happens in Databricks. Notebooks, the SQL editor, a dashboard someone keeps open — you want to ask other platforms a question without leaving any of it, and without the answer arriving somewhere else.
- Your detections, alerts or findings live in Delta tables, and you want them enriched or investigated without moving them.
- You want to reach platforms you have not ingested, and do not want to build a pipeline per source to do it.
- The answers need to land where your tooling already looks — a dashboard, a scheduled query, a Genie space, a join against a table you already maintain.
- You want investigations running on a schedule or a trigger, in the same place as the rest of your workflows.
The first of those matters more than it sounds. Context-switching is most of what makes an investigation slow, and the console someone has open is usually the one the investigation happened in. If that console is Databricks, this puts the whole mesh behind it.
What Gets Deployed
One bundle, into a workspace you choose: Lakeflow jobs, a library the notebooks import, an AI/BI dashboard, and a Genie space.
| Job | Does | Writes |
|---|---|---|
setup | Creates result tables, checks your detections table | — |
run_fsql_query | Runs one FSQL statement across the mesh | query_mesh_events, fsql_run_audit |
run_query_worker | Launches Query Worker investigations | worker_investigations |
fetch_worker_report | Collects finished reports | worker_investigations |
refresh_connectors | Snapshots what is connected | query_mesh_connectors |
ask_mcp_agent | Asks the mesh a plain-English question | mcp_agent_answers |
load_sample_detections | Synthetic detections, evaluation only | a table you name |
Each job is standalone — schedule, trigger or run any of them on its own. Nothing is scheduled at install, and the one job that carries a schedule ships paused, so the integration costs nothing until you turn something on.
Two Shapes of Answer
The mesh answers in two ways, and choosing between them is the main decision per task.
| FSQL | Query Worker | |
|---|---|---|
| Answers in | seconds | minutes |
| Returns | OCSF-normalized rows | a markdown investigation report |
| Use for | fetching context for a detection | investigating a selected one |
A federated search is a question you already know how to phrase — has this hash been seen anywhere, did this account sign in from a new country. A Query Worker is given a detection and decides for itself what to ask, chaining searches across sources and writing up what it found.
Because a Worker runs for minutes, launching and collecting are separate jobs. run_query_worker records each investigation id and returns immediately; fetch_worker_report picks the report up once it is ready.
Where Results Appear
Everything lands in Unity Catalog, so there is no new console to learn.
The bundle ships an AI/BI dashboard — connected platforms, mesh results, and worker investigations — and a Genie space that answers questions over the same tables in plain English. Anything else you have already built can read the tables directly.
Scope and Access
Read-only against the mesh: only non-mutating FSQL statements are accepted. Your detections table is read and never written.
The jobs call the Query APIs outbound using a token you store in a Databricks secret scope. Nothing of Query's is installed into your workspace, and no inbound access to it is required. See Security & Privacy and Product Architecture.
Updated 31 minutes ago