Query Databricks Bundle Setup

Deploy the bundle, point it at your detections, and give your team access

📘

TL;DR

Store your Query API token in a Databricks secret scope, point detections_table at a table or view of your own, then databricks bundle deploy and run setup once. Everything else is optional.

What You Are Installing

A Databricks Asset Bundle, delivered as a release archive. Unpack it and everything is there: the jobs, the library they import, a dashboard and a Genie space, deployed as one unit with databricks bundle deploy.

tar xzf query-security-mesh-<version>.tar.gz
cd query-security-mesh-<version>

Run the commands on this page from that directory. The bundle names no workspace, catalog or schema of its own — you supply those — so the same archive installs into dev and production without editing anything.

Setup is three decisions and two commands. Where do your detections live, where should results be written, and where does the API token come from. Everything after that is optional.

Prerequisites

  • A Query API token. Create one in the Query console under Settings → Organization → your team → Integrations.
  • A Databricks workspace, Premium or above, Unity Catalog enabled.
  • A SQL warehouse — the dashboard and Genie space read through it.
  • The Databricks CLI, v0.230 or newer.
  • Your detections, in a table or view meeting the contract below.

The bundle names no workspace, so it installs wherever your profile points.

databricks auth login --host https://<your-workspace> --profile <profile>

1. Point It at Your Detections

Read-only — the integration never writes your detections. A view is the usual way.

Columns
Required (setup fails without these)id, name, severity, status
Recommended (each absent narrows what an investigation is told)description, product_name, event_time, actor_user_name, device_hostname, src_ip
Optional, used when presentmitre_tactic, mitre_technique_uid, mitre_technique, severity_id

status of anything other than Resolved or Suppressed counts as open.

CREATE VIEW my_catalog.my_schema.detections AS
SELECT alert_id   AS id,
       alert_name AS name,
       priority   AS severity,
       state      AS status,
       *          EXCEPT (alert_id, alert_name, priority, state)
FROM my_catalog.my_schema.alerts;

2. Store the API Token

Scope and key names are fixed — the notebooks look for exactly these.

databricks secrets create-scope query-security-mesh --profile <profile>
databricks secrets put-secret query-security-mesh query-api-token --profile <profile>
❗

Never make the token a job parameter

As a secret it stays out of run history and off screens. As a parameter it appears in both.

3. Deploy and Run Setup

databricks bundle deploy --profile <profile> \
  --var catalog=my_catalog \
  --var schema=my_schema \
  --var detections_table=my_catalog.my_schema.detections

databricks bundle run setup --profile <profile>

catalog and schema are where the integration writes its own tables, defaulting to workspace.security.

Read what setup prints: detections_columns_ok false means a required column is missing, and missing_recommended lists what an investigation will not be told. Re-running is safe, and it adds any column a later version introduces — so run it after every upgrade.

4. Deploy the Dashboard

./dashboard/deploy.sh <profile>                 # first time; prints a dashboard id
./dashboard/deploy.sh <profile> <dashboard-id>  # thereafter

Deploying runs every dataset query against the warehouse first and refuses to publish if one fails.

The dashboard is published without embedded credentials deliberately: each viewer's queries run as that viewer. So opening it is not enough on its own — see below.

5. Give People Access

Whoever deploys owns every object and needs no grants. The second person needs several, and each failure names the wrong thing.

🚧

Grant to an account-level group

A workspace-local group — admins notably — is not a Unity Catalog principal. Granting to it fails with Could not find principal with name admins.

GRANT USE CATALOG ON CATALOG my_catalog TO `security-readers`;
GRANT USE SCHEMA  ON SCHEMA  my_catalog.my_schema TO `security-readers`;
GRANT SELECT      ON SCHEMA  my_catalog.my_schema TO `security-readers`;

Then, per task:

ToAlso needs
Read the dashboardCAN READ on it; CAN USE on the warehouse
Read a full reportnotebook access; serverless compute
Ask the meshUSE_CONNECTION, EXECUTE on the MCP service, and their own sign-in
Ask GenieCAN RUN on the Genie space
See or run jobsCAN VIEW / CAN MANAGE RUN

A deployed bundle shares nothing, so jobs are invisible to everyone but the deployer until shared. Unity Catalog grants and object permissions are independent: someone can hold SELECT on every table and still find Jobs & Pipelines empty.

Troubleshooting

SymptomCause
setup reports detections_columns_ok falseA required column is missing from detections_table
Jobs & Pipelines empty for a colleagueThe bundle shares nothing; share each job
Requires permission USE_SCHEMA on SCHEMA …The data grants above are missing. The error names the schema, not the dashboard
Unsupported cell during executionThe report notebook is on a SQL warehouse. Use serverless
invalid_grant: Unknown or invalid refresh tokenMCP authorization expired. Sign in again

Still stuck: contact [email protected] or use in-product Chat.

Upgrading

Unpack the new release beside the old one and deploy from it. VERSION in each directory says which build it is.

tar xzf query-security-mesh-<version>.tar.gz
cd query-security-mesh-<version>
databricks bundle deploy --profile <profile> --auto-approve \
  --var catalog=my_catalog --var schema=my_schema \
  --var detections_table=my_catalog.my_schema.detections
databricks bundle run setup --profile <profile>
./dashboard/deploy.sh <profile> <dashboard-id>

Pass the same variables you first deployed with. A bundle deploys what the command says, not what the last one said, so an omitted --var silently reverts to its default.

--auto-approve matters: a release that removes a resource asks for confirmation, and CI has nobody to answer. Run setup after every upgrade so it reconciles any new column onto tables you already have.

Your data is unaffected — the result tables, the secret scope and the MCP connection live in your workspace, not in the unpacked directory.


Did this page help you?