Query Databricks Bundle Setup
Deploy the bundle, point it at your detections, and give your team access
TL;DRStore your Query API token in a Databricks secret scope, point
detections_tableat a table or view of your own, thendatabricks bundle deployand runsetuponce. Everything else is optional.
What You Are Installing
A Databricks Asset Bundle, delivered as a release archive. Unpack it and everything is there: the jobs, the library they import, a dashboard and a Genie space, deployed as one unit with databricks bundle deploy.
tar xzf query-security-mesh-<version>.tar.gz
cd query-security-mesh-<version>
Run the commands on this page from that directory. The bundle names no workspace, catalog or schema of its own — you supply those — so the same archive installs into dev and production without editing anything.
Setup is three decisions and two commands. Where do your detections live, where should results be written, and where does the API token come from. Everything after that is optional.
Prerequisites
- A Query API token. Create one in the Query console under Settings → Organization → your team → Integrations.
- A Databricks workspace, Premium or above, Unity Catalog enabled.
- A SQL warehouse — the dashboard and Genie space read through it.
- The Databricks CLI, v0.230 or newer.
- Your detections, in a table or view meeting the contract below.
The bundle names no workspace, so it installs wherever your profile points.
databricks auth login --host https://<your-workspace> --profile <profile>
1. Point It at Your Detections
Read-only — the integration never writes your detections. A view is the usual way.
| Columns | |
|---|---|
Required (setup fails without these) | id, name, severity, status |
| Recommended (each absent narrows what an investigation is told) | description, product_name, event_time, actor_user_name, device_hostname, src_ip |
| Optional, used when present | mitre_tactic, mitre_technique_uid, mitre_technique, severity_id |
status of anything other than Resolved or Suppressed counts as open.
CREATE VIEW my_catalog.my_schema.detections AS
SELECT alert_id AS id,
alert_name AS name,
priority AS severity,
state AS status,
* EXCEPT (alert_id, alert_name, priority, state)
FROM my_catalog.my_schema.alerts;
2. Store the API Token
Scope and key names are fixed — the notebooks look for exactly these.
databricks secrets create-scope query-security-mesh --profile <profile>
databricks secrets put-secret query-security-mesh query-api-token --profile <profile>
Never make the token a job parameterAs a secret it stays out of run history and off screens. As a parameter it appears in both.
3. Deploy and Run Setup
databricks bundle deploy --profile <profile> \
--var catalog=my_catalog \
--var schema=my_schema \
--var detections_table=my_catalog.my_schema.detections
databricks bundle run setup --profile <profile>
catalog and schema are where the integration writes its own tables, defaulting to workspace.security.
Read what setup prints: detections_columns_ok false means a required column is missing, and missing_recommended lists what an investigation will not be told. Re-running is safe, and it adds any column a later version introduces — so run it after every upgrade.
4. Deploy the Dashboard
./dashboard/deploy.sh <profile> # first time; prints a dashboard id
./dashboard/deploy.sh <profile> <dashboard-id> # thereafter
Deploying runs every dataset query against the warehouse first and refuses to publish if one fails.
The dashboard is published without embedded credentials deliberately: each viewer's queries run as that viewer. So opening it is not enough on its own — see below.
5. Give People Access
Whoever deploys owns every object and needs no grants. The second person needs several, and each failure names the wrong thing.
Grant to an account-level groupA workspace-local group —
adminsnotably — is not a Unity Catalog principal. Granting to it fails withCould not find principal with name admins.
GRANT USE CATALOG ON CATALOG my_catalog TO `security-readers`;
GRANT USE SCHEMA ON SCHEMA my_catalog.my_schema TO `security-readers`;
GRANT SELECT ON SCHEMA my_catalog.my_schema TO `security-readers`;
Then, per task:
| To | Also needs |
|---|---|
| Read the dashboard | CAN READ on it; CAN USE on the warehouse |
| Read a full report | notebook access; serverless compute |
| Ask the mesh | USE_CONNECTION, EXECUTE on the MCP service, and their own sign-in |
| Ask Genie | CAN RUN on the Genie space |
| See or run jobs | CAN VIEW / CAN MANAGE RUN |
A deployed bundle shares nothing, so jobs are invisible to everyone but the deployer until shared. Unity Catalog grants and object permissions are independent: someone can hold SELECT on every table and still find Jobs & Pipelines empty.
Troubleshooting
| Symptom | Cause |
|---|---|
setup reports detections_columns_ok false | A required column is missing from detections_table |
| Jobs & Pipelines empty for a colleague | The bundle shares nothing; share each job |
Requires permission USE_SCHEMA on SCHEMA … | The data grants above are missing. The error names the schema, not the dashboard |
Unsupported cell during execution | The report notebook is on a SQL warehouse. Use serverless |
invalid_grant: Unknown or invalid refresh token | MCP authorization expired. Sign in again |
Still stuck: contact [email protected] or use in-product Chat.
Upgrading
Unpack the new release beside the old one and deploy from it. VERSION in each directory says which build it is.
tar xzf query-security-mesh-<version>.tar.gz
cd query-security-mesh-<version>
databricks bundle deploy --profile <profile> --auto-approve \
--var catalog=my_catalog --var schema=my_schema \
--var detections_table=my_catalog.my_schema.detections
databricks bundle run setup --profile <profile>
./dashboard/deploy.sh <profile> <dashboard-id>
Pass the same variables you first deployed with. A bundle deploys what the command says, not what the last one said, so an omitted --var silently reverts to its default.
--auto-approve matters: a release that removes a resource asks for confirmation, and CI has nobody to answer. Run setup after every upgrade so it reconciles any new column onto tables you already have.
Your data is unaffected — the result tables, the secret scope and the MCP connection live in your workspace, not in the unpacked directory.
Updated about 2 hours ago