Databricks

Run SQL on a warehouse, start and follow jobs, manage clusters and pipelines, browse Unity Catalog, read the workspace and DBFS, and query model serving endpoints. Works with Databricks on AWS, Azure and GCP.

Connecting

Every credential needs the Workspace host, for example dbc-a1b2c3d4-e5f6.cloud.databricks.com or adb-1234567890123456.7.azuredatabricks.net. If you paste a full URL, Flero keeps only the host.

Personal access token (simplest)

1. In Databricks, open Settings → Developer → Access tokens and generate a token.
2. In Flero, choose the token auth type and paste the dapi… token.

A service principal's token works the same way, and keeps working when the person who set it up leaves.

OAuth (user-to-machine)

1. In the Databricks account console, register a custom OAuth app with the Redirect URL from Flero's credential dialog.
2. In Flero, choose OAuth 2.0, enter the workspace host and the app's client ID (and secret, for a confidential app).
3. Click Connect with Databricks.

Flero requests the all-apis and offline_access scopes by default.

What the workflow can do is whatever the token's or user's Databricks permissions allow.

Operations

Warehouse, job, cluster, pipeline, catalog and serving endpoint fields are dropdowns filled from your workspace.

Running SQL

Execute SQL statement waits up to wait_timeout (default 30 seconds, maximum 50). If the statement finishes in time, the result comes back inline. If not, you get a statement_id to poll with Get statement.

Whenever a result is present, Flero adds rows: one object per row, keyed by column name, next to Databricks' own result.data_array. Read them with {{ $node["Daily Orders"].output.data.rows }}.

Pass values with :name markers in the SQL and the parameters list, rather than building the SQL string from expressions:

SELECT region, count(*) AS orders
FROM main.sales.orders
WHERE order_date = :day
GROUP BY region

Job output

Get run output needs a task run ID. If you pass a job run ID whose run has exactly one task, Flero looks up the task run for you. For a multi-task job, read tasks[].run_id from Get run and pass that.

Triggers

The Databricks Trigger polls List jobs or List runs. Use List runs to react to finished runs, for example filtered by job with completed_only on. Other operations are not offered as triggers.

Limits

• Cluster operations need a workspace with classic compute. A serverless-only workspace refuses Create cluster with "does not have any associated worker environments".
• Databricks omits empty lists from responses ({} rather than {"jobs": []}). Check for a missing field, not just an empty one.
• Databricks' own rate limits apply.

Common errors

Related

• Connector event triggers
• Schedule trigger

Found something out of date? This page lives in the Flero docs content set.