Databricks@1

Databricks@1 reads data from Databricks. Nothing to install.

Connects to #

Databricks on Azure, AWS and Google Cloud: SQL warehouses (classic and serverless — the right choice for tests) and all-purpose clusters.

Copy the two values a connection needs from the workspace:

Example #

• Properties
• YAML

Data sources:
- Name: MyDatabricksData
  Provider: Databricks@1
  Technology: Databricks
  Connection string: >
    adbc.spark.host=%DATABRICKS_HOST%;
    adbc.spark.path=%DATABRICKS_HTTP_PATH%;
    adbc.spark.auth_type=token;
    adbc.spark.token=%DATABRICKS_TOKEN%;

Connection string #

The keys are the driver’s — Key=Value pairs separated by semicolons, handed to the Apache Arrow ADBC driver’s connection object exactly as written. The keys you will use most:

A service principal authenticates with grant_type=client_credentials and its client id and secret instead of a token:

• Properties
• YAML

Data sources:
- Name: MyDatabricksData
  Provider: Databricks@1
  Technology: Databricks
  Connection string: >
    adbc.spark.host=%DATABRICKS_HOST%;
    adbc.spark.path=%DATABRICKS_HTTP_PATH%;
    adbc.spark.auth_type=oauth;
    adbc.databricks.oauth.grant_type=client_credentials;
    adbc.databricks.oauth.client_id=%DATABRICKS_CLIENT_ID%;
    adbc.databricks.oauth.client_secret=%DATABRICKS_CLIENT_SECRET%;

Any other parameter the driver documents passes through unchanged — for example adbc.databricks.cloudfetch.enabled=false, or a server-side SET through adbc.databricks.ssp_<name>=…. CAT reads no key from the string, adds none and validates nothing; a missing or misspelt key fails with the driver’s own error message, at the data source.

The full parameter reference is in the driver’s own documentation: the inherited Spark keys in the Spark driver readme, and the Databricks-specific keys in the readme on the NuGet package page.

A token, a secret, or the whole string, can come from an environment variable — %NAME% is replaced with the value of NAME; see How every provider is used.

Settings #

None beyond Connection string.

Query #

One SQL statement in Databricks SQL that returns a result set. Use three-part names (catalog.schema.table), or set adbc.connection.catalog and adbc.connection.db_schema in the connection string to default unqualified names.

Prerequisites #

None. The driver ships with CAT — Windows, Linux and macOS alike, nothing to install.

A warehouse that is stopped starts when CAT connects, and the first test on it waits rather than fails; a cluster must already be running.

Types #

Serving test and data source definitions #

Databricks@1 can also serve lists of definitions: any statement that returns the columns of a test, data source or query definition works. Ready-made table scripts are in Store definitions in a database.

Related #

• Technologies — what CAT can test, with the provider behind each.
• Test Databricks — a full walk-through, with the guided form in CAT Studio.
• Odbc@1 — through the ODBC driver instead, when your organisation standardises on it.