Get Help

CAT in Databricks notebooks

Can CAT run inside a Databricks notebook? Yes — with the Python module, a .NET runtime on the cluster, and a few limits worth knowing. Usually it is better run from outside.

Testing Databricks data — from where?

CAT tests data in Databricks Delta tables through the Odbc@1 provider and the Simba driver; it can even generate metadata-driven tests from Unity Catalog. The setup is in Test Databricks.

The reason is what CAT is for: checking that the data is correct end to end, which means cross-system comparisons — sources against the lakehouse, the lakehouse against Power BI, incremental loads against their sources. Tests of that kind (data readiness in the sources, incremental-load correctness, every Power BI test, every cross-system check) belong to a place that can see all the systems; a notebook inside one of them is a witness with a limited view. Run CAT from somewhere else. Really.

But I want to run tests from a notebook

It is possible. The notebook needs two things: the justcatit package and a .NET runtime on the cluster nodes.

Library. In the cluster’s Libraries settings, install the PyPI package justcatit (https://pypi.org/project/justcatit/).

.NET runtime. Create InitCluster.sh in your workspace or repository (or add the lines to an existing cluster init script). It installs the .NET 10 runtime with Microsoft’s install script, which works on every Ubuntu release Databricks runs — the apt packages do not (Microsoft’s package feed has no .NET 10 for Ubuntu, and Ubuntu’s own feed has it only from 24.04):

#!/bin/bash
echo 'Installation of .NET 10 runtime start.'
curl -sSL https://dot.net/v1/dotnet-install.sh -o /tmp/dotnet-install.sh
sudo bash /tmp/dotnet-install.sh --runtime dotnet --channel 10.0 --install-dir /usr/share/dotnet
sudo ln -sf /usr/share/dotnet/dotnet /usr/bin/dotnet
echo 'Installation of .NET 10 runtime end.'

Then edit the cluster, open Advanced options and add the init script (type Workspace).

Use CAT. Put a .cat.yaml project file in your workspace and run it from the notebook:

from justcatit import cat
summary = cat.invoke_project("/Workspace/Users/you/tests/Dwh.cat.yaml")
print(summary.GetShortSummary())

What is different on a cluster

The cluster nodes run Linux, and the Python module is built and tested on Windows — see Platforms. In short: the providers built on Windows components (Dax@*, PowerBI@*, CsvOleDB@1, ExcelOleDB@1) are unavailable; the DuckDB-based Csv@2 / Excel@2 providers lack their Linux native library in the package; the xlsx output fails after the tests ran — use json, yaml, junit or a database output, or skip_outputs=True; log files land in ./CAT/Logs/ under the working directory. A cluster is also an automated environment: the Team plan, which is interactive-only, may be refused there — use the Enterprise plan with a license key set through cat.set_instance(...). The failed-test signal is the summary, as everywhere: summary.FailedCount.

← Test Databricks
Test Microsoft Fabric →