CAT in Databricks notebooks

Can CAT run inside a Databricks notebook? Yes — with the Python module (it carries its own .NET runtime, so the cluster needs nothing installed first) and a few limits worth knowing. Usually it is better run from outside.

Testing Databricks data — from where? #

CAT tests data in Databricks Delta tables through the Databricks@1 provider; it can even generate metadata-driven tests from Unity Catalog. The setup is in Test Databricks.

The reason is what CAT is for: checking that the data is correct end to end, which means cross-system comparisons — sources against the lakehouse, the lakehouse against Power BI, incremental loads against their sources. Tests of that kind (data readiness in the sources, incremental-load correctness, every Power BI test, every cross-system check) belong to a place that can see all the systems; a notebook inside one of them is a witness with a limited view. Run CAT from somewhere else. Really.

But I want to run tests from a notebook #

It is possible. The notebook needs one thing: the justcatit package — the wheel carries its own .NET runtime, so there is nothing else to set up on the cluster nodes.

Library. In the cluster’s Libraries settings, install the PyPI package justcatit (https://pypi.org/project/justcatit/). That is the whole setup.

Use CAT. Put a .cat.yaml project file in your workspace and run it from the notebook:

from justcatit import cat
summary = cat.invoke_project("/Workspace/Users/you/tests/Dwh.cat.yaml")
print(summary.GetShortSummary())

What is different on a cluster #

The cluster nodes run Linux, and the Linux x64 wheel of the Python module is the one we test on Linux — see Platforms. In short: the providers built on Windows components (Dax@*, PowerBI@*, CsvOleDB@1, ExcelOleDB@1) are unavailable; log files land in ~/.local/state/CAT/Logs/ — the state folder of the account the cluster runs CAT under. A cluster is also an automated environment: the Team plan, which is interactive-only, may be refused there — use the Enterprise plan with a license key set through cat.set_instance(...). The failed-test signal is the summary, as everywhere: summary.FailedCount.

Related #

• Python module — Introduction · invoke_project.
• Test Databricks — connection string, first test.
• CAT and Microsoft Fabric.