Get Help

CAT in Databricks notebooks

Can CAT run inside a Databricks notebook? Yes — with the Python module, a .NET runtime on the cluster, and a few limits worth knowing. Usually it is better run from outside.

Testing Databricks data — from where?

CAT tests data in Databricks Delta tables through the Odbc@1 provider and the Simba driver; it can even generate metadata-driven tests from Unity Catalog. The setup is in Test Databricks.

The reason is what CAT is for: checking that the data is correct end to end, which means cross-system comparisons — sources against the lakehouse, the lakehouse against Power BI, incremental loads against their sources. Tests of that kind (data readiness in the sources, incremental-load correctness, every Power BI test, every cross-system check) belong to a place that can see all the systems; a notebook inside one of them is a witness with a limited view. Run CAT from somewhere else. Really.

But I want to run tests from a notebook

It is possible. The notebook needs two things: the justcatit package and a .NET runtime on the cluster nodes.

Library. In the cluster’s Libraries settings, install the PyPI package justcatit (https://pypi.org/project/justcatit/).

.NET runtime. Create InitCluster.sh in your workspace or repository (or add the lines to an existing cluster init script):

#!/bin/bash
echo 'Installation of .NET 8 runtime start.'
sudo apt-get update && \
  sudo apt-get install -y dotnet-runtime-8.0
echo 'Installation of .NET 8 runtime end.'

Then edit the cluster, open Advanced options and add the init script (type Workspace).

Use CAT. Put a .cat.yaml project file in your workspace and run it from the notebook:

from justcatit import cat
summary = cat.invoke_project("/Workspace/Users/you/tests/Dwh.cat.yaml")
print(summary.GetShortSummary())

What is different on a cluster

The cluster nodes run Linux, and the Python module is built and tested on Windows — see Platforms. In short: the providers built on Windows components (Dax@*, PowerBI@*, CsvOleDB@1, ExcelOleDB@1) are unavailable; the DuckDB-based Csv@2 / Excel@2 providers lack their Linux native library in the package; the xlsx output fails after the tests ran — use json, yaml, junit or a database output, or skip_outputs=True; log files land in ./CAT/Logs/ under the working directory. A cluster is also an automated environment: the Team plan, which is interactive-only, may be refused there — use the Enterprise plan with a license key set through cat.set_instance(...). The failed-test signal is the summary, as everywhere: summary.FailedCount.

← Test Databricks
Test Microsoft Fabric →