Get Help

Run your tests

You have a project file with a data source and a test. Run it — all of it, or only the suite, the tags or the names you care about right now — with the tool you have at hand.

Every CAT tool runs the same project file. CAT Studio while you write tests, the PowerShell module or CAT CLI when you script or schedule them, the Python module when they are part of a notebook or a Python pipeline — the project file does not care.

Run everything

Open the project (Project page → Open existing project, or a recent one), go to Tests and press F9 — Run all tests. Results fill the grid as tests finish; click a test to read its message.

cd D:\AutomatedTests\Flights
catcli run

With no path, catcli run takes the one .cat.yaml file in the current directory. Otherwise name it: catcli run --project D:\AutomatedTests\Flights\Flights.cat.yaml (a directory works too, when it holds exactly one project file). See run.

cd D:\AutomatedTests\Flights
Invoke-CatProject

With no path, Invoke-CatProject takes the one .cat.yaml file in the current location. Otherwise name it: Invoke-CatProject -Path D:\AutomatedTests\Flights\Flights.cat.yaml (a directory works too, when it holds exactly one project file). See Invoke-CatProject.

import justcatit as cat
summary = cat.invoke_project("D:/AutomatedTests/Flights/Flights.cat.yaml")

The path is required — a file, or a directory with exactly one project file. See invoke_project.

Read the result

A command-line run prints one line per test as it finishes and a summary at the end:

• 00:00:00.0755756 - [Smoke tests].[Airline dimension matches the source] finished with result Passed
• 00:00:00.1389169 - [Data quality].[No booking without a passenger] finished with result Failed
• 00:00:06.2492205 - [Smoke tests].[Flights table is loaded] finished with result Error

😺 passed: 1     😿 failed: 1     🙀 error: 1     🐱 inconclusive: 0

Three things to read:

  • A result per test — Passed, Failed, Error (or Inconclusive); see Results.
  • The summary — how many of each, how long it took.
  • The log, when you turn it on. Anything can go wrong — a typo in the project file or in a query, a data source that does not exist, a missing permission — and the log is what says which. CAT CLI is quiet by default (-l Information to see it, -l Error for the failed tests’ details only); the PowerShell module logs at Information by default (-LoggingLevel None to silence it).

The details of a failed test — the message with the sample of offending rows, the queries — are in CAT Studio’s test detail, in the PowerShell module’s Get-CatTestResult, in the Python summary’s Results, and in every output you ask for — an Excel file, JSON, a database table, a report for your pipeline.

Run only a part of the project

Test bases grow; sooner or later running everything on every occasion stops making sense. Two mechanisms narrow a run down, and they combine — a test runs only when it satisfies every condition you give.

By name

A filter runs the tests whose full name — [Suite].[Order].[Test case].[Name] — contains the text. Substring, not case-sensitive, no wildcards. Because the full name is built from the categorization, a filter is also how you run one suite or one test case.

Filter the grid — the column headers filter and search — tick the tests you want, and press F8, Run selected tests.

catcli run --filter "Airline"
catcli run --filter "[Smoke tests]"
catcli run --filter "[Smoke tests].[Dimensions]"

Invoke-CatProject -Filter "Airline"
Invoke-CatProject -Filter "[Smoke tests]"
Invoke-CatProject -Filter "[Smoke tests].[Dimensions]"

cat.invoke_project(path, filter="Airline")
cat.invoke_project(path, filter="[Smoke tests]")
cat.invoke_project(path, filter="[Smoke tests].[Dimensions]")

By tag

Names were chosen for other reasons; tags are the deliberate way of marking a group. Add Tags to a test — one, or several separated by a comma or a semicolon — then include or exclude by tag. Matching is not case-sensitive; include runs tests that carry at least one of the listed tags, exclude skips tests that carry any of them.

Tests:
- Name: Full reconciliation of the fact table
  Tags: departures, slow
  # …

Excluding is the more useful direction: tag the tests that should never run unattended — long ones, tests against production, tests someone has to look at — ManualOnly, and exclude that tag wherever the project is scheduled. The tests stay in the project and stay maintained; the pipeline stops tripping over them.

The run button’s dropdown on the Tests page has the include and the exclude tag lists; they combine with the ticked tests.

catcli run --includeTags "departures, passengers"
catcli run --excludeTags ManualOnly

Invoke-CatProject -IncludeTag departures, passengers
Invoke-CatProject -ExcludeTag ManualOnly

cat.invoke_project(path, include_tags="departures, passengers")
cat.invoke_project(path, exclude_tags="ManualOnly")
← Work with secrets
Run tests in parallel →