Get Help

Properties

A test definition names two things: the data to fetch, and what has to be true about it.

These properties apply wherever the definition lives — inline in the project file, in other YAML files, or in any provider that returns rows (Excel, a database query, …); sources can be combined. In YAML they are keys; in a tabular source they are the columns of one row. See Lists.

Tests:
- Name: Currency dimension is loaded
  Suite: Smoke tests
  Data source: FlightsSystem
  Query: SELECT * FROM Dimension.Currency
  Expectation: set is not empty

Properties

Property Also accepted as Required Default Meaning
Name Test name yes — Name of the test. Need not be unique
Suite Test suite no — Grouping level, part of the test’s full name
Order Test order no — Whole number, part of the full name. CAT does not order tests by it
Test case — no — Second grouping level, part of the full name
Description — no — Free text. Carried into results and into failure messages
Expectation — yes — What has to be true about the data. See Expectations
First data source Data source yes — Name of a data source definition
First query Query yes — The statement, in whatever language that data source’s provider understands
Second data source — for two-set expectations — Name of a data source definition
Second query — for two-set expectations — The statement for the second set
Timeout Timeout in seconds no 0 Seconds to allow the test. 0 means no limit. On expiry the test result is Error and CAT moves on — see Results
Maximum errors logged Maximum error count (sets match and contains only) no 1 How many offending rows the failure message shows, 0 to 50. See Failure message
Maximum sample column length — no 25 Characters kept per column in that sample, 10 to 32767. Longer values are cut
Log number of errors — no false When true, CAT keeps scanning after the sample is full so the failure message can report the total number of offending rows. Read by Set is empty
Output details — no false Experimental. When true, all offending rows — not just the sample — are written to the output as JSON. Supported for the set is empty expectation with a SQL Server output
Tags Test tags no — Comma- or semicolon-separated list. Tools filter which tests to run by tag
Metadata Test metadata no — Name of a query definition. Turns the definition into a template — see Templates

Property names are matched by synonym and are not case sensitive, so test_name and TestName address the same property. See Naming conventions.

%NAME% in any value is replaced with the value of the environment variable NAME. See Environment variables.

Expectation-specific properties

An expectation may read properties of its own. They are written on the test definition like any other property, and they are ignored by the expectations that do not know them.

Property Also accepted as Default Read by
Key — — Sets match and Contains — see Order and key
Sort data Sort my data false Sets match and Contains — see Order and key
Tolerance — 0 Sets match and Contains — see Tolerance
Tolerance mode — Absolute The same two. Accepts Absolute or Percent
Ignore case — true The same two. Text values compared without regard to case
Culture Locale en-US The same two. Culture used to read numbers and dates out of text values
Expected row count Expected rowcount — Set row count. When absent, that expectation reads the number from Second query instead

What each of them means, and which of the query properties each expectation needs, is documented on the expectation’s own page. See Expectations.

Full name

Suite, Order, Test case and Name together form the test’s full name: each part that has a value is wrapped in square brackets, and the parts are joined with dots. Parts left empty are skipped.

  • [Smoke tests].[Currency dimension is loaded]
  • [Smoke tests].[10].[Dimensions].[Currency dimension is loaded]

The full name is what tools display and what a name filter matches against — the match is a case-insensitive substring, so filtering on Smoke tests selects the whole suite.

CAT does not require full names to be unique and does not reject duplicates. Two tests that share one are both run and both reported, which makes the report ambiguous rather than wrong. Avoid square brackets inside the four parts for the same reason.

First and second sets

Every expectation needs one set of data: First data source names the data source, First query holds the statement CAT sends to it. Both properties drop the word First as a synonym, which reads better on a test that only ever has one set.

Expectations that compare two sets — sets match, contains — need Second data source and Second query as well. The two sets may come from the same data source or from two different ones, in different technologies.

An expectation that works on one set only does not ignore second-set properties, it refuses them: a Second data source on such a test is an error naming the expectation. Second query is refused the same way, with one exception — set rowcount reads the number of rows it expects from it.

First query and Second query hold statement text. They are not names of query definitions and are not resolved against them; the only property that refers to a query by name is Metadata — see Templates.

← Properties
Templates →