Lists — implicit and explicit

Every definition CAT loads comes from a list. Writing definitions in the project file is one way of declaring one.

CAT does not read definitions from the project file. It reads them from lists, and the project file is one of the things a list can point at. There are three kinds of list — lists of data sources, lists of queries and lists of tests — and each kind is resolved the same way: a provider is opened, rows are read, and each row becomes one definition.

A list is declared in one of two ways.

The implicit list #

Writing definitions directly under Data sources:, Queries: or Tests: in the project file declares a list. CAT builds it for you, with these settings:

Nothing downstream distinguishes that list from any other. The definitions under Tests: are read by the YAML provider out of the project file, exactly as definitions in a separate YAML file would be.

Data sources:
- Name: FlightsSystem
  Provider: SqlServer@2
  Connection string: data source=localhost;initial catalog=DWH;integrated security=true

Tests:
- Name: Departures are loaded
  Data source: FlightsSystem
  Query: SELECT * FROM FACT.DEPARTURES
  Expectation: set is not empty

Explicit lists #

Get list of data sources from:, Get list of queries from: and Get list of tests from: each take a YAML list. Every entry registers one external list.

Any further keys on the same entry are handed to the provider as its settings. Each provider page documents the settings it accepts and what it expects in Query.

One example per kind #

# data source definitions, read from a table
Get list of data sources from:
- Provider: SqlServer@2
  Connection string: data source=localhost;integrated security=true;initial catalog=Testing;trust_server_certificate=true;
  Query: SELECT * FROM [DataSource]

# query definitions, read from another YAML file
Get list of queries from:
- Provider: Yaml@1
  Connection string: SharedQueries.yaml
  Query: /Queries

# test definitions, read from a worksheet
Get list of tests from:
- Name: tests from business
  Provider: Excel@1
  Connection string: MyTests.xlsx
  Query: SELECT * FROM [Sheet1]

Rows are definitions #

A list returns rows. Each row is one definition, and the row’s columns are that definition’s properties — the same properties, under the same names, as the keys used when the definition is written in YAML. A test row carries Name, Expectation, Data source, Query and the rest; a data source row carries Name, Provider and Connection string.

Column names are matched by synonym and are not case sensitive, so each storage can name its columns in its own style. See Naming conventions.

The properties themselves are documented once, per kind of definition:

• Data source properties
• Query properties
• Test properties

How lists combine #

All three keys and the inline keys can be used in the same project file, in any combination. A kind of definition may be declared any number of times: several Get list of tests from entries, several providers, and a Tests: key alongside them.

Data sources:
- Name: DWH
  Provider: SqlServer@2
  Connection string: "%DWH_CONNECTION_STRING%"

Get list of tests from:
- Provider: SqlServer@1
  Connection string: "%DWH_CONNECTION_STRING%"
  Query: EXEC [Test].[GetSmokeTests]
- Provider: SqlServer@1
  Connection string: "%DWH_CONNECTION_STRING%"
  Query: EXEC [Test].[GetIntegrationTests]
- Provider: Excel@1
  Connection string: TestsFromBusiness.xlsx
  Query: SELECT * FROM [Tests]

The definitions from all lists of one kind form a single set. Lists are read in the order they appear under their key; the implicit list from the project file is read last.

A project must end up with at least one list of data sources and at least one list of tests, whether implicit or explicit. A project with no list of data sources is an error, and so is a project with no list of tests — see What a project must contain.

Behavior #

A list that returns no rows is not an error. CAT writes a warning to the log naming the list, and continues with the definitions from the other lists. A project whose test lists all return nothing runs no tests.

A list that cannot be read fails the run. An unreachable connection string, a query the provider rejects, an unknown provider name — each raises an error that names the failing list by its Name, position and provider, so the entry to fix can be identified.

Provider, Connection string and Query are validated before anything is opened. A missing one is reported as an error naming the key and the position of the entry within it, for example Get List of Tests from # 2.

Only Yaml@1 lists can be written back to. Tools that edit definitions — CAT Studio, for one — can save changes into a YAML-backed list, including the implicit list in the project file. Definitions from a list served by any other provider are read-only to CAT; they are maintained where they are stored.