Get Help

Overview

Every output format on one table — what it is for, when it is written, where it runs — and the sample run the format pages share.

Format Best for Written Linux
JSON — json scripts and further automation in any language after the run yes
YAML — yaml the same data as JSON, easier to read by eye after the run yes
XLSX — xlsx sharing results with people; the one format every plan writes after the run needs libgdiplus
JUnit — junit test reports in Azure DevOps, GitLab, GitHub Actions, Jenkins after the run yes
TRX — trx test reports in Azure DevOps and Visual Studio after the run yes
SQL Server · PostgreSQL · Oracle history and reporting on top of results one row per test, as it finishes yes · yes · untested

The format token in the second column is what the Output setting takes — Output: json, junit. See Settings.

The sample run

Every format page shows the same run, so the formats can be compared side by side: three tests against one SQL Server data source, with one result of each kind.

Data sources:
- Name: DWH
  Provider: SqlServer@2
  Connection string: "%DWH_CONNECTION_STRING%"

Tests:
- Name: Airline dimension matches the source
  Suite: Smoke tests
  Description: Every airline in the source must land in the dimension, nothing more.
  First data source: DWH
  First query: SELECT 'CSA' AS Code UNION ALL SELECT 'LH' ORDER BY Code
  Second data source: DWH
  Second query: SELECT 'CSA' AS Code UNION ALL SELECT 'LH' ORDER BY Code
  Expectation: sets match

- Name: No booking without a passenger
  Suite: Data quality
  Description: A booking must reference an existing passenger.
  Data source: DWH
  Query: SELECT 'BK-1001' AS BookingId, 'CSA' AS Airline UNION ALL SELECT 'BK-1002', 'LH'
  Expectation: set is empty

- Name: Flights table is loaded
  Suite: Smoke tests
  Data source: DWH
  Query: SELECT * FROM dbo.Flights
  Expectation: set is not empty

The first test passes, the second fails (the query returns two rows where none were expected), the third ends with Error — there is no dbo.Flights table, and the provider says so:

• 00:00:00.0755756 - [Smoke tests].[Airline dimension matches the source] finished with result Passed
• 00:00:00.1389169 - [Data quality].[No booking without a passenger] finished with result Failed
• 00:00:06.2492205 - [Smoke tests].[Flights table is loaded] finished with result Error

😺 passed: 1     😿 failed: 1     🙀 error: 1     🐱 inconclusive: 0

The failed test’s message — the part every output carries as the raw message — reads:

No row was expected, but at least 1 row exists.
MaximumErrorsLogged setting is set to 1. To see more data in the error message, set it to value between 2 and 50.
If the logs should not contain any data, set MaximumErrorsLogged to 0.

┌──────────┬────────┐
│BookingId │Airline │
╞══════════╪════════╡
│BK-1001   │CSA     │
└──────────┴────────┘

CAT did NOT scan the entire set. There may be additional rows.

and the error’s: Error when executing the test: Invalid object name 'dbo.Flights'. The full message of each result adds the description, the data sources and the queries under it. See Failure message.

← Introduction
Settings →