Get Help

Results

Every executed test ends with one result and one message. This page lists the results and the fields every result carries.

Results

Result Meaning
Passed The queries ran and the expectation holds.
Failed The queries ran and the expectation does not hold. The message says why — see Failure message.
Error The expectation could not be evaluated. The message carries the cause: a data source or query the project does not define, a statement the provider refused (with the provider’s own error), a second-set property on a one-set expectation, a key that cannot be compared, or a Timeout that expired.
Inconclusive The test was not run: on the Starter plan, the daily limit of checks was reached before its turn, and the message says so. Counted separately from failures by every tool.

There is no other result. A test that is filtered out of a run — by name, tag or suite — has no result at all; it simply is not in the run.

Tools show the four results the same way: CAT Studio in the test list, CAT CLI and the PowerShell module on the console as each test finishes, the Python module on the returned result objects; every output writes them as text. How a pipeline turns results into a pass/fail verdict is a property of the tool — see CAT CLI run, Invoke-CatProject and get_test_results_summary.

What a result carries

Field Meaning
Test result One of the four values above.
Message The complete message — test name, queries, the failure or error text. A passed test’s message is short or empty.
Raw message The failure or error text alone, without the test name and queries.
Exception For Error: the exception details — type, message and stack trace of every exception in the chain.
Started on, Finished on When the evaluation started and finished (UTC); the duration is their difference.
Thread number The thread the test ran in — 0, 1, 2, … — for troubleshooting parallel runs.
Number of errors The total number of offending rows, when the test asked for it with Log number of errors — see Set is empty. Otherwise empty.
Execution ID A GUID shared by every result of one run — groups the tests that were run together.

A result also carries the test’s own definition — name, suite, description, data sources, queries, expectation — so an output can be read without the project file. Which of these fields each output writes, and under which column names, is on Output properties.

← Tolerance
Failure message →