Tolerance

Is the difference between two values negligible? Use the Tolerance property.

Two sets can differ by an amount that does not matter to the test:

Tolerance tells Sets match and Contains how far apart two numbers may be and still match. It applies to every pair of values that both read as numbers; text, dates and NULLs are unaffected.

Absolute tolerance #

Tolerance states how large a difference still counts as equal, in the unit of the values:

- Name: compare actual results with expected results
  Expectation: sets match # works for the contains expectation too
  Tolerance: 0.01
  # other test settings here

Tolerance mode: absolute is the default and may be omitted.

Tolerance in percent #

With Tolerance mode: percent the tolerance is a percentage. The value is given in percent — 1.5 means 1.5 %; CAT divides by 100.

The base of the percentage is always the higher of the two numbers.

A complete project using tolerance in percent:

Data Sources:
- Provider: Yaml@1
  Connection string: MyProject.cat.yaml
  Name: ProjectFile # hardcoded data in this file

Tests:
- Name: Within percent tolerance
  Order: 1
  Description: This should pass, as the difference is within percent tolerance
  First Data Source: ProjectFile
  First Query: /Expected
  Second Data Source: ProjectFile
  Second Query: /Actual
  Expectation: sets match
  Tolerance: 1.5 # this means 1.5 % (so it is exactly 1.5 when the base is 100)
  Tolerance mode: percent

- Name: Outside percent tolerance
  Order: 2
  Description: This should fail, as the difference is outside the defined tolerance
  First Data Source: ProjectFile
  First Query: /Expected
  Second Data Source: ProjectFile
  Second Query: /Actual
  Expectation: sets match
  Tolerance: 0.0001
  Tolerance mode: percent

- Name: Tolerance in percent the same as difference
  Order: 3
  Description: This should pass, as the difference is within the percent tolerance
  First Data Source: ProjectFile
  First Query: /Expected
  Second Data Source: ProjectFile
  Second Query: /Actual
  Expectation: sets match
  Tolerance: 1 # the difference is 10; one percent of the higher value, 1010, is 10.1
  Tolerance mode: percent

Expected:
- Year: 2021
  Profit: 1000 # ten less than in the second set
- Year: 2022
  Profit: 1500

Actual:
- Year: 2021
  Profit: 1010 # ten more than in the first set
- Year: 2022
  Profit: 1500

Defaults #

Without Tolerance and Tolerance mode, CAT uses Tolerance mode: absolute and Tolerance: 0 — the slightest difference between numeric values fails the test. A Tolerance mode other than absolute or percent is an error.

Related #

• Sets match · Contains — the expectations that read Tolerance.