Tolerance

Is the difference between values negligible? Use Tolerance setting.

Occassionally you might be comparing two datasets and find out your test is failing because of very small difference between the two:

Absolute tolerance #

If such difference is of no importance for your test, you can instruct CAT to “tolerate” such differences. Typically you specify what difference is still “OK” in Tolerance setting. Just add this to your test definition:

- Name: compare actual results with expected results
  Expectation: sets match # works also for "contains" expectation
  Tolerance: 0.01
  # other tests settings here

You can also set Tolerance mode: absolute, but that is default, so you can omit that.

Tolerance in Percents #

You can also set tolerance in percents and don’t care about absolute numbers.

There are two values that are being compared. So, what is the base for calculate the percentage? In CAT it is always the lower of the two numbers. This is probably a minor thing (the values are supposed to be almost the same anyway), but it’s good to know about that.

You provide the value in percents (don’t divide by 100 yourself, CAT will do).

Example project that uses tolerance in percents:

Data Sources:
- Provider: Yaml@1
  Connection string: MyProject.cat.yaml
  Name: ProjectFile # hardcoded data in this file

Tests:
- Name: Within percent tolerance
  Order: 1
  Description: This should pass, as the difference is within percent tolerance
  First Data Source: ProjectFile
  First Query: /Expected
  Second Data Source: ProjectFile
  Second Query: /Actual
  Expectation: SetsMatch
  Tolerance: 1.5 # this means 1.5 % (so it is exactly 1.5 when the base is 100)
  Tolerance mode: percent

- Name: Outside percent tolerance
  Order: 2
  Description: This should fail, as the difference is within outside defined tolerance
  First Data Source: ProjectFile
  First Query: /Expected
  Second Data Source: ProjectFile
  Second Query: /Actual
  Expectation: SetsMatch
  Tolerance: 0.0001
  Tolerance mode: percent

- Name: Tolerance in percent the same as difference
  Order: 3
  Description: This should pass, as the difference is the same as defined percent tolerance
  First Data Source: ProjectFile
  First Query: /Expected
  Second Data Source: ProjectFile
  Second Query: /Actual
  Expectation: SetsMatch
  Tolerance: 1 # difference is 10, which is exactly one percent of the lower value
  Tolerance mode: percent

Expected:
- Year: 2021
  Profit: 1000 # ten less than in the second set
- Year: 2022
  Profit: 1500

Actual:
- Year: 2021
  Profit: 1010 # ten more than in the first set
- Year: 2022
  Profit: 1500

Defaults #

By default, if you don’t specify Tolerance and Tolerance mode, CAT uses Tolerance mode: absolute and Tolerance: 0, which means the slightest difference between numeric values will cause your test to fail.