Tests

Learn how to create, edit, run, and manage automated tests in CAT Studio.

Tests Page Structure

Tests page displays quite a lot of things. In general, the page is split to two parts - left contains list of tests and right detail of one test.

There is a splitter between the two, you can choose whether you focus on list of tests or on detail of one of them.

On the left list of tests, on the right detail of a test

List of Tests

List of tests has this structure:

Toolbar, followed by grid with tests, progressbar and statistics

The toolbar contains action related to more tests.

The data grid with list of tests is very advanced. Try to right click header of any column - there is lots of magic. Searching, sorting based on more columns, filtering, columns chooser, grouping, custom filters and much more.

Below the grid there is a progress bar and statistics about the last run. When you don’t run all tests in your project, the total does NOT correspond with number of tests in your project.

On the right side, you see how many tests in total you have in your project. (Project, 337 on the picture). This is total number of tests in your project and is not affected by filtering in the grid.

Focused Test Details

Test detail part is split to two tabs:

  • Query tab - it contains one or two queries and your expectation about results
  • Test tab - attributes of test that you built on top of the query/queries - such as test name, categorization etc. It also contains result of last test run.

Two tabs, named “Query” and “Mandatory columns sys_ingested_at in table aodb_gates”

Query Tab

On Query Tab you need to specify one or two queries and what you expect from them.

Query tab contains these parts:

Query Tab split to 5 parts

Expectation
Here you set what do you expect from the query. Should it return nothing? Should the result be the same as result of another query? See Expectations. Depending on expectation you select, additional settings may appear. If you are just starting wiht CAT, the defaults should be OK for you. For details, see documentation of the selected expectation. Some expectations require two queries (sets match, first contains second, second contains first) - in this case CAT Studio displays two queires on the query tab.
Query Toolbar
On the left, there is a dropdown for selecting data source. Query will be run against the selected data source. When creating new tests, CAT automatically adds the first one in your project. When you have more data sources, pay attention to this dropdown. There is also a button for running the query.
Query Editor
Here you can edit your query (SQL, DAX, MDX - depends on what data source you are querying). At the moment, there is no smart autocomplete or intellisense in place, just syntax highlighting (SQL).
Query Results
The results of your query. You can use advanced grid features - right click any column header to see options you have. Remember the query results are kept in memory - the editor is not yet designed to work with millions of rows. Tip: you can save your project and reload it - this way you get rid of any datasets you don’t work with.

Test Tab

On test tab, you automate your test. Here you give the test a name, description, categorization info, tags etc. It also contains the results of the test.

Tab with lots of information about the selected test

Detail Info About Test
Give the test a short descriptive name. Optionally add tags separated by comma or semicolon. Fill description with information why the test exists and what to do when it fails.
Categorization
CAT organizes test into this hierarchy: Test Suite -> Test Case -> Test. The order setting is metadata only - CAT does NOT use it for ordering the tests when you run them.
Other Test Settings
Timeout is total time in seconds for CAT to try retrieving results of query or queries, before it stops trying and ends the test with error timeout. In test results, CAT provides examples of erroneous data. By default it does not display lengthy strings there - long strings are truncated if they are longer than 25 characters - you can change this behavior using the “Maximum Sample Column Length” setting.
Test Result
The result of the test - whether it passed, failed or ended up with an error. The result of a test with failed or error results contains descriptive information about the findings.
Status Bar
Shows statistics about the test execution, such as execution time.

Query, Test - But What is the Difference?

There is a difference when running your query and when running your test. Those are two different actions triggered by two different buttons. The difference might seem confusing.

You run the query to check what it returns. You do it when you are authoring or tuning a test or troubleshooting a failed test. This just returns to the grid the result of the query, as is. Just like in DBeaver, SSMS or other tool.

You run the test to verify your expectation in automated way. You usually run many test at once. This does NOT return the result of the query as is. CAT is trying to check whether your expectation is met and return sample of problematic rows. CAT does not even have to read all the query results if it spots problems earlier.

Simply said - on query tab, you get raw result of the query. On test detail tab, you get result of what you are testing.

← Data Sources
Queries →