Get Help

Organize a growing test base

One project file holds everything at first. When it stops being enough, the definitions can live in several files, in a database or in a workbook — and every CAT tool, CAT Studio included, keeps reading them.

Every definition — data source, query, test — has the same properties wherever it is stored. The project file holds them inline, or it names other places to read them from: other YAML files, a database table, a worksheet, anything a provider can return rows from. The mechanism is Lists; this page is about when to reach for it and how the tools behave once you do.

When one file stops being enough

  • Too many tests for one file. A few hundred tests in one YAML file are hard to navigate and every change is a merge conflict waiting to happen. Split them by area into files and folders; the project file lists the files.
  • The tests are about a database anyway. Then the database can hold them — rows in a table, or rows a view or procedure returns — generated, versioned and reviewed with the tools the team already uses. See Store definitions in a database.
  • The people who know the data do not write YAML. Business colleagues know the rules and some SQL; give them a worksheet with the same columns and load it.

Several YAML files 

Get list of data sources from:
- Provider: Yaml@1
  Connection string: Connections.yaml
  Query: /Data sources

Get list of tests from:
- Provider: Yaml@1
  Connection string: Tests/SmokeTests.yaml
  Query: /Tests
- Provider: Yaml@1
  Connection string: Tests/Finance.yaml
  Query: /Tests

Each file holds the same keys the project file would — Tests: with a list under it. Paths are relative to the project file.

A database 

Get list of tests from:
- Provider: SqlServer@2
  Connection string: "%DWH_CONNECTION_STRING%"
  Query: SELECT * FROM test.cat_test_definition WHERE is_active = 1

The statement can be anything that returns the columns — a table, a view, EXEC of a procedure that builds the list. Table scripts per database platform are on Store definitions in a database.

A workbook 

Get list of tests from:
- Provider: Excel@1
  Connection string: TestsFromBusiness.xlsx
  Query: SELECT * FROM [Tests$]

A header row with the property names, one test per row.

Mixing

All of these combine, and with inline definitions too — data sources inline and tests from a database is a common shape:

Data sources:
- Name: DWH
  Provider: SqlServer@2
  Connection string: "%DWH_CONNECTION_STRING%"

Tests:
- Name: Smoke test kept in the project file
  Data source: DWH
  Query: SELECT * FROM dbo.ErrorLog WHERE ThrownOn > DATEADD(DAY, -1, SYSUTCDATETIME())
  Expectation: set is empty

Get list of tests from:
- Provider: SqlServer@2
  Connection string: "%DWH_CONNECTION_STRING%"
  Query: SELECT * FROM test.cat_test_definition

What happens in CAT Studio

Splitting the project does not take CAT Studio away. Studio reads every list the project file names — YAML files, database, workbook — and shows all tests together, runs them, reports them. Tests that came from a YAML file are saved back to that file when you save; tests that came from a database or a workbook are read-only in Studio, you edit them where they live. The one thing Studio does not do is add a Get list of … from entry itself: you write those lines in the project file once, in any editor, and Studio picks them up on the next open or reload.

← Create a project file
Store definitions in a database →