Execute Tests

How to Execute Tests defined in a Project File?

Well, you defined your data source and you have your first test. Time to run it, isn’t it?

Run all your tests

Open a PowerShell session. Remember you need PowerShell 7! (see Prerequisites).

Navigate to the directory where you have you project file. 5, 4, 3, 2, 1, run your first test:

cd "D:\AutomatedTests\TryingCAT"
Invoke-CatProject

If your current working directory contains only one .cat.yaml file, it will pick it up and process it.

Alternatively, you can add a path:

Invoke-CatProject -Path "D:\AutomatedTests\TryingCAT\TestMyData.cat.yaml"

The provided path can be:

  • absolute or relative path to your project file (.cat.yaml file)

  • absolute or relative path to a directory that contains exactly one file with .cat.yaml extension (this means you can omit name of the file from the path, if it is the only one .cat.yaml file in your directory)

Result

Once you run the command, you should get results:

Executing CAT project

Might look like a mess, but actually it is handy. If that scared the hell out of you, don’t worry, GUI also exists, see CAT Studio.

What the output contains?

  • Log messages - they are crucial for troubleshooting. Virtually anything can go wrong. You can have incorrect syntax of your project file, typo in a SQL query, reference non-existing data source, missing permissions, … Logs can help you identify what’s wrong, that’s why they are turned on by default.

  • Passed, Failed or Error for each test.

  • Summary: how many tests passed, how many failed etc.

Running only part of the project

Over time, your project file will contain more and more tests. You’ll even start loading tests from other YAML files or other sources.

Often it is handy to keep related tests together, but at the same time have possibility to run only a subpart of all the tests you have. You have two options - filter and tags. These can be even combined.

Filter

You can run tests that contain some string in their full name. (Full name of a test is combination of Suite, Test Case, Order and Test name, see Test Definition for details.)

Invoke-CatProject -Filter "Airplane"

The example above will run all tests that have ‘Airplane’ in test suite, test case or in test name. To run a specific test suite, test case or combination of test suite/test case, do this:

Invoke-CatProject -Filter "[my test suite]"
Invoke-CatProject -Filter "[my test case]"
Invoke-CatProject -Filter "[my test suite].[my test case]"

Tags

You can define Tags property in your test definition (either one tag or more tags separated by comma or semicolon):

Tags: departures

In PowerShell, you can then run only tests with one of the tags you specify:

Invoke-CatProject -IncludeTag @("Departures", "Passengers")
# or simply
Invoke-CatProject -IncludeTag "Departures, Passengers"

If you specify more tags, CAT will evaluate all tests that have defined at least one of the provided tags.

Sometimes you do NOT want to execute tests with some tags. E.g., if you have some tests you don’t want to run in you pipelines, add them a tag ManualOnly. Then in Azure DevOps / GitLab / GitHub actions / Jenkins or wherever, just run:

Invoke-CatProject -ExcludeTag "ManualOnly"
← Create a Project File
Load Tests from ... →