CAT in Azure DevOps Pipelines

Run CAT tests in an Azure DevOps YAML pipeline and publish the results to the Tests tab

Prerequisites #

• A project in Azure Devops
• A repository containing your CAT project along with test definitions
• A basic understanding of CAT

Before you start #

A pipeline is automated use: CAT allows it on the Enterprise plan only, unlocked by a license key stored on the machine that runs the tests — the Starter, Professional and Team plans refuse a run started by a CI/CD platform, and from CAT 3.0 a pipeline run without the key stops before any test runs. The examples below keep two values in pipeline variables — CAT_VERSION, the CAT version to install (pin it; upgrading is then one variable change), and CAT_LICENSE_KEY, the key, as a secret — and set the key on the agent from it once per run; a self-hosted agent can have the key set once and for all. What a pipeline needs — plan, key, tool, secrets, results, exit codes — is on one page: Integrations.

Secrets #

To set up a variable and then set it as secret, navigate to this page:

and click here:

Once you add a new variable group, you can start creating variables. You can make these variables known or you can hide them as shown in the picture below:

There are also other ways how to define variables, feel free to do it your way. The script provided in the next section demonstrates how these variable groups can be referenced in code.

Pipeline Code #

Once you are all set up and have a variable group with the appropriate values, navigate to the pane on the left and click on Pipelines, then New pipeline:

Next, choose the location of your code. In this tutorial, we will be using an Azure Repos Git.

The next screen will ask you to choose the location of your repository:

Select Starter pipeline and replace its content with the example below, or Existing Azure Pipelines YAML file if the file is already in the repo. The example expects the .cat.yaml project file in a directory named tests in the root of the repository, and an agent pool of your own — change AeroPool to yours (a Microsoft-hosted windows-latest pool works too).

If you have an existing YAML file you would like to use, select Existing Azure Pipelines YAML file; if you would like to use our example code, select Starter pipeline.

Example code:

variables:
- group: AeroPipelineVariables

trigger: none

stages:
- stage: execute_testing_pipeline
  displayName: Execute CAT tests
  jobs:
  - job: execute_testing_pipeline
    pool:
      name: AeroPool

    steps:
      - checkout: self   # the repository that holds the project file; implicit, shown for clarity

      - task: PowerShell@2
        displayName: 'Install CAT'
        inputs:
          targetType: 'inline'
          workingDirectory: $(System.DefaultWorkingDirectory)
          pwsh: true # important! this tells the agent to use PowerShell 7
          script: |
              # pinned to the version in the variable group; bump CAT_VERSION there to upgrade
              Install-Module CAT -RequiredVersion $env:CAT_VERSION -AcceptLicense -Force -AllowClobber -Scope CurrentUser
              # can be skipped if you use self-hosted agent and the module is already installed
        env:
          CAT_VERSION: $(CAT_VERSION)

      - task: PowerShell@2
        displayName: 'Execute tests'
        inputs:
          targetType: 'inline'
          workingDirectory: $(System.DefaultWorkingDirectory)/tests
          pwsh: true # important! this tells the agent to use PowerShell 7
          script: |
              $file = "$(System.DefaultWorkingDirectory)\tests\TestingProject.cat.yaml"
              Import-Module CAT

              # the license key (Enterprise plan) - from a secret variable; skip on a self-hosted agent where it is set once
              Set-CatInstance -LicenseKey $env:CAT_LICENSE_KEY

              Invoke-CatProject -Path $file
        env:
          CAT_LICENSE_KEY: $(CAT_LICENSE_KEY)
          # if you have secret variable TESTING_PASSWORD, it can be used in your project file and CAT will understand that
          TESTING_PASSWORD: $(TESTING_PASSWORD)
          # https://learn.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&tabs=yaml%2Cbatch#secret-variables
          # no additional preprocessing or replacement steps are needed

      - task: PublishTestResults@2
        displayName: 'Publish test results'
        inputs:
          testResultsFormat: VSTest
          searchFolder: $(System.DefaultWorkingDirectory)\tests
          testResultsFiles: '**\*.trx' # Don't forget Output: trx in your project file
          mergeTestResults: true
          failTaskOnFailedTests: true
        continueOnError: true

The pipeline checks out the repository, installs the CAT PowerShell module at the pinned version, sets the license key, runs the project from the tests folder and publishes the results in TRX format. failTaskOnFailedTests: true on the publish task is what fails the pipeline on a failed test — the CAT step itself does not, see Integrations.

Note:

• Don’t forget to change the path to your project, name of the agent pool you use, change / add other secrets and environment variables etc.

• The example uses TRX format for publishing the test results, but JUnit is also supported in both CAT and Azure DevOps — see TRX and JUnit.

Results #

To find the test results, navigate to the Pipelines tab, click the pipeline you want to see the results for, and choose one of the runs:

Then click Tests:

The results should look something like this:

Classic Releases #

If you use classic releases, the integration of CAT is a bit different - actually, everything is a bit different, even the process of pipeline creation (it is not YAML code, you define them in Azure DevOps GUI). Follow Azure DevOps Classic Releases if you use classic releases.

Classic Releases documentation: https://learn.microsoft.com/en-us/azure/devops/pipelines/release/?view=azure-devops