CAT in GitLab

Run CAT tests in a GitLab CI/CD pipeline and show them on the pipeline's Tests tab

Prerequisites #

• An existing project in GitLab
• A repository containing CAT project files along with test definitions
• A basic understanding of CAT

Before you start #

A pipeline is automated use: CAT allows it on the Enterprise plan only, unlocked by a license key stored on the machine that runs the tests — the Starter, Professional and Team plans refuse a run started by a CI/CD platform, and from CAT 3.0 a pipeline run without the key stops before any test runs. The examples below keep two values in pipeline variables — CAT_VERSION, the CAT version to install (pin it; upgrading is then one variable change), and CAT_LICENSE_KEY, the key, as a secret — and set the key on the agent from it once per run; a self-hosted agent can have the key set once and for all. What a pipeline needs — plan, key, tool, secrets, results, exit codes — is on one page: Integrations.

Secrets #

As shown above, the environment variables can be defined in Settings of your project, in section CI/CD. You can modify the variable itself to make it secret or determine the scope:

When you define environment variable TESTING_PASSWORD, you can use it in your CAT project file as %TESTING_PASSWORD%. CAT will automatically replace it. You don’t need any preprocessing steps to tweak your project file.

Pipeline Code #

Once your project is set up in GitLab, navigate to the Build icon on the left pane, then click on Pipeline editor, then Configure pipeline as shown in the image below:

This will lead you to this page:

Here you can write your custom code to the pipeline using the editor and commit the changes. If you do not change the settings, it will automatically trigger the pipeline you just created. If the pipeline does not exist, it will create it in the root folder of your current repository.

Below is an example of the code we used to run CAT in the official PowerShell 7 container image (PowerShell 7 is what the CAT PowerShell module needs); a Windows runner with PowerShell 7 works the same way.

scriptjob:
  stage: test
  image:
    name: "mcr.microsoft.com/powershell:latest"
  script:
    - |
      pwsh -c '
        # CAT_VERSION and CAT_LICENSE_KEY are CI/CD variables of the project (the key masked)
        Install-Module CAT -RequiredVersion $env:CAT_VERSION -Force -AcceptLicense
        Import-Module CAT
        Set-CatInstance -LicenseKey $env:CAT_LICENSE_KEY
        Set-Location tests          # the folder with the .cat.yaml project file
        Invoke-CatProject
      '
  artifacts:
    when: always
    paths:
      - tests/tests.xml
    reports:
      junit: tests/tests.xml

The job installs the CAT PowerShell module, sets the license key, and runs the CAT project that lives in the repo; the connection string comes in through the CI/CD variable the project file references:

The repo also hosts definitions of these tests:

At the end of the job GitLab collects the JUnit report — tests/tests.xml, written by CAT next to the project file because the project file says so:

Output:
- Format: junit
  File: tests.xml

See JUnit. GitLab marks the pipeline failed only when the job fails; the CAT step exits 0 whatever the results — see Integrations for how to fail the job on a failed test.

Note:

Don’t forget to change name of your project, change and add other secrets / environment variables if needed etc.

Results #

Navigate to the pipeline you want the results for:

Open it and go to tests:

Tests in GitLab cannot be sorted by any category.