CAT in Jenkins

Run CAT tests in a Jenkins pipeline and show them with the JUnit plugin

Prerequisites #

• Jenkins installed and set up on your machine.
• An existing CAT project file along with test definitions
• A basic understanding of CAT

Before you start #

A pipeline is automated use: CAT allows it on the Enterprise plan only, unlocked by a license key stored on the machine that runs the tests — the Starter, Professional and Team plans refuse a run started by a CI/CD platform, and from CAT 3.0 a pipeline run without the key stops before any test runs. The examples below keep two values in pipeline variables — CAT_VERSION, the CAT version to install (pin it; upgrading is then one variable change), and CAT_LICENSE_KEY, the key, as a secret — and set the key on the agent from it once per run; a self-hosted agent can have the key set once and for all. What a pipeline needs — plan, key, tool, secrets, results, exit codes — is on one page: Integrations.

Secrets #

To create variables, navigate to this page and click on Add Credentials:

There are several options (certificate, token etc.) for a new, global, environment variable:

We will use a secret text which contains the connection string to the database on which we want to run the tests:

We then reference this variable by its ID (this will be shown in the code below):

Pipeline Code #

Once your Jenkins is set up, you navigate to the pane on the left and you click New Item:

This will lead you to this page:

Click on Pipeline and then name it.

There are plenty of options for configuring the pipeline. For our purposes it is enough to just scroll down, choose try sample Pipeline and then click on Pipeline Syntax:

This will prompt the Snippet Generator page to appear. Here we can generate the pipeline syntax for various tasks within the pipeline:

Let’s fill in the code within the task we chose and then click Generate Pipeline Script. We can then copy the generated code in the pipeline:

If you are following this demo step-by-step, generating a pipeline script is not necessary as you will be provided with a sample code below; the code you will be provided with can be input here:

Below is the example code used to run CAT on a Windows agent with PowerShell 7 installed (what the CAT PowerShell module needs).

pipeline {
    agent any

    stages {
        stage('Run CAT') {
            environment {
                ConnectionString = credentials('78597b3e-XXXX-XXXX-XXXX-54e2b72f8007')
                CAT_LICENSE_KEY = credentials('cat-license-key') // secret text with the Enterprise license key
                CAT_VERSION = '3.0.0'                               // pin; bump here (or in a global variable) to upgrade
            }
            steps {
               pwsh '''
                    Install-Module CAT -RequiredVersion $env:CAT_VERSION -Force -AcceptLicense
                    Import-Module CAT
                    Set-CatInstance -LicenseKey $env:CAT_LICENSE_KEY
                    Invoke-CatProject
                    '''
            }
        post {
        always {
         junit(testResults: 'TestResults\\*.xml', allowEmptyResults : true)   // the folder CAT writes to next to the project file
        }
            }
        }
    }
}

The pipeline installs the CAT PowerShell module, sets the license key, runs the project and publishes the results through the JUnit plugin (Output: junit in the project file — see JUnit). The global secret credential holds the connection string the project file references. The JUnit step marks the build unstable on failed tests; the CAT step itself exits 0 whatever the results — see Integrations.

Note the directory workspace and within that is your item (in our case the pipeline is named “testing”, hence the folder “testing”). If neither of these folders exist, you may create them. Then copy the CAT project and test definitions to the directory (in this case, “testing”) — that is the shortcut for this demo; a real pipeline takes the project file from source control (Pipeline script from SCM, or a checkout scm step) instead.

Results #

Navigate to the pipeline you want the results for:

Open it and go to tests:

Tests in Jenkins are sorted by categories and the failed tests are highlighted: