CAT in GitHub Actions

Run CAT tests in a GitHub Actions workflow and publish the results as a check

Prerequisites #

• GitHub public or private repository set up
• CAT project along with test definitions created and pushed to the repository
• A basic understanding of CAT

Before you start #

A pipeline is automated use: CAT allows it on the Enterprise plan only, unlocked by a license key, which the pipeline passes to CAT from a secret variable — the Starter, Professional and Team plans refuse a run started by a CI/CD platform, and from CAT 3.0 a pipeline run without the key stops before any test runs. The examples below keep two values in pipeline variables — CAT_VERSION, the CAT version to install (pin it; upgrading is then one variable change), and CAT_LICENSE_KEY, the key, as a secret — and CAT reads the key from it; a self-hosted agent can instead have the key stored once and for all. What a pipeline needs — plan, key, tool, secrets, results, exit codes — is on one page: Integrations.

Secrets #

First of all, let’s set up environment secrets and then pass them as variables. Navigate to this page in your repo:

Click on Settings, then Environments, and New environment. Name it and click Configure environment. After this, navigate to this environment and create a secret:

The secret contains the connection string of our database. The syntax for passing it to the pipeline via Powershell is shown below.

Pipeline Code #

Once variables are set up, navigate to the pane on the top left and click on Actions and then New workflow:

This will lead you to the page shown below where you can either choose from a template or create a workflow yourself:

After clicking on set up a workflow yourself, you can start working and commit your code to repo.

Below is the workflow we used to run CAT; the .cat.yaml project file sits in the root of the repository (add working-directory: to the step when it lives elsewhere) and asks for Output: trx. Commit the workflow and it runs on the next push.

name: testing_pipeline
on: push
permissions:
  checks: write
  pull-requests: write
jobs:
  deployment:
    runs-on: windows-latest
    environment: dev
    steps:
      - name: checkout
        uses: actions/checkout@v3
      - name: run cat
        shell: pwsh
        env:
          AERO_STRING_PROD: ${{ secrets.AERO_STRING_PROD }}
          # the license key (Enterprise plan) - CAT reads the variable itself, the mapping is all it needs
          CAT_LICENSE_KEY: ${{ secrets.CAT_LICENSE_KEY }}
          CAT_VERSION: ${{ vars.CAT_VERSION }}
        run: |
          # pinned to the repository variable; change CAT_VERSION to upgrade
          Install-Module CAT -RequiredVersion $env:CAT_VERSION -Force -AcceptLicense
          Import-Module CAT
          Invoke-CatProject

      - name: Publish Test Results
        uses: EnricoMi/publish-unit-test-result-action/composite@v2
        if: always()
        with:
          files: |
            TestResults\*.trx

The workflow installs the CAT PowerShell module, reads the license key from the secret, runs the project and publishes the results with a community action that reads the TRX file (it reads JUnit too — Output: junit). The secret passed as an environment variable is the connection string the project file references. The CAT step exits 0 whatever the results, with the PowerShell module; the publish action is what turns failed tests into a failed check — see Integrations.

Results #

Navigate to the pipeline you want the results for via actions and click on the latest run. The result will look like this:

The user can see the list of the tests conducted and failed tests, along with their output as well.