Run Tests in Parallel

How to execute tests simultaneously in more parallel executing threads?

By default, CAT runs tests one by one untill all of them are executed. It does not start execution of a test until the previous one is completely evaluated.

Why Should I Care? #

If you have many tests and their execution takes long time, you should consider running tests in parallel. That means CAT will not execute one test, but two or more tests at a given point in time.

How Do I Set It? #

As all other stuff in CAT, you manage the degree of parallelism in your CAT project file:

# multithreading settings:
Threads: 3

Put that to the root of your project file - that means there must be no whitespace before the Threads keyword.

Setting the value to 1 is the same like deleting the line - it has the same effect. CAT will use only a single thread.

Setting the value to 0 or less will generate an error.

What Does It Do? #

If you set Threads to 3, like in the example above, CAT will do this:

• It creates a queue of tests to be executed (this one is common for all threads).

• It starts 3 threads, all of them run independently on each other at the same time.

• Each thread “steals” one test from the queue (so the others will not see that one anymore) and evaluates it.

• A thread finishes once if finds out the queue is empty.

All results from all threads are assembled together and processed (to MS Excel file, JSON, TRX etc.). This is already done in a single (in the “main”) thread.

Best Practices #

• If you don’t have any reason to use parallelism, don’t use it.

• If your tests run long time, consider trying out the parallelism in “steps”, like first in two threads, then in three…

• If you are about to tweak the parallelism, communicate this with your database administrator(s). Let them know you might create a heavier load.

• Monitor with your DBA the traffic your tests create on the servers.

• If you use Excel@2 provider, beware that every thread first loads all the specified worksheets to memory. Especially if you test worksheets with hundreds of thousands of rows, avoid using parallelism or separate those tests into a dedicated project file that uses one thread.

• In log files you can find details about what thread executed how many tests, when tests started etc. This may help you understand what CAT does and when. This is only available in Debug logging level. Use it only for troubleshooting or evaluating how CAT executes tests.