Parallel Test Execution Strategy

Parallel test execution is key to executing the test pack faster. In agile development when your tests run in a build pipeline, it is expected to finish in a few minutes to speed up the delivery process. Many teams struggle to minimise the execution time because of flawed or missing parallel test execution strategies. 

At what stage parallel test execution is configured?

There are a minimum of 2 stages where parallel test execution is configured in a standard test automation framework.

1. CI Pipelines

Rather than executing all tests in a single test run, tests can be distributed across multiple test suites. These test suites can execute in parallel as multiple tasks at the test run step in the build pipeline.

For example,

• There are 300 tests in your test pack.
• These 300 tests can be distributed across 10 test suites, an average of 30 tests per test suite.
• At the test execution step in a pipeline, you can execute these 10 test suites as 10 tasks in parallel.
• Provided that there are enough parallel execution threads available, 300 tests can be executed in the time to run 30 tests.
• Hence it’s possible to reduce the overall test execution time by 10 times. An hour-long test execution can be achieved in 6 minutes.

2. Test Suites

Most frameworks available in the market allow parallel execution of tests at the test suite level, where multiple tests in a test suite are executed in parallel.

For example,

If you are using the Cucumber framework, you can define the parallel execution count in a configuration which decides the number of features that can be executed in parallel. This count should be defined carefully considering the number of parallel execution threads and a number of test suites executed in the build pipeline.

How many parallel execution threads are enough?

There are 3 criteria to decide the number of parallel threads.

1. Maximum Parallel Threads

Maximum execution speed can be achieved when you have maximum parallel execution threads.

Maximum Parallel Threads = Total number of tests in all test suites executed.

2. Optimum Parallel Threads

Maximum Parallel Threads are not recommended because

a) It's expensive and

b) Execution times of all tests are not the same. Every test suite has longer and shorter execution tests, so when a longer execution test is running, a single thread can be used to run multiple shorter execution tests.

Hence an Optimum Parallel Thread is recommended, which can be calculated by following simple calculations.

Number of Longer Execution Tests (LT) = Number of Tests with execution time greater than average test execution time.

Number of Shorter Execution Tests (ST) = Number of Tests with execution time shorter than average test execution time.

If ST > LT, Optimum Parallel Threads = ST,

otherwise if LT > ST Optimum Parallel Threads = LT.

3. Minimum Parallel Threads

Minimum Parallel Threads are recommended when :

a. You are not worried about the total test execution time and it doesn’t delay the release or delivery.

b. Minimum acceptable execution time is defined and it's generous.

b. You are trying parallel execution first time.

c. Budget constraints.

The minimum Parallel thread count can be calculated as follows.

Average Number of Tests Executed in the acceptable time slot (AT) =

Minimum Acceptable Execution Time / Average Execution Time per test

Minimum Parallel Threads = Total Number of Tests / AT

Example:

Total number of Tests = 300

Number of test suites = 10

Average number of tests in each suite = 30

Number of Longer Execution Tests = 170

Number of shorter Execution Tests = 130

Minimum Acceptable Execution time for the test pack = 15 minutes

Average Execution Time per test = 1 minute

Average Number of Tests Executed in the acceptable time slot (AT) = 15

Maximum Parallel Threads = 300

Optimum Parallel Threads = 170

Minimum Parallel Threads = 20

What should be the total execution time for the given parallel execution threads?

Total execution time =

(Total number of Tests / Parallel Execution Threads) * Average Execution Time.

For example,

Total number of Tests = 300

Parallel Execution Threads = 50

Average Execution Time = 1 minutes

Total execution time = (300 / 50) * 1 = 6 minutes.

What other factors delay the test execution?

There are many factors which can delay the overall execution and should be added up to the numbers calculated above.

1. Internet speed
2. VPN requirement
3. Slower machines provided by parallel threads
4. Entry and exit criteria for tests
5. Open and pending parallel thread sessions
6. Parallel threads queues - delayed thread release
7. Test report generation

Distributing tests across multiple test suites is key to speedup the test execution. Considering the above parameters of your test packs and selecting the correct parallel thread strategy will help to improve the total execution time hence reducing the overall build and release time exponentially.