Distributed Testing: Pros, Cons & Best Practices

In agile projects with short iterations, when new versions come up very frequently, optimization of the testing process is the main goal for the QA team. Hence, distributed testing is a very popular method as it significantly reduces the testing time. It is obtained by a simultaneous run of tests on multiple computers interacting with each other.

Let’s dive into distributed testing implementation process, it’s pros and cons as well as the way to utilize it efficiently.

The general objective of distributed testing is to run multiple tests in parallel in multiple environments. The motivation for such an activity can range from reducing test execution time to achieving cross-platform coverage.

Imagine you work on a large scale project with short iterations and you have more than 800 automated tests to run. Even high performance tests need about eight hours to do their job (depending on the test type). The team cannot wait for hours until testing is finished.

In this case, adopting distributed testing can reduce testing time to one hour (if we add five more servers and run the tests in parallel).

In order to implement distributed testing we need to take care of the following three conditions (it relates only to the distributed testing of one single test suite).

• First, all shared resources should operate correctly during the distributed testing process. The most frequent shared resource in test automation is a file system. It is especially important when files are uploaded or downloaded during the testing process.

• A separate data usage should be enabled during the parallel testing on one test stand. It’s necessary that each test uses its own data to avoid any interference with other tests.

• Finally, test that includes environment configuration changes cannot be executed “in parallel” with others.

Let us take a look at how the distributed testing process can be organized using open source tools like Selenium and Appium.

Selenium Grid

One of the most popular open source tools that supports distributed testing is Selenium Grid.

Selenium Grid allows running tests on remote machines in parallel. It represents a client-server solution comprising a master server and node servers.

Distributed testing can also be executed on mobile devices via Appium Server.

What are the advantages and disadvantages of Selenium Grid?

Advantages

1. First of all, Selenium Grid allows parallel test execution without extra efforts saving testing time. We can drastically cut time for tests execution.
2. Furthermore, it provides an option to select Runtime Environment based on the required parameters. Tests can be easily setup and supported. E.g. We launched the tests but they failed for some reason. Selenium Grid provides us with tools to diagnose the problem and reconstruct the same environment for a new test iteration.
3. It is possible to run test suites on a local machine that will be running on remote computer(s). With Selenium Grid we can assign particular tests to computers, flexibly manage test distribution to machines.

Disadvantages

1. The main peculiarity of Selenium Grid is that the code is executed on the same local machine the tests were launched, whereas the remote machine only receives the browser/mobile device control commands. It causes additional difficulties such as, uploading/downloading files, launch of any additional software on the same machine together with a browser.
2. The performance of Selenium Grid is very low in case of a large amount of node servers.
3. Significant time might be required for the implementation, if a parallel test running was not priory set up in the test automation framework.

Continuous Integration allows setting up execution of automated tests depending on the requirements (we can schedule the tests to particular time or an event).

Additionally, CI has an option to execute tests on a remote machine. The main difference is that a full copy of testing codes is executed on a remote machine rather than just transmitting browser/mobile device control commands.

Benefits

1. The CI approach provides flexibility for test planning and execution. E.g. we can set up automatic test launch upon the event ‘Project commit’, when developers commit new functionality autotest start verifying the new build.
2. It also saves test execution history to get the statistics report later on. For a large-scale projects it’s particularly important to hold the history and have accountability on the product quality.
3. As full testing code is executed on the remote machine, no extra actions from automation tests are required to implement such an approach. It means that tests which require file download/upload are easier to configure via CI.
4. Finally, it is possible to determine on which remote machine particular test suite will be running.

Limitations

1. The system provides a possibility to set up a remote testing of the entire test suite. To execute a parallel testing of one test suite, it’s necessary to divide this suite into parts or use external tools to enable it.
2. In case when a test suite is splitted, some difficulties might arise, e.g. analyzing general statistics of the whole test suite execution (there are no results on the whole suite just some separate results on each part of a suite).
3. Furthermore, it brings additional difficulties when it is required to add or delete a remote machine from the cloud environment.

Qulix QA-Team Best Practices

The best solution is to use a combination of Continuous Integration and Selenium Grid.

For the parallel execution it’s better to use Selenium Grid, however it might require some time for implementation. A practice we follow is to use test development standards which allow launching tests on Selenium Grid by default.

Nevertheless, it only regards the web and desktop testing. For testing on mobile devices Selenium Grid is by far the best tool.