Get the Most ROI From Test Automation with Parallel Testing

Wednesday, March 28, 2018

The pressure to quickly deliver high quality mobile apps and mobile websites has resulted in enterprise mobility teams leveraging test automation. But, to fully get the most value out of test automation, mobile testers and quality assurance teams should implement parallel testing.

One of the biggest forces currently driving mobile innovation is digital transformation. Thanks to digital transformation, enterprise mobility teams have been presented with the task of creating innovative mobile experiences for consumers to increase productivity, improve customer satisfaction and to make it easier to access data and information.

But, with innovation and the pressure to build new mobile apps and mobile websites and to release updates faster, mobile developers, testers and quality assurance engineers have had to find new ways to increase speed and raise the quality of releases. So far, automated testing, continuous delivery and continuous deployment are the chosen methods to help enterprise mobility teams achieve the needed boost to mobile DevOps to keep up with this demand.

However, if automated testing is a key component of continuous integration and continuous deployment, then how can testers and quality assurance teams get the most value out of automated testing? The answer is parallel testing.

Why Parallel Testing?

The beauty of mobile test automation lies in its role in enabling enterprise mobility teams to test faster and more efficiently. When compared to manual testing, which can be a lengthy and cumbersome process, automated testing by design is faster with the right resources in place.

Resources include having the proper automated testing tool or tools in place and enough real devices for testing. Yet, if resources are stretched, running one automated test at a time can still be a time-consuming process.

By definition, parallel testing is the act of running automated tests concurrently instead of running them sequentially. In other words, with parallel testing, enterprise mobility teams can boost DevOps and increase efficiency by running multiple tests at one time, instead of one at a time.

Therefore, to fully realize the return on investment (ROI) of automated testing, enterprise mobility teams should implement parallel testing for maximum efficiency and speed.

Although automated testing is faster than manual testing, in reality, test automation may only be marginally faster. This distinction will really depend on the number of apps or mobile websites that need to be tested and the number of available devices for testing. But, in many cases test automation with sequential tests may not be enough to move the needle when it comes to speed and agility.

Automated tests are good because they are repeatable, accurate and do not require a human to run. But, when tests are run one at a time, in sequence, they are still slow, while with parallel testing, test automation becomes faster.

The True ROI of Parallel Testing

With parallel testing, the time it takes to complete the full test suite is shorter because tests are run simultaneously. For mobile, enterprise mobility teams will want to run the same tests on many different combinations of devices for the greatest efficiency. Using parallel testing, testers and quality assurance can run one test against several devices at one time.

Here is an example of how this would work.

Consider an enterprise mobility team that has to run 100 tests, on an iPhone, iPad, Android Phone, and an Android Tablet. Or in other words, 100 tests need to be run against four OS/device combinations. On average, running each test takes around three minutes.

Without parallel testing, it takes the team 20 hours to run all 100 tests, one at a time across the four devices. By implementing parallel testing, the team can run 25 tests in parallel which reduces the time it takes to run the whole test suite to 48 minutes.

Parallel testing results in a greater ROI for test automation because it reduces the time to execute all tests with only slightly more resources. Thanks to the boost in efficiency as a result of parallel testing, any additional funds spent for additional resources or infrastructure to accommodate parallel testing are offset by the time saved in running all tests.

To get a better idea of how running tests in parallel increases efficiency, examine the table below.

As more tests are run in parallel, the time it takes to complete the entire test suite is shorter. As an added bonus, parallel testing enables enterprise mobility teams to know the results of tests faster, meaning that any bugs can be found and fixed quickly.

Getting Started with Parallel Testing

As the numbers suggest, implementing parallel testing has vast benefits for enterprise mobility teams who are under pressure to develop, test and release mobile apps and mobile websites faster than ever.

For teams that want to implement parallel testing in their mobile testing lab, teams should consider the following when building their testing strategy:

#1: Start with a test automation framework that has built-in parallel testing

When implementing parallel testing, it will be easier to use a testing framework that handles parallel testing instead of writing a custom parallel testing framework. For most teams, managing the threads that spawn parallel tests can be cumbersome and error prone. In fact, poor implementations can cause collisions in processes if the threads are still open.

This is particularly true for Appium, where Appium sessions may not be fully shut down. New threads can cause the old test sessions to stop and fail.

In addition, a custom parallel testing framework adds extra code for teams to maintain. With a custom framework, less time is spent writing tests, because teams will spend time on maintenance and upkeep. By using an automated testing tool with built-in parallel testing capabilities, the tool or framework manages the code for you.

#2: The automation tool or framework that is used for test automation will determine the type of parallel testing and resources that are required for implementation.

Appium Users

For teams leveraging Appium, testers can choose a test framework like TestNG, JUnit or Nunit which allow for multiple tests to be run from the same computer. Because parallel testing is built into these frameworks, the parallel tests spawn new threads on the computer to execute the tests.

Mobile Labs’ GitHub has several examples of frameworks for Appium users interested in parallel testing.

For TestNG:

For JUnit:

UFT Users:

Teams using UFT for automated testing will need multiple computers with UFT to run tests in parallel. The tests will be controlled by ALM, which will manage the test distribution.

#3: Testers and quality assurance teams have to make sure that the resources or dependencies of tests are managed appropriately.

For example, a test that requires another test to run before it cannot run at the same time as the dependency test. To prevent any issues, there may be some serialization involved with implementing parallel testing.

Leverage Parallel Testing with a Mobile Device Cloud Solution

To get the most value out of test automation and parallel testing, incorporating a mobile device cloud solution can help manage the resources and processes required for execution. By having one central platform from which to manage and share devices and to execute all tests, mobile developers, testers and quality assurance members can easily collaborate.

Mobile Labs’ private device cloud solution, deviceConnect™ is tool-agnostic and supports most open-source and commercial testing frameworks, including Appium and UFT. Available as an on-premises and as a hosted solution, deviceConnect offers flexibility for teams who are geographically dispersed who need to share the same device pool for development and testing.

For example, deviceConnect will handle the device selection and retention automatically. This means that when tests are running in parallel, deviceConnect will find another available device to use for testing.

Also, for teams leveraging Appium, deviceConnect comes with a built-in Appium server. Because the server is built-in, deviceConnect offers better performance than using Appium on its own.

Therefore, not only does parallel testing make test automation more efficient, a mobile device cloud solution can add an additional boost to mobile DevOps through its device management and sharing functionality in one solution.

Discover how deviceConnect helps enterprise mobility teams leverage automated and parallel testing by watching our video demo.

Steve Orlando

Steve Orlando is a seasoned development and quality assurance professional with experience testing and developing mobile, Web, mainframe, CRM and desktop applications. In his role as director of product marketing for Mobile Labs, Steve drives the design and implementation of the company’s private mobile device clouds, GigaFox™ and deviceConnect™.

More Posts | Website