In part 3 of this 5-part blog series, we explore how a solid automation testing strategy can improve agility and DevOps
How do you create an automation strategy and overcome common testing challenges?
Now that we have discussed the importance of setting up a device cloud for mobile development, testing and QA, and the benefits of testing and debugging on real devices, it’s time to move on to the next step in launching a continuous delivery strategy - setting up automation strategy by building a suite of automated tests and overcoming automation challenges.
When considering building out a suite of automated tests, it is important to consider the types of tests that make sense for your various apps. Automation testing is a great fit for apps that continuously require updates and new releases to stay fresh and relevant. By setting up automation testing, your team can test quickly and efficiently to release on time to meet demand.
When automated testing is done accurately and correctly, you can increase app quality while reducing app delivery time dramatically. This also holds true for regression testing to make sure that the software is still working properly.
Mobile Testing and Automation Challenges
Whether your enterprise mobile lab solely runs automated tests or a mix of manual and automated tests, here are 4 common challenges that you might run into when setting up your suite of automated tests for mobile.
#1: So Many Devices, So Many Operating Systems
It’s commonly said that variety is the spice of life. Well, that’s certainly true in mobility.
With so many different types of devices available today, from smartphones, to tablets, to smart speakers, and various wearables, it seems that new devices pop up daily. Along with this cornucopia of various mobile devices comes a slew of unique operating systems. Naturally, for a busy, resource-limited mobility team, manually testing mobile apps across each unique devices and operating systems can be time consuming.
Although many devices do run on the same OS platform, there is no guarantee, for example, that an app created for the iPhone will behave the same way on an iPad. Sometimes a mobile app can have a different effect on devices in the same product family thanks to elements such as screen resolution, handset, memory level, battery, and manufacturer settings.
Automated testing in this scenario brings much needed agility to what could potentially be a time-consuming process for manual testing.
#2: Tools of the Trade – Open Source Tools v. Commercial
One cannot begin automated testing without picking the right tools for the job.
Today’s industry-standard tools typically fall into two camps – open source and commercial. With so many tools available, it is important to research and consider which tools work best for your team and that best suit its skill set.
Also, when choosing a device cloud solution, you should look for one that is tool-agnostic that can support today’s industry standard testing tools for automated testing. A single automation tool may not work for every application, therefore a device cloud should be able to support many different automation tools.
We also recommend looking for a vendor who provides a solution that is flexible and able to accommodate any new tools that may be introduced. This way your team is prepared for the future and can stay nimble.
But, ultimately, how do you know which tools to choose? What are the pros and cons?
Open source automation testing tools are becoming popular because they are developed to solve a particular problem. At the beginning of mobile app development, a lot of open source tools were created because there was a void of mobile support from the big automation vendors.
Today, there are many options to choose from in the open source community such as Appium, Selenium, Calabash, etc. One reason the open source tools are attractive is because they are free to implement. The most popular tool, Appium builds on the most widely used open source automation tool for web testing, Selenium. This allows testers to utilize their existing automation framework design and skill sets for mobile. One thing to consider when using open source software is the lack of commercial support that you may get from a third-party vendor.
There are also commercial mobile automation tools available to purchase from vendors. These options usually give you a comprehensive product that fits into a larger testing suite of tools such as performance and test management tools. They often have an easier implementation and can perform a wider set of testing than some of the open source tools. Some of the tools available are HP Unified Functional Testing, IBM Rational, and Tricentis Tosca.
Finding the right tool requires that it can perform the right tests and fit within your budget.
#3: Volume of Automated Tests
Does automated testing make sense for all apps?
For example, if you have an app that is only deployed one time per year for a special event, such as a conference or meeting, or even an app that only requires a few yearly updates, then manual testing may be a better fit for your organization. If the application has a repeatable process, it is a very good candidate for automated testing.
Automated testing is done in various ways. You have automated unit tests, which should be created to cover the methods within your code. Having a comprehensive unit test suite lets you know if changes to code break existing functionality. This is done at build time and you should target to have all of your code covered by unit tests.
Functional automated tests cover the flows and features of your application and often test the application UI. The focus of functional tests should cover as much of the application as possible that has a repeatable and consistent outcome.
From the traditional standpoint of performance testing, you wouldn’t need real devices to test the performance of your systems because you would use a load testing tool that can measure servers under load. When testing the performance of your servers, it is often a good idea to test the experience that users on real devices may encounter. Does it take a long time to perform a transaction on a device? Does an error message appear because of the poor responses the app is receiving form the servers?
The other type of performance test, and one that is often overlooked, is how does the app perform in general. For instance, does the user wait for images and data to load as they scroll down the screen? Testing the performance of the application itself is important and when an application is performing well, it usually leads to a good user experience. Mobile Labs’ deviceConnect™ supports both scenarios and can give application and device metrics to pinpoint problem areas.
#4: Skilled Automation Specialists
Hiring the right automation engineers is an import aspect to making sure that your automation strategy is robust and efficient. Some mobile automation tools allow experienced automation engineers to use their same skill sets. For instance, Appium builds upon Selenium and therefore your Selenium engineers can quickly ramp up Appium automation. Our Mobile Labs Trust™ add-in for HPE Unified Functional Testing (UFT) allows automation scripters who are experienced using UFT to extend their automation into mobile.
Some automation tools use basic scripting languages and some use an object-oriented language such as Java and C#. Make sure that your automation engineers are experienced using the languages needed for automation. It may not be a good idea to turn manual testers into automation specialists because they may not be experienced in writing efficient code.
Want to learn more about continuous delivery and setting up a device cloud? Download our recent webcast to learn more about the 5 steps to launch a continuous delivery strategy and the benefits of testing and debugging on real devices.