Welcome to the Mobile Application Best Practices Whitepaper. Reed along or use the jumplinks below to read a specific subject!
The explosion of smartphone adoption in the past few years has had wide-ranging impact on Corporate America—companies are starting to leverage mobile devices and applications to reach out to their customers and improve the internal operations of their businesses. The research firm Gartner currently projects that worldwide mobile application store revenue will surpass $15.1 billion in 2011 from both consumer- and corporate-driven applications. This is a 190 percent increase from $5.2 billion in revenue during 2010. It is not a surprise that mobile-centric applications and interfaces top the list of Gartner’s “Top Ten Strategic Technologies for 2012.” By 2014, analysts expect more than 70 billion mobile application downloads from app stores every year, shifting more and more from a consumer-only phenomenon, to an enterprise focus.
Rapid growth of this application market comes with an expectation of equally rapid rollouts of application improvements and bug fixes. Because companies are eager to take advantage of this golden business opportunity, they push new applications to the marketplace quickly. The result is that QA testing, which can normally take 3-6 weeks, depending on the size and complexity of the application, is now being condensed to just one to two days. Unfortunately, these quick rollouts can become extremely expensive in terms of your company’s reputation and the actual cost to fix the problems if your mobile applications do not meet the quality your users expect.
According to a January 2010 study by Localytics, one in four downloaded mobile applications are used once and then removed from the phone or never used again. If your application falls into this category, the business goals behind the application will not be met.
- Device variety—In theory, there are many devices available that run on the same platform. However, if an application is created for one device or handset, there is no guarantee that the application will run with 100 percent integrity on another device—even in the same product family. This is due to differences in the screen resolution, the handset, memory level, battery, operating system, and differences in how manufacturers customize the OS for their device.
- Industry standards—Because mobile applications are at the beginning of the adoption phase, no industry standards have been established for mobile application testing.
- Mobile testing tool availability—Current tools for recognizing screen objects for web and PC applications do not work for applications on mobile devices. In the absence of this object recognition, complex scripting techniques are required.
- Skilled QA automation specialists—With few available tools to conduct automated testing for mobile applications, there is a lack of trained specialists available to develop the automation scripts.
These are key challenges because automated testing of mobile applications—when done accurately and correctly—can potentially reduce test development and execution time dramatically.
Today, almost all automated testing is done on applications designed for Web and Windows deployment using tools such as Quality Center® and Quick Test Professional® (QTP) from HP and Visual Studio Team® System from Microsoft. For these tools to be used successfully on mobile applications, extensive customization is required. The automation engineer must extend the tool’s capabilities by writing an advanced set of functions to replicate current testing functionality available for Web and Windows applications to mobile applications.
Many people know how to use automated testing tools, but until now organizations lacked the extremely specialized skills and depth of experience needed to customize the tools for mobile applications. Recent breakthroughs are changing that paradigm by providing a best practice approach to test automation for mobile applications. This white paper describes the components of that approach, providing you with the tools needed to reduce testing cycle times and free testers to focus on other critical tasks.
There are 7 components to developing a best practices approach to automating mobile testing:
- Feasibility analysis— determines which tests will achieve the greatest return on investment through automation
- Proof of concept— validates the value of test automation.
- Best practices test framework— provides a methodology specific to mobile applications that can serve as the foundation for the testing process going forward.
- Customized testing tools— automates and condenses the testing cycle for mobile applications by leveraging profound knowledge of advanced scripting techniques.
- Testing under real-world conditions—confirms how the application runs on the actual mobile device, in its environment, using the processor/memory of that device—not an emulator on a desktop.
- Rapid defect resolution—speeds implementation through smart, automatic creation of defect submissions, which include supplementary documentation when available.
- Reuse of test scripts— perpetuates cost savings by eliminating the need to start over whenever enhancements are made.
Before any automation is attempted, careful evaluation and analysis of the existing test cases should be performed. Test cases that are highly complex, full of technological hurdles, or run infrequently, are probably poor candidates for test automation. Based on the returns that testing automation typically generates, these types of test cases do not justify the investment. Realistically only 50-60 percent of manual test cases should be automated, according to the following criteria:
- Automate repeatable test cases. The more likely a script is to be reused, the greater the justification (and ROI) for automation.
- Apply the 20-80-20 rule. Automate 20 percent of scripts that take 80 percent of the execution time and 20 percent of the test engineers’ skills.
- Automate test cases that require comparisons of numbers and strings.
To confirm the feasibility of test automation, a proof of concept should be completed as part of any test automation effort for a mobile application. A limited number of the mobile application’s manual test scripts should be automated. To prove the value of automation, the initiative should also:
- Test the application in its environment using the physical mobile device.
- Define the expected ROI.
- Demonstrate the scalability of the process and the likelihood that test scripts can be reused on an ongoing basis.
If performed by experts who specialize in mobile test automation, the proof of concept can be delivered within 5-10 days. A proof of concept recently completed for a global retailer involved two of its most complex scripts, which take half an hour to run manually. Automation reduced the testing time to two minutes, and the scripts were available for immediate reuse in subsequent testing cycles. The analysis results from the proof of concept were continuously applied as the number of scripts increased in the testing phase that followed.
Serving as the guidelines for best practices, frameworks define the rules for implementation and testing software. Before automation begins, a test framework specific to the mobile platform and tailored to the organization’s application suite should be developed. The test automation framework is a key factor in the ability to maintain high levels of consistency and quality across testing processes.
The proof of concept example cited previously was followed with the development of a 50-plus- page framework document that addressed the uniqueness and differences of mobile device application testing. The document also addressed integration with the testing tool platforms already in use by the IT organization, such as HP’s Quality Center and Quick Test Professional.
Mobile test automation demands profound knowledge that encompasses analysis, the application itself, and the testing tools used by the IT organization. The tools must be customized to accommodate the unique characteristics of mobile platforms. In addition, an intimate knowledge of the application under test is essential.
A popular mobile automation method today is to use emulation technology, which is not valid because it uses a simulated version of the mobile application on the desktop. Test integrity is breached, critical defects are missed, and efficiency is lost due to the reporting of false-positive defects. Rather, testing should be performed on the actual devices that the application will run on, in the actual environment in which it will be used.
Furthermore, the architecture created for the automation process should provide maximum flexibility and cost efficiency. The following diagram illustrates the ideal approach for the architecture design.
In addition to automating the mobile application test, it is also possible to accelerate the resolution workflow by automating the submission of defects uncovered by the test engineers. Supplementary documentation, such as available screenshots or recorded video, is passed along to the developer as part of the submission. By eliminating the time it takes test engineers to manually compile and submit performance discrepancies, this additional automation capability allows test engineers to focus on other critical tasks. Developers can also address the defects earlier, speeding both the resolution cycle and time to market.
As shown in the diagram, the ideal architecture for a mobile test automation solution is not built around an automation tool. Instead, it is one that separates interfaces and logic. Actions that involve interfaces are wrapped in reusable functions. The IT operation gains the ability to minimize the time required to accommodate changes in the tool version and even in the tool itself. The ideal architecture solution is designed with built-in flexibility, so that it can be easily applied to new mobile devices. Thorough documentation of the testing scripts greatly facilitates any ongoing changes.
Mobile technology has its own unique development nuances and challenges. It can be extremely difficult, if not impossible, to take standard techniques used to automate testing of Web and Windows applications and directly apply those techniques to test automation in the mobile device arena. Adjustments are required, and the crossover involves a significant learning curve.
By relying on highly skilled automation specialists who understand the complexities of mobility, who are experienced in implementing mobile automation solutions, and who consistently adhere to best practices, your organization can expect to achieve an exceptional return on investment from automating the testing of mobile applications. This approach:
- Produces dramatic savings in time, effort, and cost by reducing testing cycles.
- Establishes a best-practice framework for future projects, ultimately reducing onshore or offshore development costs.
- Increases testing team efficiency, allowing releases to get to production ahead of schedule.
- Accelerates the ability to accurately identify and report defects so that corrections and implementation can proceed quickly.
- Establishes a repeatable process that can be duplicated throughout the organization to deliver ongoing time and cost savings.
Would You Like This Whitepaper As A PDF?