Tuesday, September 27, 2016
While creating a testing program for enterprise mobile applications can seem like a daunting task due to the constantly changing mobile landscape, it doesn’t have to be. Here at Mobile Labs, we’ve had the opportunity to observe some of the most successful mobile device testing labs and have found that the best practitioners are able to maintain their QA testing goals while also applying solutions that solve the unique testing challenges mobility imposes on applications, devices, testers, developers and users alike.
Tip #1: Use Real Devices
Whether using a public cloud, a secure private mobile device cloud or a device plugged into a desktop, always plan to test on real devices. While many testers today use emulation technology, it isn’t the most trustworthy solution. As emulation technology uses a simulated version of the mobile application on a desktop, these tests often overlook critical errors and report false defects. For this reason, testing should always be performed on the actual devices on which the applications will run.
Once you’re ready to test, it is critical to plan for how your team will gain quick, easy access to the necessary devices. For example, consider how to accommodate the testers if they are in different places geographically—do they work from home or are they a few states away? However, keep in mind that while testing on real devices can present new administrative challenges, it is the only way to truly understand the user experience and will greatly improve the success of your mobile device testing lab.
Tip #2: Master Apple Code Signing and Provisioning
A common challenge that testers face when creating a mobile device testing lab is iOS provisioning, the process by which Apple allows an app to be installed and launched on a real device. It’s critical that developers testing iOS apps master this process as there is virtually no way – other than by jailbreaking – to avoid provisioning an app before it runs on real devices.
Today, there are several different methods enterprises use when provisioning. For example, some use a single enterprise wildcard provisioning profile and ask developers to sign apps with Xcode while others use individual app profiles. Regardless, Apple suggests using the enterprise’s Apple developer account to perform app provisioning, whether developers are in-house or external.
While iOS provisioning and code signing are admittedly difficult tasks, proper planning will give developers a head start when the testing process begins.
Tip #3: Use a Secure, Private Mobile Device Testing Cloud
Using a secure and private device cloud is a crucial for enterprises with a large number of testers, especially when they aren’t located under the same roof. This separation can range anywhere from a different building to another continent.
While testing with real devices can easily become expensive, utilizing a testing cloud keeps budgets in check by reducing project costs and achieving high ROI. Also important to note is that mobile device clouds are essential in preventing delays caused by shipping devices back and forth to testers in stationed in different locations.
When using a testing cloud, it’s important for it to be high performing and secure. Therefore, it should not run on Wi-Fi or use cellular data links to carry test traffic.
While some small businesses can gain security from the public cloud, large enterprises have more sophisticated security requirements and may have a problem meeting them if they do not have complete control over the cloud. Private mobile device clouds allow enterprises to have total control over their apps and data.
Tip #4: Don’t Get Crushed by the Numbers
According to data compiled by mobile metrics vendor Crittercism, there are 2,582 device types running 106 operating system versions serviced by 691 carriers worldwide. If the goal is to test every permutation possible, then every test case will be run 189,121,172 times!
With numbers like these it’s hard not to be overwhelmed when building a mobile device testing lab. But, fortunately, the major mobile operating systems use logical screen sizes mapped to physical screens, so testing operating system versions on representative phones and tablets will get the needed coverage.
With so many devices and operating systems, the numbers suggest a need for convergence. Your testing strategy should not be to test absolutely everything, but to simply test enough. Most enterprises test devices popular in the market and add or subtract devices as they come in and out.
Tip #5: Join the DevOps Revolution
As Apple and Android tools have at times proven to be both difficult to use and time consuming, development, IT and QA staffs are now working to automate and field their own production environments. Third parties have the means to move apps from cloud-based websites onto devices, but without a full-blown MDM, such an install requires a manual, hand-operated process. Additionally, security concerns are arising from storing apps in the cloud.
However, if you are using a private mobile device cloud, keep in mind that it must enable fast and seamless movement of an app from the build system to the device. This eliminates the need to use Apple or Google tools, or cloud-based websites. Further, upon connecting to a device, a mobile device cloud should be able to install and launch an app, offer scripting capabilities that make it possible to automatically transfer new app versions to a database and automatically run existing test cases against new app versions.
Ultimately, the mobile device cloud should make it easy for both developers and testers to enhance efficiency in getting apps out of the build system and onto a real device for testing.
Tip #6: Don’t Automate Everything
A solid mobile device testing strategy takes into consideration when to automate and when to fall back on manual testing. The fact remains that some tests are simply faster, easier and less expensive to run manually, such as trainability tests, configuration tests, performance testing and usability tests.
The best candidates for manual mobile device testing are those that are easy to carry forward onto new devices or operating systems but that might have both infrequent use and need major rewrites for new platforms.
Tip #7: Find the Developers
Communication between mobile device test teams and development teams shortens cycle times and is crucial when ensuring apps are properly signed for installation on real devices. Screen sharing sessions and shared devices makes this process quick and easy by allowing for code to be examined early in the QA process. This ultimately reduces the number of cases during which an app has trouble running on a particular class of phone or tablet.
Utilizing a mobile device cloud is a simple way to share a rare or rarely-used device so that developers can find and fix a problem. If developers have access to devices during initial code development, they can more easily ship code that works the first time. By enabling testers and developers to collaborate instantly – even when separated by an ocean –they can better re-create and understand mobile app issues. Problems can also be documented by desktop-based video recording – a movie worth a million words.
Tip #8: Don’t Try to Find the “Swiss Army Knife” of Testing
Because enterprise mobile apps almost always use back-end services, thorough testing involves both manual and automated UI testing, but may also include security testing, service virtualization, load testing, performance profiling and network condition testing. Most enterprises use tools that manage test cases – both automated and manual – and that include versioning, storage of results and the ability to data-drive and to automate concurrent testing.
While it may be tempting to look for an all-in-one tool, instead choose vendors who are open to integrations among tool sets and that show leadership in their area of specialty. For example, consider how to factor mobility into load tests, service virtualization tests or network degradation tests. While a load simulator or network virtualization framework is exercising the back-end services, the question most asked in mobility is, “What is the performance seen by a typical user under current load conditions?”
The answer? A UI automation framework that can measure transaction times and detect when the user sees the expected result, which requires an automation script run on a real mobile device. Look for tools that know how to combine traditional back-end testing with data sampled from interaction with a real device.
Initiating a testing program for enterprise mobile apps can sometimes appear to be an uphill battle. While mobility inevitably presents new challenges, once we better understand them, we often find that they have logical solutions. To help further this understanding, we’ve provided insight into what to consider when creating a device testing lab over the past couple of weeks.
Tip #9: Use What You Know
Using and implementing new tools often results in uncovering many hidden costs. Therefore, it’s important to examine the tools you already use for web and desktop apps to determine if they include the ability to extend automation to real, cloud-based mobile devices.
Using existing scripting tools for user interface testing can be effective if they (1) are compatible with underlying mobile app object structures, (2) are able to automate objects using any of the three dominant app models and (3) can run an unmodified app on a secure, real mobile device. Existing tools that meet these criteria represent a pool of expertise and infrastructure that quickly can be transferred to mobile app testing.
Key questions to consider when evaluating existing tools for your mobile device testing lab:
These questions are particularly important when evaluating open-source projects that are in mid-adoption cycle for mobile. If you need to test mobile apps now rather than later, use existing tool sets that have successfully transitioned to mobile and have a stable implementation.
Tip #10: Know What You Are Testing
Mobility has turned the client-server model on its head, thus creating a huge DevOps challenge. Desktop web tools test back-end services in conjunction with a small set of browsers on a single desktop, and the code is almost always served up by a Web server, which makes syncing their functions fairly easy. This isn’t the case for mobile apps, which install each build on a real device where it stays until replaced. Therefore, the process of quickly building production test environments is challenging, requiring matching the right app version to the right platform.
The best defense for this mobile application testing challenge is to use a mobile device cloud that automates the process of moving apps from the build system onto the right devices without human intervention.
Tip #11: Satisfy the CISO
As mobile devices depend on wireless network connectivity—public and private networks—for much of their function, testing must not only be done on real devices, but also on real devices that have public network access. Understandably, many CISOs are concerned with mixing public network access by a mobile device with the ability to test the device from a system connected to the internal LAN. To lesson this concern, device clouds must have a strategy for isolating testing assets (apps, data, and network packets) from public access or exploitation.
However, the public cloud cannot offer full control of these assets, and as a result, may not be subject to security controls and instruction analytics. While the use of devices tied to individual tester desktops solves the public cloud exposure issue, it may open a path into your internal network. The best solution is to establish a test-only wireless network for WAN access that uses enterprise standard proxy and intrusion detection mechanisms while keeping the network separate and unreachable from internal systems.
For even more information, download our whitepaper, Jump Start a Mobile Testing Lab.
Don’t forget to check out our eBook on Amazon, Enterprise Mobile App Development & Testing: Challenges to Watch Out For in 2017 !