Last updated July 28, 2016
Too many devices, too little time to test.
It often seems like an overwhelming task to choose which devices you will use for mobile application testing. Today’s growing reliance on bring-your-own-device (BYOD), makes deciding which devices to use for mobile app testing even more difficult.
It’s easy to fall into thinking that your app has to be tested on every possible mobile device.
Some app testers may also get distracted by variables such as the carrier’s network — questioning whether an app will run on the Verizon LTE network as well as it does on other networks. However, apps aren’t built that way. We don’t write code customized for a specific carrier network.
Let’s shift gears for just a moment and think about how apps are tested in the PC world.
There are numerous types of PC’s in any given company: Lenovo, Dell, ASUS, HP and Samsung, just to name a few.
Is QA busy testing to ensure each app runs on each type of PC? No.
QA focuses on testing the app – the software – that needs to run on PCs. Mobile apps aren’t really that different.
Today’s mobile testing professionals need to test the apps themselves to ensure the software functions appropriately. It’s not about the hardware; it’s about the software.
Focus on Testing App Functionality
When it comes to testing mobile applications, we recommend taking an approach that focuses primarily on testing the app’s functionality – rather than the device features. At the end of the day, what you really want to know is, “does the software simply work the way it’s supposed to?”
The next most important criterion is the performance of the app on various mobile platforms.
To make your life a little easier, here is a list of questions to help you narrow the focus to only the most important criteria for choosing devices for mobile app testing:
1. Are you building a consumer or an enterprise app?
If it’s a consumer app, like a shopping app, for example, you will typically want to choose more devices for mobile app testing than you would need to test an enterprise app.
In many enterprise organizations, even when BYOD is supported, there is often some standardization as to supported devices and operating systems (OS’s).
While not everyone in the organization may follow the policy, IT often has a target list of devices and OS’s on which an enterprise app should run. The same is not true for consumers – there is no control over the device or age of the OS on which that shopping app may run, so testing needs to include a larger pool of devices and OS’s.
2. What platforms and operating systems apply for your app?
In some cases, an app is written to run on a very specific device type. In those cases, the app only needs to be tested for that use case.
For example, United Airlines has an app for its electronic flight bags. That app is written specifically for the iPad® so there is no need to test the app on Android™ tablets or any type of smartphone.
While it’s not always this cut and dry, most enterprise and consumer apps have common platforms on which they need to run. Most often, those include iOS, Android and possibly Blackberry devices. You’ll need to factor in various OS versions when testing apps on those device types, but you don’t have to go buy or rent access to every flavor of mobile device on the market.
3. Are there hardware specifications that factor into testing your app?
Look at which processors are on the device, including graphics. For Android devices, this is especially important because each processor may make the operating system work differently. Some features may not work with certain processors. For media-rich or gaming applications, this is important.
Also, hardware acceleration can make an application perform better, so it’s important to test your app on a device with a processor that has this feature as well as on one that doesn’t. If you’re building a consumer app, you might want to support streaming media or other major graphics.
On the other hand, a bank or insurance app may not require a robust graphics processing unit (GPU). Your selection of devices will be different depending on these hardware specifications.
When you consider that there are close to 100 Android devices that run the Adreno chip for graphics, and then factor in the variations in SoC types, you end up with a lengthy list of devices.
Does that mean you test all of them? No, you find the devices that have commonalities for processing and graphics support and test the app on those select devices.
At the end of the day, you could test on as few as 10 or 20 devices, using different hardware configurations and you could effectively test your app to make sure it will work with many more devices than those on which you tested.
4. What is the screen size?
If you have an app intended for both smartphones and tablets, how does the app make the transition from one form factor to the other?
Or, if you have an iOS app intended to run on both iPhone® and iPad®, which device-specific controls, features, layout, interfaces, etc. need to be taken into consideration for app testing purposes?
For example, if you’re testing the iOS notes app the view is different depending whether it is launched on an iPad or an iPhone. With the iPad, the UI for the notes app includes a “table of contents” section that runs vertically on the left side of the screen to show all of the notes saved on the device.
However, the iPhone version of the app does not contain the same field. Since a tablet screen is bigger, you can have more advanced app functionality and need to test for such differences.
The point is that asking these critical questions can help you focus your testing efforts.
5. Which manufacturing models should I test?
There could be hundreds of potential Android device models that you could use for mobile application testing. However, in many cases, you may only need to test a single flavor of each of the most popular devices – for example, Samsung, HTC, LG, Motorola and Sony.
One question to ask yourself is, “what is the market share for use of a particular device?”
If only 5% to 10% percent of the market uses one type of device, you may only need to spend a little time testing on it – if you test it at all. Instead, you can focus your efforts testing apps on the more widely used devices.
With the proliferation of mobile devices in the enterprise and new models and OS’s coming out constantly, it’s next to impossible to certify every device and test every device and OS combination on the market.
The bottom line is that testers need to be smart about choosing what to test and what not to test.
You have to make tradeoffs because it isn’t cost effective to buy or rent every new device that comes out so you can test your app. Ask some strategic questions about the nature and use of your app and you can narrow down the device field considerably.
Want to learn more? Click to see our recent posts!
Don’t forget to download our eBook on Amazon, to stay ahead of the curve in 2017!