Kobiton has acquired Mobile Labs, Inc

Should I Use a Simulator or Real Device When DevTesting Mobile Apps?

Wednesday, April 25, 2018

While simulators and emulators play a role in the mobile app development and testing process, there are certain tasks that are best suited for a real device. Mobile Labs’ checklist will help you decide which to use for mobile app development and testing.

There never seems to be a shortage of components and features to test in enterprise mobility. From testing code in the development phase, to testing functionality, to testing device components and operating systems, mobile developers, testers and quality assurance teams have their work cut out for them.

In response, many enterprise mobility professionals rely on simulators and/or emulators to expedite the testing process.

At Mobile Labs, we have always advocated for testing on real mobile devices. We believe that testing on a real device, in users’ hands, is the best way to get the comprehensive picture of how an app or mobile website will behave in the real world.

However, we also recognize that many teams may not have the resources in place to completely rely on testing on real devices. This is where simulators and emulators are effective because they can enable busy mobility professionals to test certain elements quickly, even if they do not have access to the actual device. But, while simulators and emulators certainly have their place in a mobile testing lab, there are certain tests that are best left to using real devices.

How can you tell when a specific test or activity is best suited to be tested on a real device vs a simulator or emulator?

Mobile Labs recently put together a checklist, Should I Use a Simulator/Emulator or a Real Device When Developing/Testing Mobile Applications?” Enterprise mobility teams can use this checklist as a guide to determine what tests should be conducted using a real device or when a simulator or emulator can be an appropriate substitute.

Checklist: Should I Use a Simulator/Emulator or a Real Device When Developing/Testing Mobile Applications?

Click here to download a PDF version of this checklist

Although this checklist is a useful resource to consult at a glance, it is important to dig deeper and to gain a better understanding of the relationship between simulators/emulators and real devices.

Simulators and Emulators: What is the Difference?

Although these terms are often used interchangeably, there is a marked difference between a simulator and an emulator. Below are the traditional definitions of simulators and emulators normally found in the technology world.

A simulator is a program that enables a computer to execute programs written for a different operating system. While an emulator is hardware or software that allows one computer system (known as the host) to behave like another computer system (referred to as the guest). In this host and guest arrangement, an emulator allows the host system to run software or use peripheral devices, such as printers or USB devices, that are designed for the guest system.

But, for enterprise mobility professionals, the definitions of simulators and emulators are different based on whether you are dealing with Apple or Android.

Apple uses the term simulator to describe their program that enables enterprise mobility professionals to run and test mobile apps on a computer without having to use a real device. Android uses the term emulator to describe their version of this program. Therefore, in this blog the term simulator will be used when discussing Apple, while emulator will be employed when discussing Android.

Why Use Simulators or Emulators?

In a perfect world, with unlimited resources, testing on real devices is preferable. Although Apple Simulator and Android Emulator are powerful, robust tools, they do not completely replicate the full experience of using an actual device by not taking into account hardware elements such as the battery, CPU, networking, etc.

Below are a few reasons why enterprise mobility teams may choose to use an Apple Simulator and/or an Android Emulator as a substitute to testing on a real device.

#1: Not Every Organization Can Afford Multiple Real Devices

With so many different devices on the market today, organizations sometimes cannot afford to buy and manage multiple versions of costly devices for developers and testers. Without enough real devices to go around, it is more cost-effective, not to mention efficient, for team members to use simulators and emulators for testing.

#2: Ability to Mock Up an Application

For busy enterprise mobility teams, simulators and emulators make it easy for developers and testers to quickly mock up an application. In the mock up, quick tests can be conducted, such as testing different screen sizes to verify layouts, without having to have a real device with each screen size for testing.

#3: Easy Debugging/Testing Before Code is Committed to Source and Deployed

For mobile developers, simulators and emulators enable teams to quickly verify the application state when certain UI events are triggered. Running these tests on a simulator or an emulator enables teams to make any adjustments to code before committing it to the source code or to actual deployment.

#4: Ability to Run Unit Tests

By using a simulator or an emulator, testers can run unit tests in a build system without having to use a real device.

#5: Browser Simulation with Google Chrome for Mobile Web

For developers and testers working with mobile websites, using a simulator enables the use of Google Chrome for browser simulation. With this simulation, testers can see how a page renders by screen size and can quickly make code changes and refresh the page to check the rendering. As an added benefit, the browser simulation with Google Chrome still enables teams to access the Chrome developer tools.

Comparison: Apple Simulator v. Android Emulator

For mobility teams working with both Apple and Android, there are some key factors to consider when using Apple Simulator or Android Emulator.

First, when working with iOS it is important to note that Apple’s code is compiled differently. Therefore, an app built for a simulator will not run on a real device as the architectures for the simulator are built for the Macintosh chips and the real devices are built for ARM chips.

Developers and testers should run the app from Xcode and choose a simulator from the dropdown menu located by the run button, as seen below.

For developing and testing for an Android device, developers and testers should run the app from Android Studio by hitting the run button. Next, choose the configured Android Virtual Device in the “Select Deployment Target Window,” as seen below.

Real Device Testing = Real Results

Although Apple Simulator and Android Emulator can help developers and testers quickly test certain elements of mobile apps and mobile websites before deployment, the majority of important tasks and tests are best suited to be tested on a real mobile device.

Consider this excerpt from Apple’s Simulator Help Overview (emphasis added):

“Simulator is a great tool for rapid prototyping and development of your app allowing you to see the results of changes quickly, debug errors, and run tests. It is also important to test your app on physical devices as there are hardware and API differences between a simulated device and a physical one. In addition to those differences, Simulator is an app running on a Mac and has access to the computer’s resources, including the CPU, memory, and network connection. These resources are likely to be very different in capacity and speed than those found on a mobile device requiring tests of performance, memory usage, and networking speed to be run on physical devices.”

Source: https://help.apple.com/simulator/mac/current/#/dev...

Next, consider this quote from the Android Studio User Guide (emphasis added):

“When building an Android app, it's important that you always test your app on a real device before releasing it to users.”

Source: https://developer.android.com/studio/run/device.ht...

Despite having built a powerful Simulator (Apple) and an Emulator (Android), both vendors still advocate for testing on a real device. Clearly, while simulators and emulators have their role in the mobile development and testing process, they should not be considered a replacement for testing on a real device.

For real results and to get the most accurate insight into how an app or mobile website will function in the real world, enterprise mobility teams should test on real devices for the following reasons:

#1: Testing on a Real Device Gives a True End-User Experience

As Apple noted in its Simulator Help Overview, there will be differences between a computer’s CPU, memory and network connections and those of a mobile device. Because a mobile app or mobile website uses the device’s CPU, memory and networking, running tests on a simulator or an emulator will not accurately predict the effect of these device elements on the mobile experience.

#2: Influence of Manufacturer on Android Devices

Because Android runs on several devices from different manufacturers, there can be different modifications by each vendor that Android Emulator may not be able to effectively test. In this situation, enterprise mobility teams should use real Android devices, running a real OS across different manufacturers for the best results.

Android Operating Systems are also different between manufacturers. These different Operating Systems are not loaded into an Android Virtual Device, making testing just with an emulator virtually impossible.

#3: Use the Real Device Hardware Features for Testing

The best way to effectively test hardware features such as the device’s camera, GPS, battery, etc., is on the actual device itself. A simulator or emulator cannot predict the behavior or replicate these features accurately.

#4: Certain Tests Are Best Suited for Real Devices

As Mobile Labs’ checklist reveals, most tests are best suited for testing on real devices as simulators and/or emulators cannot give accurate results. Examples of tests that are best suited to a real device fall into the following categories:

  • Tests requiring hardware features (touch sensors, GPS, camera, etc.)
  • Tests requiring different OS versions
    • Especially important for Android due to multiple device manufacturers
  • Functional tests
    • All regression tests should mimic real world conditions on real devices
  • User acceptance tests
    • Real devices enable teams to make sure that what is delivered is what is expected by the end-user
    • Testing colors of the app on real device screens with different brightness settings gives the best results

Even though testing on real devices can be daunting based on the volume of devices and different operating systems available, ultimately enterprise mobility teams need to provide the best and most innovative mobile experiences to users. Real device testing allows mobile developers and testers to produce high-quality app experiences that are required to keep up with digital transformation. It is an investment worth making for quality results.

Want to learn more about manual and automated testing on real devices? Mobile Labs’ deviceConnect™, available as both an on-premises and as a hosted mobile device cloud, provides device access and management for a team’s dedicated pool of real devices, regardless of geographical location for increased collaboration and DevOps. Learn more about deviceConnect by watching our video demo.

Steve Orlando

Steve Orlando is a seasoned development and quality assurance professional with experience testing and developing mobile, Web, mainframe, CRM and desktop applications. In his role as director of product marketing for Mobile Labs, Steve drives the design and implementation of the company’s private mobile device clouds, GigaFox™ and deviceConnect™.

More Posts | Website