+1 404-214-5804
Enterprise Mobility Blog, Mobile Testing Strategy and Techniques

Object-Based Mobile App Testing – What Do You Do When You Need Visual Based Reports and Verifications?

Last updated July 29, 2016.

Although Mobile Labs firmly believes in the benefits of object-based mobile testing, we recognize that smart phones and tablets are rich, graphics-based, visual tools.  Getting mobile object types and properties right (particularly, text without needing to use OCR) helps improve both productivity and accuracy when writing test cases.

Building a comprehensive set of object properties using a simple point-and-click operation is much quicker and more accurate than procedural-parameter-type operations where properties are hand-coded as arguments to function calls. The latter can make scripts difficult to refactor when properties change.

The UFT/QTP object repository makes it a snap to propagate property changes throughout a script when changes are needed.

With these considerable advantages to object-based scripting, one might wonder whether we see the need to verify that text and bitmaps are properly displayed, to wait for a screen to visually update, or to show, in a very visual way, what happened during a test case.

Our answer is an unequivocal ‘Yes!’

A number of Trust methods and QTP/UFT features are specifically designed to keep test cases visually on track.

Visual needs: verification, synchronization, and ‘what happened?’

Three main areas remain highly “visual” when writing mobility test cases:

  1. Verifying that screen content is displayed correctly
  2. Synchronizing the test case with the state of objects on the screen
  3. Gathering information needed to analyze the success or failure of a test case

Using Mobile Labs Trusttm certified add-in with HP’s QTP or UFT makes short work of such visual-based tasks.

In this post I will show examples using UFT 11.5, but the same techniques can just as easily be used in QTP 11.0.

Using a bitmap checkpoint to verify displayed graphics

There are a couple of issues that can crop up when displaying information on a mobile Web page or one of the screens of a mobile app.

The first issue is whether an image is displayed correctly. Is it the proper image?  Is it in the right position?  Is it completely displayed?

One easy way to verify graphic displays is a UFT bitmap checkpoint.

To test this capability, I selected an iPad Mini device from deviceConnecttm and connected to it using the Mobile Labs’ iOS7 demo app.  I wanted to determine whether the Mobile Labs’ logo that appears in the lower-left corner of the screen is displayed correctly.

A bitmap checkpoint is perfect for this kind of verification.

To create a bitmap checkpoint, I entered UFT’s “record” mode and selected “Create Checkpoint” from UFT’s “Design” menu. A drop-down let me further select “Bitmap Checkpoint” to create.

Telling UFT what image I want to interrogate is as simple as clicking on the bitmap and accepting the MobiImage offered by the Trust Agent (see below).


Once I clicked “OK,” the new checkpoint object was added to the repository along with the bitmap that represents the object (see below). I had the ability to make the match less precise with regard to colors and pixel dimensions, and after making those adjustments, the checkpoint statement was automatically added to my script.


When I ran the script, UFT’s run results viewer showed me the checkpoint step – the checkpoint succeeded, and the capture shows the bitmap we were looking for on the left and the bitmap that was found on the right (see below).  This test case is now ready for use in later regression testing.


Using bitmap checkpoints to validate text

It turns out that bitmap checkpoints are useful not just for graphic images; we can use them to validate the graphic appearance of text as well.  I created a bitmap checkpoint for an iOS7 button using the same point-and-click method.

Because the information (bitmap and properties) are provided by the Trust Agent, I easily got a bitmap that can be used at runtime to make sure the user sees what we expect them to see.

Here, from the run results viewer, is the step result of running a bitmap checkpoint on the button’s appearance:


Other ways to validate text

Text is a special category of information – although a bitmap or graphic image is a string of binary data, text begins with an ASCII string that is rendered by the device’s graphics engine as a property of some object (for example, a button). Where bitmap checkpoints require possible image-within-image matching, text is also susceptible to another class of visual verification:  optical character recognition (OCR). Trust supports a couple of ways to use OCR to verify that text expected on the screen is actually there.

I will examine two such means:  GetTextLocation and GetVisibleText.

It is well to emphasize that OCR is only needed for an extra, visual verification. Objects can be recognized and tested using their “text” property — as retrieved by the Agent – with complete accuracy. In fact, the “text” property can be an important part of validating the display because it represents what the app is trying to do.

Most objects do not need visual verification – but when you need it, the capability is at your disposal.

Validating text with GetTextLocation

The Mobile Labs’ Trust GetTextLocation method can be used on several of our Mobi objects.  In the example below, I used the method on the “MobiDevice” object that represents the entire device display.  Line 4 calls GetTextLocation to search the whole display for the requested text.

If the text is found, the script puts out a message box giving the rectangle where it was found. If the text is not found, the returned rectangle is all zeros, and the script displays the message “not found.”  The example below shows text that was found:


In the example below, I changed the text being sought from “Sign In” to “Log In.” Since the latter text is not on the screen, I expect the search to tell us that the text “Log In” cannot be found, and it does:


Validating text with GetVisibleText

While GetTextLocation returns a rectangle that tells me if and where it found text, I can use GetVisibleText on objects to return the text that the user can see.  This method runs an OCR scan on the object’s image just as GetTextLocation did, but GetVisibleText returns the OCR results.

Note that the information is not the same as the “text” property – that information, from the Agent, is the text that the app is trying to display.  GetVisibleText shows me the text the app succeeded in displaying. This may catch errors when a clipping region is not big enough to display all the text.

The screen capture below shows a short script that uses GetVisibleText to use OCR on the “Sign In” button. Once we have the decoded text, it is shown in a message box:


It is easy enough to compare the text returned (what’s on the screen) with the tester’s version of what should have been displayed (the ‘text’ property from our Agent).  The run results viewer shows us the object that was the subject of the test:


Easier maintenance: leveraging what the Agent knows

It is easy to use either bitmaps of text or OCR on text fields to verify that what the user sees is what we wanted her to see.  In the examples we have seen so far, we have hard coded the expected result, which is what we must always do if a tool only offers a surface-based way of obtaining text.

But the Trust Agent has one more trick up its sleeve.  What happens if the programmer changes the text of the button to “Log In?”  The script will call out an error until we manually change the compare parameters. If we want to verify that the device is displaying the text the programmer intended, regardless of its actual value, we can use the “GetROProperty” method to retrieve the intended text:

text2 = MobiDevice(“P…2”).MobiButton(“SignIn”).MobiElement(“SignIn”).GetROProperty(“text”)

With this statement executed, ‘text2’ contains what the programmer intended. I modified the MsgBox to show actual text and desired text by using GetVisibleText for the former and GetROProperty for the latter.  The result is shown below:


A complete verification would combine the two methods.  We could compare the agent’s “text” property with the known design and then use “GetVisibleText” to verify the implementation.  If the text ever changes, we will have far less refactoring if we use such an approach.

Thinking visually: waiting for the screen to appear

Mobile users often move to their next task when a device shows new information.  This is an inherently visual verification and is intuitive and natural for most people.

Trust and the Agent have a number of ways for a test case to react to the appearance of the screen just as a user does.

If we have initiated a sign-in that may be affected by Wi-Fi, WAN, and/or server performance, we may find it necessary to wait for the remote server to finish its work.  A user intuitively knows when the server has answered because the display changes.  Trust offers the “WaitProperty” method as a means of making the same visual verification.

In the script below, we paused at line 2 so that we do not try to log in to the app until its login screen is in front of us and ready to go. We then fill in the MobiEdit boxes that contain our userid and password, and we press the “Sign In” button on line 14.  We then pause the script again until we know that the sign-in process has completed successfully.

That pause is on line 16.

When the login has completed, we present a message box announcing the fact.


The secret to the synchronization is the “visible” property of Mobi objects.  The “visible” property turns from “False” to “True” when an object appears on screen.  Our test at line 2 looks for the Mobile Labs logo; our test on line 16 looks for the title of the first screen the app presents after sign in completes.

The screen capture in run results shows us, visually, what object satisfied the WaitProperty method:


Seeing the results: visual reports

Finally, we put all of these techniques to work in the short script above; we have seen how the data capture ability of UFT shows us each screen that was presented on the mobile device, how that screen looks at the moment the test reaches the step, and how UFT highlights the target object to show us the subject of the step.

We have also seen how several techniques can be used to verify the visual contents of the screen – we used UFT bitmap checkpoints to verify a logo and the appearance of the “Sign In” button, and we used GetVisibleText to compare the text showing on the screen to the text the app is attempting to display (retrievable with GetROProperty).

Visual reports are a part of the standard run results available from UFT.  We can request screen captures for each step in the test, which gives us the screen contents and shows us the targeted object(s), and we can request that UFT record a movie that shows the entire user interface (both UFT and the device under test).

These options are easily selected from the “Tools” menu, “Options” command, which presents the dialog below.  We move to the “GUI testing” tab.  If we want to see screen captures of every step in the test, we can set the parameters here. If we want to see a recording of the entire test, what UFT calls a “test movie,” we can also request that here.

When these options are set, the captured screens and the movie are conveniently stored with each set of run results.


The video link shows the playback of our test script as it was captured in the UFT video recorder.

Conclusion: thinking visually with object-based Trust

Although Mobile Labs is a leader in object-based support, enabling scripting productivity and accuracy with our Agent (an instrument that runs on the mobile device during automated test cases), we also know that the device is inherently visual and that both the test cases and their results need to accommodate the visual nature of mobile testing.

To that end, Trust offers image and text verification through a variety of methods to ensure that what is displayed on the device is what is expected. The scripting language ensures that a script can react when the device changes appearance in the same way a mobile user does, and finally, screen capture detail and even complete movies of test cases are routinely available to understand the success – or failure – of a test case using the actually-displayed visual appearance of the app as it ran.

We at Mobile Labs think mobile apps are best tested by capitalizing on the depth and accuracy that our Agent offers for classifying objects and supporting their natural methods, but we also know that a script sometimes has to deal with the actual appearance of things to ensure that the app is ready-to-go with the highest possible quality.

Want to learn more? Check out our latest posts!

Don’t forget to download our eBook on Amazon, to stay ahead of the curve in 2017!

Michael Ryan

Michael Ryan serves as Mobile Labs’ chief technology officer. In this role, Ryan provides the technological vision and drives Mobile Labs Trust’s product road map. Ryan has more than 35 years of experience in leading software development teams that design and build robust and market-leading solutions for large-scale enterprise customers among Fortune 1000 companies. Most recently, Ryan was with Fundamental Software where he worked on large-scale systems CPU emulation architecture, design, and implementation. Prior to Fundamental Software, Ryan was director of development, Sr. VP of R&D, and finally, Chief Technical Officer for CASE tool vendor KnowledgeWare, Inc. Ryan served as senior staff systems engineer, field manager, and regional technical support manager for mainframe manufacturer Amdahl Corporation.

More Posts - Website

Leave a Reply

You must be logged in to post a comment.

Why Mobile Labs?

Mobile Labs provides enterprise-grade, next generation mobile application testing tools. With a focus on security, agility and affordability, Mobile Labs delivers solutions to help you deliver quality mobile apps for Android, iOS and Windows platforms while also helping manage mobile devices in a private, secure cloud.

Contact Mobile Labs

3423 Piedmont Road NE
Suite 465
Atlanta, GA 30305
+1 404-214-5804
twitter  facebook linkedin google-plus SlideShare