Although leveraging Appium for mobile test automation gives enterprise mobility teams more freedom, it does not provide test run results. Here’s how to get these results by leveraging tools like Maven and Allure.

Based on the amount of buzz from industry thought leaders it seems that Appium has taken the industry by storm. For mobile testers and quality assurance teams who are already leveraging Selenium, it only seems natural to explore its sister tool Appium for mobile app and mobile web test automation.

But, while enterprise mobility professionals commonly praise Appium for the freedom it allows in writing and running test scripts, leveraging Appium is not without its unique challenges. While many vendors offer help getting up and running with Appium by including a built-in Appium server as part of their product offerings, once it is time to write and run tests, things can get tricky without the right support in place.

Ultimately, when it comes to using Appium for test automation, teams have two choices for getting the reports they need – they can rely on commercial reporting solutions or build their own solution in-house.

Commercial Solutions for Testing with Reporting Built-in

For teams nervous to implement their own frameworks and solutions with an open source tool like Appium, testers and quality assurance teams can explore commercial solutions for testing and reporting, such as Tricentis Tosca and MicroFocus’ UFT. With commercial solutions, these teams can skip the time-consuming tasks of building a testing and reporting framework. It is important to note that while Tricentis Tosca is Appium-based, UFT is not.

Both Tosca and UFT provide test results that can be tracked back to the team’s test management tools used for reporting. Because both tools are part of the Tricentis and MicroFocus ecosystems, each solution includes dedicated resources and support from each vendor for customers leveraging these reporting tools. This level of support can be beneficial to teams that cannot devote the time and resources required to build a solution for adding test run results.

Teams leveraging commercial solutions need to take into consideration the resources and available skill set of their testers and quality assurance teams when choosing the correct strategy/methodology for incorporating test run results into Appium.

But, some teams may choose to go a different route and decide to build their own reporting solution. Here are some important things to consider when building a reporting module in-house and how to incorporate reporting into Appium tests.

Building and Adding Reporting to Appium Tests

Naturally, writing a separate reporting tool can be cumbersome, time consuming and may strain limited resources, but for teams who want the freedom and the ability to customize, this can be a viable solution to leverage alongside Appium tests.

It is important to note that mobile testers and quality assurance teams that choose to build their own reporting tool will be juggling the build and maintenance of the tool, which will take time away from writing and maintaining Appium test code.

For teams reading to move forward, here are a few things to consider:

#1: Use Build Process Tool Reports

Teams may decide to use the reports that come from their build process tool, such as the Surefire plugin for Apache Maven for Java. With a robust open-source community supporting multiple projects, teams that choose to leverage Maven can use this tool to generate unit test results and coverage. But, it is important to note that the reports provided by Maven can be basic and may not provide all the required results without customization.


If this is the case, then there are other tools available that can track test results back to Appium, such as Allure.

#2: Track Test Results with Allure

What is the allure of Allure? As an open-source testing tool it gives a lot of freedom for testers and quality assurance to build a solution that provides test run results from Appium tests. By using Allure, teams can create a nice, viewable detailed report. Allure alleviates the need to continually maintain reporting code and gives a lot of features that can help testers tie back tests to bug and story tracking systems. It is also very easy to implement and works out of the box with very little customization.

Allure also works with many of the same languages that Appium supports, such as Ruby, Python and Java, fitting into the desired test framework of the team. It also supports Jenkins, which makes the test results and reports centrally available for the build system. Since the build system tends to be the centralized system for all builds and automated tests, it is only natural to have the reports available and visible to all of the teams involved in all aspects of the builds.

For teams interested in exploring Maven with Allure, a full project is currently available from Mobile Labs’ GitHub page.

And for teams that are interested in adding Allure to Appium test scripts to track results, here are the steps to follow.

Adding Results to Appium Test Scripts With Allure

Step #1: Add Allure to Maven Project

Scroll the below code in its own text box. Keep it a fixed width font.

POM.XML

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.mobilelabsinc.appium.example-appium-testng-native-app</groupId>
    <artifactId>example-appium-testng-native-app</artifactId>
    <version>1.0-SNAPSHOT</version>
    <name>Appium TestNG Native App Example</name>
    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <selenium.version>3.6.0</selenium.version>
        <appium.version>5.0.3</appium.version>
        <mavensurefire.version>2.19</mavensurefire.version>
        <phantomjs.version>1.3.0</phantomjs.version>
        <mavencompiler.version>3.6.1</mavencompiler.version>
        <mavensurefirereport.version>2.19</mavensurefirereport.version>
        <allure.version>1.5.2</allure.version>
        <aspectj.version>1.5.4</aspectj.version>
        <alluremaven.version>2.5</alluremaven.version>
        <junit.version>4.12</junit.version>
        <testng.version>6.14.2</testng.version>
        <alluretestng.version>2.6.0</alluretestng.version>
        <slf4j.version>1.6.1</slf4j.version>
    </properties>
    <dependencies>
        <dependency>
            <groupId>org.testng</groupId>
            <artifactId>testng</artifactId>
            <version>${testng.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>${junit.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.seleniumhq.selenium</groupId>
            <artifactId>selenium-java</artifactId>
            <version>${selenium.version}</version>
        </dependency>
        <dependency>
            <groupId>io.appium</groupId>
            <artifactId>java-client</artifactId>
            <version>${appium.version}</version>
        </dependency>
        <dependency>
            <groupId>com.codeborne</groupId>
            <artifactId>phantomjsdriver</artifactId>
            <version>${phantomjs.version}</version>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-api</artifactId>
            <version>${slf4j.version}</version>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-simple</artifactId>
            <version>${slf4j.version}</version>
        </dependency>
        <dependency>
            <groupId>io.qameta.allure</groupId>
            <artifactId>allure-testng</artifactId>
            <version>${alluretestng.version}</version>
            <scope>test</scope>
        </dependency>
    </dependencies>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.7.0</version>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>${mavensurefirereport.version}</version>
                <configuration>
                    <suiteXmlFiles>
                            <suiteXmlFile>src/test/resources/testng.xml</suiteXmlFile>
                        </suiteXmlFiles>
                    <testFailureIgnore>true</testFailureIgnore>
                    <argLine>
                        -javaagent:${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar
                    </argLine>
                    <systemProperties>
                        <property>
                            <name>allure.results.directory</name>
                            <value>${project.build.directory}/allure-results</value>
                        </property>
                    </systemProperties>
                </configuration>
                <dependencies>
                    <dependency>
                        <groupId>aspectj</groupId>
                        <artifactId>aspectjweaver</artifactId>
                        <version>${aspectj.version}</version>
                    </dependency>
                </dependencies>
            </plugin>
            <plugin>
                <groupId>io.qameta.allure</groupId>
                <artifactId>allure-maven</artifactId>
                <version>2.8</version>
            </plugin>
        </plugins>
    </build>
    <reporting>
        <excludeDefaults>true</excludeDefaults>
        <plugins>
            <plugin>
                <groupId>io.qameta.allure</groupId>
                <artifactId>allure-maven</artifactId>
                <version>2.8</version>
            </plugin>
        </plugins>
    </reporting>
</project>

Step #2: Add Screenshot Attachments Capabilities

To add screenshots attachments, add the following method to the TestNG or JUnit test:

@Test
@Feature("Login")
@Story("Valid Login")
@Description("Verifies that the Logout Button appears on the Search Screen after entering the username and password and then clicking the Sign In button on the login screen")
public void ScreenshotTest() {
  WebElement userField = driver.findElement(By.id("UsernameTextField"));
  userField.sendKeys("user");
  WebElement passField = driver.findElement(By.id("PasswordTextField"));
  passField.sendKeys("pass");
  WebElement signinButton = driver.findElement(By.id("SignInButton"));
  signinButton.click();
  WebElement logoutButton = driver.findElement(By.id("LogoutButton"));
  Assert.assertTrue(logoutButton.isDisplayed());
  getScreenshot("Logout button page");    
}

Step #4: Run the Test

Use the following Maven Commands:

test site

Note: Historical results are saved and will be present in the test report. To remove past runs, use the Maven “clean” command.

Once the test is completed, open the report in the Target folder.

Step #5: View the Test Run Results

Once the test is complete, a nice HTML report is generated that shows:

  • past test run results (if they are still available)
  • the timeline it took to run tests
  • pass/fail of the tests including the stack trace of the failures

To learn more about Allure, the following resources may be useful:

Ready to jump start your mobile testing lab? Check out our white paper to learn more about the strategies and characteristics of the most successful enterprise mobility testing labs and how your organization can become one.

Steve Orlando

Steve Orlando is a seasoned development and quality assurance professional with experience testing and developing mobile, Web, mainframe, CRM and desktop applications. In his role as director of product development for Mobile Labs, Steve drives the design and implementation of the company’s private mobile device cloud, deviceConnect™ as well as its automated mobile app testing solution, Mobile Labs Trust™.

More Posts | Website