Testing native Mobile Apps with TestComplete and Appium – An Alternative

[Note] This post was written well before SmartBear released TestComplete 10 with support for testing iOS and Android mobile apps. Please visit the SmartBear TestComplete product page for more details. This post is no longer valid and kept here as a reference.

My day job needs me to work on TestComplete a popular commercial testing tool from SmartBear Software. We do a lot of serious testing using TestComplete for our Desktop applications.

Recently I saw discussions on TestComplete User Forums about possibility of testing mobile apps using TestComplete.  While TestComplete supports testing on wide range of technologies and application platforms including Web and Desktop applications developed on Windows, .NET, WPF, Delphi, C++, Java and so on. However while writing this post it does not have in-built mobile testing capabilities. SmartBear is working on adding mobile testing features which are in beta phase. There are third party tool/plug-in(s) available which use Image Recognition based approach to test the applications. But there is no native support to test iOS or Android Apps.

With these thoughts in my mind, I started to explore if I could mix Appium with TestComplete which will allow testing mobile Apps without leaving the TestComplete environment? With couple hours of hacking, here are the results!

About Appium

Appium is an open source test automation framework for automating native and hybrid mobile apps.
It drives iOS and Android Apps using the WebDriver JSON wire protocol. Appium tests are built on Selenium WebDriver API.  The WebDriver API is used for driving user actions in Browsers, but now it can be used to drive the user interaction on native and hybrid mobile Apps. WebDriver API provides various language bindings which allow you to use them with your choice of programming language and platform, very cool!

How does this works?

TestComplete is a versatile tool and provides supports for more than dozen technologies and extensibility features. TestComplete provides a .NET Bridge which allows access to individual objects of .NET assemblies, their internal properties and methods. This is done by a special dotNET object that provides access to .NET assemblies, types and type members. This feature is available as part of .NET Classes Support plug-in in TestComplete.

Since Appium is driven by WebDriver API, we can connect the WebDriver .NET APIs with TestComplete using the .NET bridge and drive the mobile Apps using Appium. Following diagram provides an overview of this interaction

How to do it?

Setting up the WebDriver .NET API with TestComplete is relatively easy task. Following section describe the detailed steps for driving Appium through TestComplete for testing a sample iOS native App.

Step 1 – Download the WebDriver .NET API Binaries

Download the latest binaries from http://docs.seleniumhq.org/download/. You need to click on Download link in front of C# option in Selenium Client & WebDriver Language Bindings section.

Step 2 – Add WebDriver assemblies to TestComplete CLR Bridge

In next step make the WebDriver assemblies available to TestComplete with following steps. These assemblies will be added to the list in the project’s CLR Bridge settings group. This group contains the list of .NET assemblies whose functions will be available to your scripts through the dotNET object:

  1. Go to TestComplete Project Setting by selecting Current Project Settings from Tools Menu
  2. Select CLR Bridge option on left side of Project Properties Pane
  3. Click Browse Files button to select the WebDriver files. This will display Select Assembly dialog box
  4. On Select Assembly dialog navigate to the net40 folder in WebDriver .NET API files downloaded in Step 1 and select the WebDriver.dll and WebDriver.Support.dll files
  5. Also make sure System.dll is added to CLR Bridge. If it’s not added already, you can add this assembly by Browser GAC button. You will see screen similar to following screen shot after you add all the assemblies to the CLR Bridge
TestComplete CLR Bridge

TestComplete CLR Bridge

This will add WebDriver API to the TestComplete project and provide access to the internal classes and functions using the dotNET object. Assemblies that are added to the CLR Bridge become available in scripts as child objects of the dotNET object. Types defined in these assemblies are available as child of dotNET object. All available assemblies, types, and type members are displayed in the Code Completion window of TestComplete. For more information on dotNET object refer Calling .NET Routines via dotNET Object section in TestComplete Help documentation (also available at http://support.smartbear.com/viewarticle/28697/)

Step 3 – Setting up Appium

Before we start testing iOS or Android apps, we need to setup Appium in a local environment. If you wish to run tests for iOS, then you need to setup Appium on a Mac OSX machine. For testing Android applications you can setup environment on a Windows or Linux machine. Setting up Appium is fairly easy with new Appium App for Mac OSX.

In this example we will test a sample BMI Calculator application for iOS. This is a pretty simple App with bunch of UI elements on a single view. You can get the App source from https://github.com/upgundecha/BmiCalculator.

Let’s setup Appium on a Mac and TestComplete on a Windows machine

  1. On Mac machine download and install the Appium from http://appium.io
  2. Before we launch Appium, download the sample App from https://github.com/upgundecha/BmiCalculator/blob/master/Sample%20App/BmiCalculator.zip,
  3. Unzip BmiCalculator.zip file to get the BmiCalculator.app file
  4. Next, launch Appium on Mac. Select the App Path checkbox and choose the location of the App. Click on Launch button, Appium will launch the application in iOS Simulator as shown in below screenshot, now it’s ready for testing the App
Appium.app running on Mac

Appium.app running on Mac

Step 4 – Writing a test

TestComplete offers multiple language options to write your tests. You can write tests in JavaScript, VBScript, Delphi or C++/C# scripting languages. This example uses JavaScript.

Our test need to tell Appium about what platform is needed for executing the tests. This is done by creating an instance of DesiredCapabilities class of WebDriver API and setting up configuration information in key/value  pair.

//Set up the capabilities to run test on iOS application on local Appium server
Capabilities.setCapability(CapabilityType.BrowserName, "iOS");
Capabilities.setCapability(CapabilityType.Version, "6.1");
Capabilities.setCapability(CapabilityType.Platform, "Mac");

Create an instance of  RemoteWebDriver instance and point it to http://mymacmachine:4723/wd/hub in the test by using Uri class from .NET System namespace and desired capabilities defined earlier. Here mymacmachine could be different in your environment. You can specify Hostname or IP of the Mac machine where Appium is running.

//Set up the RemoteWebDriver
var uri = dotNET.System.Uri.zctor("http://mymacmachine:4723/wd/hub");
var driver = dotNET.OpenQA_Selenium_Remote.RemoteWebDriver.zctor_3(uri,Capabilities);

Here is complete script which tests the BMI calculator functionality (also available as a Gist @ https://gist.github.com/upgundecha/5894126)

function BmiCalculatorTest()
    	//Create an instance of DesirecCapabilities, CapabilityType and By Class using CLR Bridge & dotNET Object
    	var Capabilities = dotNET.OpenQA_Selenium_Remote.DesiredCapabilities.zctor();
    	var CapabilityType = dotNET.OpenQA_Selenium_Remote.CapabilityType;
    	var By = dotNET.OpenQA_Selenium.By;

    	//Set up the capabilities to run test on iOS application on local Appium server
    	Capabilities.setCapability(CapabilityType.BrowserName, "iOS");
    	Capabilities.setCapability(CapabilityType.Version, "6.1");
    	Capabilities.setCapability(CapabilityType.Platform, "Mac");

    	//Set up the RemoteWebDriver
    	var uri = dotNET.System.Uri.zctor("http://mymacmachine:4723/wd/hub");

    	var driver = dotNET.OpenQA_Selenium_Remote.RemoteWebDriver.zctor_3(uri,Capabilities);

    	//Get the Hight text field and enter value
    	var HeightField = driver.FindElement(By.Name("Height"));

    	//Get the Weight text field and enter value
    	var WeightField = driver.FindElement(By.Name("Weight"));

    	//Get the Calculate button and tap/click on it
    	var CalculateButton = driver.FindElement(By.Name("Calculate"));


    	//Get the Bmi lables
    	var BmiLabel = driver.FindElement(By.Name("bmi"));
    	var BmiCategoryLabel = driver.FindElement(By.Name("category"));

    	//Check labels for expected values
    	if(aqObject.CompareProperty(BmiLabel.Text, cmpEqual, "24.42")) {
      	  Log.Checkpoint ("Calculated BMI is correct!");
    	else {
      	  Log.Error( "Calculated BMI does not match expected value");

    	if(aqObject.CompareProperty(BmiCategoryLabel.Text, cmpEqual, "Normal")) {
      	  Log.Checkpoint( "BMI Category is correct!");
    	else {
      	  Log.Error( "BMI Category does not match with expected value");

If you prefer TestComplete with VBScript, you can get the VBScript version @ https://gist.github.com/upgundecha/5894141

How this works?

When this test is executed, TestComplete calls the WebDriver API and connects to the Appium server using the RemoteWebDriver. Then it locates the UI elements using FindElement() method of WebDriver API. Test can locate the elements using various locator strategies provided by the By class of WebDriver API. In following example, test is locating Height text field from Bmi Calculator App using its name property. It then enters the specified text using SendKeys() method

//Get the Hight text field and enter value
var HeightField = driver.FindElement(By.Name("Height"));

This is pretty much similar to how TestComplete works with UI objects while testing Web or Desktop applications. After inputting the data, TestComplete verifies the expected outcome using its inbuilt verification methods. At the end of execution TestComplete generate a result log as shown in following screenshot

Test Results

Test Results

Running test with Sauce Labs

You may not always get a dedicated hardware to test these Apps. Sauce labs, the company behind Appium, provides you an ability to run Appium tests in a virtual environment without needing you to setup everything from scratch. This also saves you costs for setting up the infrastructure on your own.

So let’s setup and run this test with Sauce Labs. You need a free Sauce Labs account to begin with. Register for a free account on Sauce Labs and get the user name and access key. Sauce Labs provides all the needed hardware and software infrastructure to run your tests in the cloud.

For running tests in Sauce Labs environment, you need to upload your App to Sauce Labs temporary storage using Temporary Storage REST API. You can get the App compiled and built by developers and upload through curl command line utility (Windows users can get curl from http://curl.haxx.se/download.html)

curl -u <sauce_user_name>:<sauce_access_key> -X POST "http://saucelabs.com/rest/v1/storage/$SAUCE_USERNAME/my_ios_app.zip?overwrite=true" -H "Content-Type: application/octet-stream" --data-binary @my_ios_app.zip

We need to modify our test a bit to run on Sauce Labs. Apart from changing the Capabilities and RemoteWebDriver Uri, we need to pass Sauce Labs user name and access key. You can save these into TestComplete project variables. With following changes to earlier test, we can now run tests with Sauce Labs:

var USER_NAME = Project.Variables.SAUCE_USER_NAME
var ACCESS_KEY = Project.Variables.SAUCE_ACCESS_KEY

//Set up the capabilities to run test on iOS application on Sauce Labs
Capabilities.SetCapability(CapabilityType.BrowserName, "iOS 6.0");
Capabilities.SetCapability(CapabilityType.Platform, "Mac 10.8");
Capabilities.SetCapability("device", "iPhone Simulator");
Capabilities.SetCapability("app", "sauce-storage:my_ios_app.zip");

//Set up the RemoteWebDriver
var uri = dotNET.System.Uri.zctor("http://" + USER_NAME + ":" + ACCESS_KEY +  "@ondemand.saucelabs.com:80/wd/hub");
var driver = dotNET.OpenQA_Selenium_Remote.RemoteWebDriver.zctor_3(uri,Capabilities);

When you run the test you can login to your Sauce Labs account and see the progress in a Browser window. Based on desired capabilities, Appium provisions an instance of Mac in Sauce Labs cloud and drives the App.

Test Running on Sauce

Test Running on Sauce

You can also view test run in Sauce Labs dashboard with details of each step along with screenshot and video of entire run.

Sauce Dashboard

Sauce Dashboard


Appium bring exciting possibilities to TestComplete users for testing mobile applications in a platform independent way. These tests can be executed in a local environment or in the cloud or real devices in very a short time, with minimal efforts and costs. This brings more test coverage and faster feedback.

Using Google Charts API for Test Results Dashboard

[This post was originally written in May 2008]

Image Charts

I am using Google Charts API to generate charts to visualize test results in Dashboard. Image charts are generated on the fly based on dynamic data and then embedded in to test results generated in HTML format. This feature can be used with any tool/framework where test results are generated in HTML format.  This does not require any software/dependency installed on user’s desktop and comes free from Google.

There are different types of charts you can generate with variety of options. Just hit http://code.google.com/apis/chart/ for API documentation.

Here is an example of Pie and Google-O-Meter types:

<a href="http://unmesh.files.wordpress.com/2012/05/google-charts2.png"><img class="aligncenter size-medium wp-image-333" title="Google Charts" src="http://unmesh.files.wordpress.com/2012/05/google-charts2.png?w=300" alt="" width="300" height="119" /></a>

You can try Pie chart example by simply copying following HTTP request in your Browser’s address bar. A chart will appear in your Browser window once you hit Enter key.

<a href="http://chart.apis.google.com/chart?chtt=Test+Results&amp;chts=000000,12&amp;chs=450x150&amp;chf=bg,s,ffffff&amp;cht=p3&amp;chd=t:66.66,33.33&amp;chl=Tests+Passed(2)|Tests+Failed(1)&amp;chco=006600,990000">http://chart.apis.google.com/chart?chtt=Test+Results&amp;chts=000000,12&amp;chs=450x150&amp;chf=bg,s,ffffff&amp;cht=p3&amp;chd=t:66.66,33.33&amp;chl=Tests+Passed(2)|Tests+Failed(1)&amp;chco=006600,990000</a>

Now, how do we implement this in QTP/VBScript or any other tool of your choice? Here is an example of VBScript function which accepts Number of Test Executed, Passed, and Failed and then generates a nicely formatted HTML report:

Public Function GenerateTestSummaryReport(ByVal intTestsExecuted, ByVal intTestsPassed, ByVal intTestsFailed)

	Set objFSO = CreateObject("Scripting.FileSystemObject")
	Set objFile = objFSO.CreateTextFile("C:\TestSummary.html",True)

	objFile.WriteLine "<!DOCTYPE html PUBLIC '-//W3C//DTD XHTML 1.0 Transitional//EN' 'http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd'>"
	objFile.WriteLine "<html xmlns='http://www.w3.org/1999/xhtml' >"
	objFile.WriteLine "<head><title>Test Execution Summary Report</title></head>"
	objFile.WriteLine "<body style='font-weight: bold; font-size: 10pt; color: black; font-family: Tahoma'>"
	objFile.WriteLine "<br />Test Execution Summary"
	objFile.WriteLine "<br /><br />Test Suite Summary<table><tr><td style='width: 200px; height: 56px; background-color: #669918'>Total Tests Executed</td><td style='height: 56px; background-color: #669918'>Passed</td><td style='height: 56px; background-color: #669918'>Failed</td></tr>"
	objFile.WriteLine "<tr><td style='width: 200px; background-color: #D0F0A2'>" & intTestsExecuted & "</td><td style='background-color: #D0F0A2'>" & intTestsPassed & "</td><td style='background-color: #D0F0A2'>" & intTestsFailed & "</td></tr>"
	objFile.WriteLine "<tr><td><img src='http://chart.apis.google.com/chart?chtt=Test+Results&amp;chts=000000,12&amp;chs=450x150&amp;chf=bg,s,ffffff&amp;cht=p3&amp;chd=t:" & intTestsPassed & "," & intTestsFailed & "&amp;chl=Tests+Passed(" & intTestsPassed &")|Tests+Failed(" & intTestsFailed & ")&amp;chco=006600,990000' alt='Test Results Chart'/></td></tr>"

	Set objFile = Nothing
	Set objFSO = Nothing

End Function

'Call Function
Call GenerateTestSummaryReport(5,3,2)

In above code, highlighted IMG tag will make a request to Google Chart API and will show a nicely built chart on the fly when user opens this file in a Web Browser or email client.

If you find API documentation not working for you, there is a very interesting service available at http://dexautomation.com/googlechartgenerator.php

This will enable you to design Charts using guided steps and then it will generate IMG tag which you can embed in your code. Using Google Charting API you could build a nice dashboard for you test suite result reporting.

However Image charts have some limitation on amount of data we can pass. Other alternative is Interactive Chart feature.

Interactive Charts

Unlike image charts which are static, Interactive are charts more advanced. Interactive charts support user interaction and event handling. These are built using JavaScript.

Google Visualization API provides different types of charts for data visualization. Please see following page for sample charts.


Here is an example in VBScript on how to use Gauge charts for showing test results/build status:

Public Function GenerateTestSummaryReport(ByVal intTestsPassed, ByVal intTestsFailed)
	Set objFSO = CreateObject("Scripting.FileSystemObject")
	Set objFile = objFSO.CreateTextFile("C:\TestSummary.html",True)
	objFile.WriteLine "<html><head><title> Test Execution Summary Report </title>"
	objFile.WriteLine "<script type='text/javascript' src='http://www.google.com/jsapi'></script>"
	objFile.WriteLine "<script type='text/javascript'>"
	objFile.WriteLine "google.load('visualization', '1', {packages:['gauge']});"
	objFile.WriteLine "google.setOnLoadCallback(drawChart);"
	objFile.WriteLine "function drawChart() { var passedData = new google.visualization.DataTable();"
	objFile.WriteLine "passedData.addColumn('string', 'Label'); passedData.addColumn('number', 'Value');"
	objFile.WriteLine "passedData.addRows(1); passedData.setValue(0, 0, 'Tests Passed');"
	objFile.WriteLine "passedData.setValue(0, 1, " & intTestsPassed & ");"
	objFile.WriteLine "var passedChart = new google.visualization.Gauge(document.getElementById('chart_passed'));"
	objFile.WriteLine "var passedOptions = {width: 400, height: 120, redFrom: 0, redTo: 30, yellowFrom: 31, yellowTo: 60, greenFrom: 61, greenTo: 100, minorTicks: 5};"
	objFile.WriteLine "passedChart.draw(passedData, passedOptions);"
	objFile.WriteLine "var failedData = new google.visualization.DataTable();"
	objFile.WriteLine "failedData.addColumn('string', 'Label');"
	objFile.WriteLine "failedData.addColumn('number', 'Value');"
	objFile.WriteLine "failedData.addRows(1);"
	objFile.WriteLine "failedData.setValue(0, 0, 'Tests Failed');"
	objFile.WriteLine "failedData.setValue(0, 1, " & intTestsFailed & ");"
	objFile.WriteLine "var failedChart = new google.visualization.Gauge(document.getElementById('chart_failed'));"
	objFile.WriteLine "var failedOptions = {width: 400, height: 120, redFrom: 31, redTo: 100, yellowFrom: 11, yellowTo: 30, greenFrom: 0, greenTo: 10, minorTicks: 5};"
	objFile.WriteLine "failedChart.draw(failedData, failedOptions);}"
	objFile.WriteLine "</script></head>"
	objFile.WriteLine "<body style='font-weight: bold; font-size: 10pt; color: black; font-family: Tahoma'><br />Test Execution Summary"
	objFile.WriteLine "<br /><br />Test Suite Summary<table><tr><td style='width: 200px; height: 56px; background-color: #669918'>Total Tests Executed</td><td style='height: 56px; background-color: #669918'>Passed</td><td style='height: 56px; background-color: #669918'>Failed</td></tr>"
	objFile.WriteLine "<tr><td style='width: 200px; background-color: #D0F0A2'>100</td><td style='background-color: #D0F0A2'>85</td><td style='background-color: #D0F0A2'>15</td></tr>"
	objFile.WriteLine "<tr><td><div id='chart_passed'></div></td><td><div id='chart_failed'></div></td></tr></table></body></html>"
	Set objFile = Nothing
	Set objFSO = Nothing
End Function

'Call Function
Call GenerateTestSummaryReport(15,85)

Here is interactive Chart Generated:

You can add more metrics to your reports and build more meaningful Charts.

TestComplete Running on 64Bit?

Here is a Quick Tip to find out if TestComplete is running on 64 Bit or 32 Bit OS Platform.

If dotNET.System.Environment.Is64BitOperatingSystem = True Then
End If

You can access the .NET Framework System.Environment class’s property Is64BitOperatingSystem to find out if the OS Platform is 64 Bit or 32 Bit using the TestComplete dotNET object. The dotNET object allows access to .NET Framework classes as well as custom assemblies through TestComplete.

Recording Screencast of TestComplete Test Runs

In this post we’ll explore the possibility of recording a screencast of TestComplete test runs, the other test tool I use extensively at work.

TestComplete does not have features to record screencast of test execution. However we can use Microsoft Expression Encoder 4 SDK to record videos programmatically from test scripts.

A recoded screencast of test run can help you in investigation of unattended execution or even demo the software or create training videos.

Microsoft Expression Encoder 4 has basic recording features with limited codec support. You can download it free from http://www.microsoft.com/expression/products/encoder4_overview.aspx

While the commercial version Microsoft Expression Encoder 4 Pro provides number of additional features and codecs.

I will use the CLR Bridge & dotNET object available in TestComplete to call the Encoder SDK functions to create the screencast.

Using CLR Bridge in TestComplete we can use .NET assemblies in test scripts and extend the test script code to use features from standard .NET classes as well as custom libraries.

Setting up Encoder SDK in CLR Bridge

Let’s add the Encoder SDK library to the CLR Bridge in TestComplete Project Settings by following steps:

1. Go to TestComplete Project Setting by selecting Current Project Settings from Tools Menu
2. Select CLR Bridge option on left side of Project Properties Pane
3. Click Browse Files button to select the Encoder SDK DLL file. This will display Select Assembly dialog box
4. On Select Assembly dialog navigate to C:\Program Files\Microsoft Expression\Encoder 4\SDK folder and select the Microsoft.Expression.Encoder.dll file

This will add Encoder library to the TestComplete Project and provide access to the internal classes and functions using the dotNET object.

Using ScreenCaptureJob Class in TestComplete Script

Let’s create a instance of ScreenCaptureJob class from the Encoder SDK and call methods to start & stop video recording:

Sub NotePadEditCutTest()
	Dim wndNotepad, objScreenCaptureJob

	'Create a new ScreenCapture Job using it's Constructor
	Set objScreenCaptureJob = dotNET.Microsoft_Expression_Encoder_ScreenCapture.ScreenCaptureJob.zctor

	'Specify the path & name of the Output File
	objScreenCaptureJob.OutputScreenCaptureFileName = "C:\Data\NotePadEditCutTest.wmv"

	'Start the Video Recording using ScreenCaptureJob's Start Method
	Call objScreenCaptureJob.Start()

	'Test Code
	Call TestedApps.notepad.Run(1, True)
	Set wndNotepad = Aliases.notepad.wndNotepad
	Call wndNotepad.Edit.Keys("SOme Text")
	Call wndNotepad.MainMenu.Click("Edit|Select All")
	Call wndNotepad.MainMenu.Click("Edit|Cut")
	Call aqObject.CheckProperty(Aliases.notepad.wndNotepad.Edit, "wText", cmpEqual, "")

	'Stop the Video Recording using ScreenCaptureJob's Stop Method
	Call objScreenCaptureJob.Stop()
End Sub

The output screencast will be created in wmv format. This screencast can be edited in Expression Encoder UI.


Get every new post delivered to your Inbox.

Join 47 other followers