A blog on Test Automation, Technical QA, DevOps, Testing Tools & Techniques and more. This is a personal blog. The content and opinions expressed in this blog are solely my own and do not express the views or opinions of my current or past employers.
TestComplete is a popular functional testing tool from SmartBear with support for testing Desktop, Web and Mobile Application. It supports more than dozen technology platforms, tools and number of third party controls.
SpecFlow.NET is popular BDD framework available on .NET platform. It integrates with Visual Studio. You can write Feature in 100% Gherkin with SpecFlow and automation code in C#.
In this video I’ll show you how to integrate SpecFlow.NET and TestComplete in Visual Studio for implementing BDD/ Automated Acceptance tests
This series was originally posted on Sauce Labs Blog. This is a reblog with revisions.
Here is last installment of Automated Acceptance Testing iOS Mobile Apps with Appium, Cucumber-JVM, Jenkins, and Sauce Labs series, which takes the mobile tests running on a local environment and running them on Sauce with Appium. If you missed the earlier parts from this series here are the links
So far we executed features in a local environment. Now this is just one feature, there could be 100s of such and you might want to run these features frequently. Setting up dedicated machine may not be always possible and involves costs. Sauce labs, the company behind Appium provides you an ability to run Appium tests in a virtual environment without needing you to setup everything from scratch. This also saves you costs for setting up your own infrastructure.
But at times you may also want to run these on a local environment, how do we support both running in local environment and sauce?
Let’s use Tags and Hook feature of Cucumber. We already have a setup method which connects to a local Appium sever using the RemoteWebDriver. Let’s add a tag to this method
@Before("local")
public void setUpLocal() throws Throwable {
...
}
And add another setup method setUpiOS() and add “ios-sauce” tag to it
@Before("@ios-sauce")
public void setUpiOS() throws Throwable
{
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(CapabilityType.VERSION, "6.1");
capabilities.setCapability(CapabilityType.PLATFORM, "Mac");
capabilities.setCapability("app", "sauce-storage:my_ios_app.zip");
capabilities.setCapability("device", "iPhone Simulator");
String sauceUserName = System.getenv("SAUCE_USER_NAME");
String sauceAccessKey = System.getenv("SAUCE_ACCESS_KEY");
//Create an instance of RemoteWebDriver and connect to the Appium server.
//Appium will launch the BmiCalc App in iPhone Simulator using the configurations specified in Desired Capabilities
driver = new RemoteWebDriver(new URL(MessageFormat.format("http://{0}:{1}@ondemand.saucelabs.com:80/wd/hub", sauceUserName, sauceAccessKey)), capabilities);
}
This allows selectively run features on desired environment. For example if you wish to run scenario on Sauce Labs, then you need to specify the “ios-sauce” tag on the scenario definition in feature file
@ios-sauce
Scenario Outline: Calculate Body Mass Index
…
This will run the feature on Sauce Labs environment. If you specify “local” then it will run the feature on local Appium setup. Here is complete code example
package bmicalculator.test;
import java.net.URL;
import java.text.MessageFormat;
import cucumber.annotation.*;
import cucumber.annotation.en.*;
import static org.junit.Assert.assertEquals;
import org.openqa.selenium.remote.CapabilityType;
import org.openqa.selenium.remote.DesiredCapabilities;
import org.openqa.selenium.remote.RemoteWebDriver;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.By;
public class Bmi_Calculator_Step_Defs {
private WebDriver driver;
@Before("@local")
public void setUpLocal() throws Throwable
{
//Setup for running Appium test in local environment
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(CapabilityType.BROWSER_NAME, "iOS");
capabilities.setCapability(CapabilityType.VERSION, "6.1");
capabilities.setCapability(CapabilityType.PLATFORM, "Mac");
capabilities.setCapability("app", "/Users/upgundecha/Desktop/AppExamples/BmiCalculator/build/Release-iphonesimulator/BmiCalculator.app");
//Create an instance of RemoteWebDriver and connect to the Appium server.
//Appium will launch the BmiCalc App in iPhone Simulator using the configurations specified in Desired Capabilities
driver = new RemoteWebDriver(new URL("http://0.0.0.0:4723/wd/hub"), capabilities);
}
@Before("@ios-sauce")
public void setUpiOS() throws Throwable
{
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(CapabilityType.VERSION, "6.1");
capabilities.setCapability(CapabilityType.PLATFORM, "Mac");
capabilities.setCapability("app", "sauce-storage:my_ios_app.zip");
capabilities.setCapability("device", "iPhone Simulator");
String sauceUserName = System.getenv("SAUCE_USER_NAME");
String sauceAccessKey = System.getenv("SAUCE_ACCESS_KEY");
//Create an instance of RemoteWebDriver and connect to the Appium server.
//Appium will launch the BmiCalc App in iPhone Simulator using the configurations specified in Desired Capabilities
driver = new RemoteWebDriver(new URL(MessageFormat.format("http://{0}:{1}@ondemand.saucelabs.com:80/wd/hub", sauceUserName, sauceAccessKey)), capabilities);
}
@Given("^I enter \"([^\"]*)\" as height$")
public void I_enter_as_height(String height) throws Throwable {
WebElement heightTextField = driver.findElement(By.name("Height"));
heightTextField.sendKeys(height);
}
@Then("^I enter \"([^\"]*)\" as weight$")
public void I_enter_as_weight(String weight) throws Throwable {
WebElement weightTextField = driver.findElement(By.name("Weight"));
weightTextField.sendKeys(weight);
}
@Then("^I press the Calculate button$")
public void I_press_the_Calculate_button() throws Throwable {
WebElement calculateButton = driver.findElement(By.name("Calculate"));
calculateButton.click();
}
@Then("^I should see \"([^\"]*)\" as bmi and \"([^\"]*)\" as category$")
public void I_should_see_as_bmi_and_Normal_as_category(String bmi, String category) throws Throwable {
WebElement bmilabel = driver.findElement(By.name("bmi"));
WebElement bmiCategorylabel = driver.findElement(By.name("category"));
//Check the calculated Bmi and Category displayed
assertEquals(bmi,bmilabel.getText());
assertEquals(category,bmiCategorylabel.getText());
}
@After
public void tearDown()
{
driver.quit();
}
}
Continuous Integration (CI) is becoming a widely accepted practice in agile projects for early feedback. Each integration is verified by an automated build to detect errors as quickly as possible. CI also gives great visibility to all stakeholders about the overall health of the project and its progress.
Jenkins is one of the popular CI tool used by the development teams. Let’s setup a build and test jobs on Jenkins for this project.
Building the App with Jenkins
For building the App with Jenkins, create a new Build a free-style software project Job. The sample App source is hosted on GitHub. Configure the Git repository and point it to the GitHub repo as show in below screen shot
Jenkins Project Configuration
Next, configure the build trigger, so every time a change is submitted to Git, this job will be trigger a new build of the App. Once the App is built it will be uploaded to the Sauce Labs temporary storage area so the test job can run acceptance tests on the build
Jenkins Build Trigger
Running Acceptance tests with Jenkins
For running acceptance test after a successful build, create a new Build a maven2/3 project job in Jenkins with following parameters
Jenkins Test Project
In the Build Trigger section configure Build after other project are built and specify name of the earlier job. This will automatically run acceptance test (i.e. current job) when the App build is successful
Jenkins Test Project Build Configuration
Cucumber provides a report plugin for Jenkins which create a nicely formatted interactive web reports as shown in below screenshot:
Cucumber Report
Summary
We can write automated acceptance tests for native mobile Apps with Cucumber, Selenium WebDriver API and Appium. These can be run on local, remote or Sauce Labs environment. We can also get benefits of whole team communication and early feedback by applying these methods in mobile application development.
This series was originally posted on Sauce Labs Blog. This is a reblog with revisions.
Welcome to the second part of Automated Acceptance Testing iOS Mobile Apps with Appium, Cucumber-JVM, Jenkins, and Sauce Labs. In this post we will define steps for the feature we identified in Part 1. We will run this feature with Maven and look at Cucumber results.
Running features with Maven
Before we move forward, we need to configure a support/helper class which tells Cucumber options for running features that we have added to the project. Add a new Java class RunCukesTest to scr/test/java/bmicalculator.test and copy following code
package bmicalculator.test;
import cucumber.junit.Cucumber;
import org.junit.runner.RunWith;
@RunWith(Cucumber.class)
@Cucumber.Options(format = {"pretty", "html:target/cucumber-html-report", "json-pretty:target/cucumber-report.json"})
public class RunCukesTest {
}
Now let’s run this feature file, in terminal navigate to the project folder and run following command
mvn test
After executing above command, Cucumber will suggest some step definitions for missing steps as below:
@Given("^I enter \"([^\"]*)\" as height$")
public void I_enter_as_height(String arg1) throws Throwable {
// Express the Regexp above with the code you wish you had
throw new PendingException();
}
@Given("^I enter \"([^\"]*)\" as weight$")
public void I_enter_as_weight(String arg1) throws Throwable {
// Express the Regexp above with the code you wish you had
throw new PendingException();
}
@Given("^I press the Calculate button$")
public void I_press_the_Calculate_button() throws Throwable {
// Express the Regexp above with the code you wish you had
throw new PendingException();
}
@Then("^I should see \"([^\"]*)\" as bmi and \"([^\"]*)\" as category$")
public void I_should_see_as_bmi_and_as_category(String arg1, String arg2) throws Throwable {
// Express the Regexp above with the code you wish you had
throw new PendingException();
}
Undefined step: Given I enter "170" as height
Undefined step: And I enter "50" as weight
Undefined step: And I press the Calculate button
Undefined step: Then I should see "17.30" as bmi and "Underweight" as category
Undefined step: Given I enter "181" as height
Undefined step: And I enter "80" as weight
Undefined step: And I press the Calculate button
Undefined step: Then I should see "24.42" as bmi and "Normal" as category
Undefined step: Given I enter "180" as height
Undefined step: And I enter "90" as weight
Undefined step: And I press the Calculate button
Undefined step: Then I should see "27.78" as bmi and "Overweight" as category
Undefined step: Given I enter "175" as height
Undefined step: And I enter "100" as weight
Undefined step: And I press the Calculate button
Undefined step: Then I should see "32.65" as bmi and "Obese" as category
1 scenario (0 passed)
16 steps (0 passed)
Implementing Step Definitions
Cucumber drives features & scenarios with automated steps using step definitions. Step definitions are glue between the steps described in a Scenario to the actual automation code which drive the App either through API or User Interface. The App user interface many not be ready at this time.
While development team is busy in developing the App we can create place holders for the steps from the sample scenario as suggested by Cucumber.
Add new Java class Bmi_Calculator_Step_Defs to src/test/java/bmicalculator.test package and implement steps for our example feature. We will mark them as pending with Cucumber PendingExeception() as below:
package bmicalculator.test;
import cucumber.annotation.en.*;
import cucumber.runtime.PendingException;
public class Bmi_Calculator_Step_Defs {
@Given("^I enter \"([^\"]*)\" as height$")
public void I_enter_as_height(String arg1) throws Throwable {
// Express the Regexp above with the code you wish you had
throw new PendingException();
}
@Given("^I enter \"([^\"]*)\" as weight$")
public void I_enter_as_weight(String arg1) throws Throwable {
// Express the Regexp above with the code you wish you had
throw new PendingException();
}
@Given("^I press the Calculate button$")
public void I_press_the_Calculate_button() throws Throwable {
// Express the Regexp above with the code you wish you had
throw new PendingException();
}
@Then("^I should see \"([^\"]*)\" as bmi and \"([^\"]*)\" as category$")
public void I_should_see_as_bmi_and_as_category(String arg1, String arg2) throws Throwable {
// Express the Regexp above with the code you wish you had
throw new PendingException();
}
}
When this feature is executed by Cucumber, it will show these steps as pending.
Once developers are done with the initial version of the App and it’s ready for testing we will refactor these steps and add code to drive the user interface of the App using Appium and Selenium WebDriver API to check the expected behavior of the App.
We need to declare an instance of WebDriver class from Selenium WebDriver API. We will add a setup method setupLocal() which will initialize the driver instance and connect to Appium running on a local machine
@Before
public void setUpLocal() throws Throwable {
//Setup for running Appium test in local environment
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(CapabilityType.BROWSER_NAME, "iOS");
capabilities.setCapability(CapabilityType.VERSION, "6.1");
capabilities.setCapability(CapabilityType.PLATFORM, "Mac");
capabilities.setCapability("app", "/Users/upgundecha/Desktop/AppExamples/BmiCalculator/build/Release-iphonesimulator/BmiCalculator.app");
//Create an instance of RemoteWebDriver and connect to the Appium server.
//Appium will launch the BmiCalc App in iPhone Simulator using the configurations specified in Desired Capabilities
driver = new RemoteWebDriver(new URL("http://0.0.0.0:4723/wd/hub"), capabilities);
}
Also implement the steps with Selenium WebDriver API to interact with the App elements in following way
public void I_enter_as_height(String height) throws Throwable {
WebElement heightTextField = driver.findElement(By.name("Height"));
heightTextField.sendKeys(height);
}
@Then("^I enter \"([^\"]*)\" as weight$")
public void I_enter_as_weight(String weight) throws Throwable {
WebElement weightTextField = driver.findElement(By.name("Weight"));
weightTextField.sendKeys(weight);
}
@Then("^I press the Calculate button$")
public void I_press_the_Calculate_button() throws Throwable {
WebElement calculateButton = driver.findElement(By.name("Calculate"));
calculateButton.click();
}
@Then("^I should see \"([^\"]*)\" as bmi and \"([^\"]*)\" as category$")
public void I_should_see_as_bmi_and_Normal_as_category(String bmi, String category) throws Throwable {
WebElement bmilabel = driver.findElement(By.name("bmi"));
WebElement bmiCategorylabel = driver.findElement(By.name("category"));
//Check the calculated Bmi and Category displayed
assertEquals(bmi,bmilabel.getText());
assertEquals(category,bmiCategorylabel.getText());
}
At the end add a tearDown() method to close the App
public void tearDown() {
driver.quit();
}
This makes the step definitions complete. When this feature is run, Cucumber will call the Selenium WebDriver API which in turn calls the Appium and steps are performed on the App’s user interface in iOS Simulator. At the end of execution Cucumber will display results as below
1 scenario (1 passed)
16 steps (16 passed)
In the next post we will explore how to run Cucumber features with Sauce Labs cloud and Jenkins.
This series was originally posted on Sauce Labs Blog This is a reblog with revisions.
Overview
Agile development projects are adopting practices such as automated acceptance testing, BDD, continuous integration, and others. There is a lot written about these approaches for developing software, their benefits and limitations etc. However, one of the fundamental benefits these practices offers is enhanced communication between project stakeholders including users, product owner, and the development team, a shorter feedback loop and agility. These practices requires all the project participants to come together, discuss, and elicit the behavior of the application in agreed upon format of features, user stories and acceptance criteria and shared definition of “Done!”. Read more about these in Wikipedia
Mobile application development project can leverage on these advantages in building mobile apps in shorter cycles and feedback loops to build a successful applications. In this series of posts I will explain how you can apply automated acceptance testing to iOS native applications using Appium and Cucumber-JVM. You will also see how these applications can be tested in cloud using Sauce Labs platform.
Cucumber-JVM
Cucumber-JVM is a pure Java implementation of original Cucumber BDD/ATDD framework. It supports the most popular programming languages on the JVM. It’s already been used by various teams along with Selenium WebDriver for testing Web applications. Cucumber support creating features files which are written in a ubiquitous language understood by the whole team. These feature file describe the expected behavior of the application and are used as tests to run against the application. Cucumber can be used for API, integration and functional testing of the application.
In this example we will see functional testing of a sample native iOS App using Cucumber-JVM and Appium.
The Sample App
This example is based on a sample BMI Calculator application which is used by Health & Nutrition specialists to calculate the Body Mass Index of patients by submitting Height and Weight values to the App.
Bmi Calculator App
Let’s work on one of the main feature of this App as described below
Feature: Calculating Body Mass Index
As a health specialist
I want a BMI Calculator
So that I can calculate patient's Body Mass Index
Setting up Test Project
Let’s setup a new project using with IntelliJ IDEA using Maven & Cucumber-JVM with following steps:
Create a new Project as Maven module, provide appropriate values for <bGroupId and ArtifactId. In this example GroupId is set with org.bmicalc.test value and ArtifactId as bmicalculator.test
Once Intellij IDEA creates project with appropriate folder structure, locate and modify pom.xml file. Add below highlighted dependencies to pom.xml
This will add Cucumber-JVM and Selenium dependencies to the project.
Writing Feature File
In Cucumber-JVM specifications or requirements are expressed in a plain text, Given/When/Then kind of syntax known as Gherkin language (https://github.com/cucumber/cucumber/wiki/Gherkin) which is understood by the whole team. So let’s create a feature file for above feature in the project.
Add a new package bmicalculator.test under src/test/resources as shown in below screenshot and add a new file with name bmi_calculator.feature
Project Folder
Copy following feature text to bmi_calculator.feature file
Feature: Calculating Body Mass Index
As a health specialist
I want a BMI Calculator
So that I can calculate patient's Body Mass Index
BmiCalculator
@ios-sauce
Scenario Outline: Calculate Body Mass Index
Given I enter &quot;&lt;Height&gt;&quot; as height
And I enter &quot;&lt;Weight&gt;&quot; as weight
And I press the Calculate button
Then I should see &quot;&lt;BMI&gt;&quot; as bmi and &quot;&lt;Category&gt;&quot; as category
Examples:
|Height |Weight |BMI |Category |
|170 |50 |17.30|Underweight|
|181 |80 |24.42|Normal |
|180 |90 |27.78|Overweight |
|175 |100 |32.65|Obese |
Every feature file contains a single feature. A feature usually contains a list of scenarios. Every scenario consists of a list of steps, which must start with one of the keywords Given, When, Then, But or And. Scenarios express expected behavior of the system under given conditions.
In addition to a scenario, a feature may contain a background, scenario outline and examples. Our example scenario contains Scenario background and examples for number of BMI calculations representing each category.
This scenario outlines allow us to more concisely express these examples through use of a template with placeholders. In this example Calculate Body Mass Index is run once for each row in the Examples section beneath it (not counting the first row which is a header). This is similar to data driven testing.
In next part, we’ll see how to run features with Maven, and enter step definitions.
Recently I started working with SpecFlow.NET a very popular BDD tool available for .NET. It does a impressive job of automating acceptance testing in agile projects.
Out of curiosity I started a small proof-of-concept to build a similar framework with HP QTP, a popular GUI automation tool. This post covers the basics of the framework I built and as I indicated in the title, this framework does not have all the features like SepcFlow or Cucumber. This is just a rough implementation to prove that tools like QTP can also be used in BDD.
Behaviour Driven Development focuses on obtaining a clear understanding of desired application behaviour through discussion with stakeholders. In a nutshell using BDD technique you author automated tests in the language of the business while maintaining a connection to implemented system. Users describe features and scenarios to test these features in plain text files using Gherkin language in Given, When and Then structure. For more information on Gherkin language see http://en.wikipedia.org/wiki/Behavior-driven_development and https://github.com/cucumber/cucumber/wiki/Gherkin
Given [Precondition]
When [Actor + Action]
Then [Observable Result]
In BDD process starts with users of the system and development team discussing features, user stories and scenarios. These are documented in feature or story files using the Gherkin language. Developers then use the redgreenre-factor cycle to run these features using the BDD framework then writing step definition files mapping the steps from scenarios to the automation code and re-running, until all the acceptance criteria are met.
Here is a list of some popular BDD tools:
Cucumber, RSpec – Ruby
Cucumber-JVM, JBehave – Java
SpecFlow.NET – .NET
This example demonstrates acceptance testing of Fund Transfer feature from an on-line banking application using BDD. Using the fund transfer application a bank account holder can transfer or wire money to a registered payee. There are some validations around this feature which are captured in the specification.
Step 1 – Creating Feature Files
The first step is to create a feature file. A feature file defines specifications using Gherkin language for a feature under development. It’s a simple text file which starts with the feature description in a user story format. Then acceptance criteria is listed in the form of scenarios in the feature file. In below feature file, sample scenarios are described to validate the fund transfer application:
Feature: Customer Transfers Fund
As a customer,
I want to transfer funds
so that I can send money to my friends and family
Scenario: Valid Payee
Given the user is on Fund Transfer Page
When he enters "Jim" as payee name
And he enters "100" as amount
And he Submits request for Fund Transfer
Then ensure the fund transfer is complete with "$100 transferred to Jim successfully!!" message
Scenario: Invalid Payee
Given the user is on Fund Transfer Page
When he enters "Jack" as payee name
And he enters "100" as amount
And he Submits request for Fund Transfer
Then ensure a transaction failure message "Transfer failed!! 'Jack' is not registered in your List of Payees" is displayed
Scenario: Account is overdrawn past the overdraft limit
Given the user is on Fund Transfer Page
When he enters "Tina" as payee name
And he enters "1000000" as amount
And he Submits request for Fund Transfer
Then ensure a transaction failure message "Transfer failed!! account cannot be overdrawn" is displayed
Feature files are kept in a Feature folder in QTP script folder. Separate feature files are created for each feature based on number of features under test. The feature file should use .feature extension.
Test data values needed to execute steps can be embedded using double quotes. In above feature there is a step which says When he enters “Jim” as payee name. Here name of the payee is quoted.
Step 2 – Creating Step Definition File
A step definition file is created which maps the steps from a feature file to the automation code. Here is a step definition file implemented for fund transfer feature as follows:
'//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
' Script Name: FundTransferStepDefs.vbs
' Author: Unmesh Gundecha
'
' Description: This file contains step definitions for Fund Transfer Feature
' While defining function, please specfiy Step definition as comment above the function name in square brackets []
'//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
'[Given the user is on Fund Transfer Page]
Function GivenUserIsOnFundTransferPage()
GivenUserIsOnFundTransferPage = Browser("Browser").CheckProperty("title","Fund Transfer Page")
End Function
'[When he enters "name" as payee name]
Function WhenUserEneteredIntoThePayeeNameField(strPayeeName)
Browser("Browser").Page("Fund Transfer Page").WebEdit("payeeTextBox").Set strPayeeName
End Function
'[And he enters "amount" as amount]
Function AndUserEneteredIntoTheAmountField(amount)
Browser("Browser").Page("Fund Transfer Page").WebEdit("amountTextBox").Set amount
End Function
'[And he Submits request for Fund Transfer]
Function AndUserPressTransferButton()
Browser("Browser").Page("Fund Transfer Page").WebButton("Transfer").Click
End Function
'[Then ensure the fund transfer is complete with "message" message]
Function ThenFundTransferIsComplete(message)
ThenFundTransferIsComplete = Browser("Browser").Page("Fund Transfer Page").WebElement("message").CheckProperty("innerText", message)
End Function
'[Then ensure a transaction failure message "message" is displayed]
Function ThenFundTransferIsFailed(message)
ThenFundTransferIsFailed = Browser("Browser").Page("Fund Transfer Page").WebElement("message").CheckProperty("innerText", message)
End Function
For each unique step in the feature file, a function is written which calls QTP code as per the behaviour specified in the step. This file is implemented as a function library file. These functions are mapped to steps using a special commenting standard. For example there is a step When he enters “Jim” as payee name in feature file. For this step a WhenUserEneteredIntoThePayeeNameField() function is defined in the step definition file . A special comment is written on top of the function declaration. This will tell driver script to map the matching step from feature file to this function.
'[When he enters "name" as payee name]
Function WhenUserEneteredIntoThePayeeNameField(strPayeeName)
Browser("Browser").Page("Fund Transfer Page").WebEdit("payeeTextBox").Set strPayeeName
End Function
Step 3 – Creating Driver
The BDD_Driver script is core of this framework. This script reads feature file and step definition file and creates a map of steps along with their corresponding functions. It then executes scenarios from feature file using this map.
QTP BDD_Framework Architecture
BDD_Driver script also provides test data embedded in a step from feature file to the function called from step definition file. A step definition function internally calls QTP code which runs the automated step on the application emulating user behaviour.
'//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
' Script Name: BDD_Driver
' Author: Unmesh Gundecha
'
' Description: This is the main script of QTP-BDD Framework. This drives the execution of features and steps
' Script first reads Step Definition File and creates a dictionary of available steps and functions.
'
' For executing a Feature file it refer the dictionary matching steps from feature file and calling the respective function
' While calling the step function it passes the test data embedded in Feature file steps to the function
'//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
Option Explicit
Dim arrStepDefs, objStepDict, intLineIdx, strLine, strNextLine, arrFeatureFile, intScenarios, strArgs, varResult
Dim strKey, arrArgs, strTestStep, strStepStatus, intCnt
'Open the Test Application, by directly providing URL, QTP will launch the Browser and navigate to the specified URL
SystemUtil.Run "http://dl.dropbox.com/u/55228056/fundTransfer.html"
'Go through the Step Definition file and create dictionary of step definition and corresponding functions
arrStepDefs = Split(DotNetFactory.CreateInstance("System.IO.File").ReadAllText(PathFinder.Locate("FundTransferStepDefs.vbs")),vbCrLf)
Set objStepDict = CreateObject("Scripting.Dictionary")
For intLineIdx = 0 To UBound(arrStepDefs)
strLine = arrStepDefs(intLineIdx)
If strLine <> "" Then
If Left(strLine,2) = "'[" And Right(strLine,1) = "]" Then
strLine = Replace(Replace(strLine,"'[",""),"]","")
strLine = ParseLine("Step", strLine)
strNextLine = Trim(arrStepDefs(intLineIdx+1))
strNextLine = Replace(Replace(strNextLine,"Function",""),"Sub","")
strNextLine = Mid(strNextLine,1,InStr(strNextLine,"(")-1)
objStepDict.Add strLine, strNextLine
End If
End If
Next
'Read Feature file and match step definitions in objStepDict
arrFeatureFile = Split(DotNetFactory.CreateInstance("System.IO.File").ReadAllText(PathFinder.Locate("FundTransfer.feature")),vbCrLf)
For intLineIdx = 0 To UBound(arrFeatureFile)
If InStr(arrFeatureFile(intLineIdx),"Feature:") <> 0 Then
Print "Feature: " & Trim(Replace(arrFeatureFile(intLineIdx),"Feature:",""))
intScenarios = 0
ElseIf InStr(arrFeatureFile(intLineIdx),"Scenario:") <> 0 Then
intScenarios = intScenarios + 1
Print vbTab & "Scenario: " & Trim(Replace(arrFeatureFile(intLineIdx),"Scenario:",""))
Else
strArgs = ""
varResult = ""
strLine = Replace(Trim(arrFeatureFile(intLineIdx)),vbTab,"")
If strLine <> "" Then
strKey = ParseLine("Step",strLine)
'After a step is found in dictionary generate the function call using the test data embedded in step from feature file
If objStepDict.Exists(strKey) Then
'Get Test Data Values from Feature File Steps and create an Argument Array
If ParseLine("Args",strLine) <> "" Then
arrArgs = Split(ParseLine("Args",strLine),";")
For intCnt = 0 to UBound(arrArgs)
strArgs = strArgs & "arrArgs(" & intCnt & "),"
Next
strArgs = Mid(strArgs,1,Len(strArgs)-1)
strTestStep = "varResult = " & objStepDict(strKey) & "(" & strArgs & ")"
Else
strTestStep = "varResult = " & objStepDict(strKey)
End If
'Execute the function call
Execute(strTestStep)
If varResult <> "" Then
If varResult Then
strStepStatus = " [PASS]"
Else
strStepStatus = " [FAIL]"
End If
End If
Print vbTab & vbTab & strLine & strStepStatus
Else
If InStr(strLine,"Given") <> 0 Or InStr(strLine,"When") <> 0 Or InStr(strLine,"Then") <> 0 Or InStr(strLine,"And") <> 0 Then
Print vbTab & vbTab & strLine & " [PENDING]"
Else
Print vbTab & vbTab & strLine
End If
End If
End If
End If
Next
Print vbCrLf & "Total " & intScenarios & " Scenarios Executed!!"
Function ParseLine(ByVal strWhatToParse, ByVal strLine)
Dim objRegEx, objMatches, objMatch, strResult
Set objRegEx = New RegExp
objRegEx.Pattern = "\""(.*?)\"""
objRegEx.IgnoreCase = True
objRegEx.Global = True
Set objMatches = objRegEx.Execute(strLine)
For Each objMatch in objMatches
If strWhatToParse = "Step" Then
strResult = strResult & Replace(strLine,objMatch,vbNullString)
ElseIf strWhatToParse = "Args" Then
strResult = strResult & Replace(objMatch,"""","") & ";"
End If
Next
If strResult <> "" Then
If strWhatToParse = "Step" Then ParseLine = strResult
If strWhatToParse = "Args" Then ParseLine = Mid(strResult,1,Len(strResult)-1)
Else
If strWhatToParse = "Step" Then ParseLine = strLine
If strWhatToParse = "Args" Then ParseLine = ""
End If
End Function
This code might look little odd due to the language constraints that we have with VBScript. However it proves that BDD is possible with tools like QTP, however a better solution can be built in detail. Here is a sample report generated by the BDD_Driver
Next steps:
This is not complete framework and needs to be enhanced further as a mature BDD framework. Here are some key items that can be added/improved in next iteration:
Support multiple Feature and Step Definition files
100% Gherkin syntax support
Suggest step definitions where functions are not implemented for steps in log/report
Support for ignoring steps where step definition is not implemented
Support for reading tabular data from steps
Report Generation like SpecFlow
Comparison with Keyword Driven testing
Implementing tests with BDD is pretty much similar to Keyword driven testing. In Keyword driven testing, Keywords or Actions are defined and mapped to functions which execute code related to the Keyword. Tests are written in tabular format which is easy to understand for users without having knowledge of the test tool or automation code. The driver script orchestrates the entire test execution.
In BDD, steps are written in a feature file can be compared to Keywords or actions and step definitions are the underlying functions created for the Keywords. However tests are expressed in a non tabular format, in a more natural plain text or domain specific language.
I’ll be happy to know your comments or suggestion to improve this further.