Recently I started working with SpecFlow.NET a very popular BDD tool available for .NET. It does a impressive job of automating acceptance testing in agile projects.
Out of curiosity I started a small proof-of-concept to build a similar framework with HP QTP, a popular GUI automation tool. This post covers the basics of the framework I built and as I indicated in the title, this framework does not have all the features like SepcFlow or Cucumber. This is just a rough implementation to prove that tools like QTP can also be used in BDD.
There is enough written about Behaviour Driven Development (BDD), and you can know more about it from Dan North’s legendary article http://dannorth.net/introducing-bdd/ and Wikipedia entry http://en.wikipedia.org/wiki/Behavior-driven_development
Behaviour Driven Development focuses on obtaining a clear understanding of desired application behaviour through discussion with stakeholders. In a nutshell using BDD technique you author automated tests in the language of the business while maintaining a connection to implemented system. Users describe features and scenarios to test these features in plain text files using Gherkin language in Given, When and Then structure. For more information on Gherkin language see http://en.wikipedia.org/wiki/Behavior-driven_development and https://github.com/cucumber/cucumber/wiki/Gherkin
- Given [Precondition]
- When [Actor + Action]
- Then [Observable Result]
In BDD process starts with users of the system and development team discussing features, user stories and scenarios. These are documented in feature or story files using the Gherkin language. Developers then use the red green re-factor cycle to run these features using the BDD framework then writing step definition files mapping the steps from scenarios to the automation code and re-running, until all the acceptance criteria are met.
Here is a list of some popular BDD tools:
- Cucumber, RSpec – Ruby
- Cucumber-JVM, JBehave – Java
- SpecFlow.NET – .NET
This example demonstrates acceptance testing of Fund Transfer feature from an on-line banking application using BDD. Using the fund transfer application a bank account holder can transfer or wire money to a registered payee. There are some validations around this feature which are captured in the specification.
Step 1 – Creating Feature Files
The first step is to create a feature file. A feature file defines specifications using Gherkin language for a feature under development. It’s a simple text file which starts with the feature description in a user story format. Then acceptance criteria is listed in the form of scenarios in the feature file. In below feature file, sample scenarios are described to validate the fund transfer application:
Feature: Customer Transfers Fund As a customer, I want to transfer funds so that I can send money to my friends and family Scenario: Valid Payee Given the user is on Fund Transfer Page When he enters "Jim" as payee name And he enters "100" as amount And he Submits request for Fund Transfer Then ensure the fund transfer is complete with "$100 transferred to Jim successfully!!" message Scenario: Invalid Payee Given the user is on Fund Transfer Page When he enters "Jack" as payee name And he enters "100" as amount And he Submits request for Fund Transfer Then ensure a transaction failure message "Transfer failed!! 'Jack' is not registered in your List of Payees" is displayed Scenario: Account is overdrawn past the overdraft limit Given the user is on Fund Transfer Page When he enters "Tina" as payee name And he enters "1000000" as amount And he Submits request for Fund Transfer Then ensure a transaction failure message "Transfer failed!! account cannot be overdrawn" is displayed
Feature files are kept in a Feature folder in QTP script folder. Separate feature files are created for each feature based on number of features under test. The feature file should use .feature extension.
Test data values needed to execute steps can be embedded using double quotes. In above feature there is a step which says When he enters “Jim” as payee name. Here name of the payee is quoted.
Step 2 – Creating Step Definition File
A step definition file is created which maps the steps from a feature file to the automation code. Here is a step definition file implemented for fund transfer feature as follows:
'////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// ' Script Name: FundTransferStepDefs.vbs ' Author: Unmesh Gundecha ' ' Description: This file contains step definitions for Fund Transfer Feature ' While defining function, please specfiy Step definition as comment above the function name in square brackets  '////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// '[Given the user is on Fund Transfer Page] Function GivenUserIsOnFundTransferPage() GivenUserIsOnFundTransferPage = Browser("Browser").CheckProperty("title","Fund Transfer Page") End Function '[When he enters "name" as payee name] Function WhenUserEneteredIntoThePayeeNameField(strPayeeName) Browser("Browser").Page("Fund Transfer Page").WebEdit("payeeTextBox").Set strPayeeName End Function '[And he enters "amount" as amount] Function AndUserEneteredIntoTheAmountField(amount) Browser("Browser").Page("Fund Transfer Page").WebEdit("amountTextBox").Set amount End Function '[And he Submits request for Fund Transfer] Function AndUserPressTransferButton() Browser("Browser").Page("Fund Transfer Page").WebButton("Transfer").Click End Function '[Then ensure the fund transfer is complete with "message" message] Function ThenFundTransferIsComplete(message) ThenFundTransferIsComplete = Browser("Browser").Page("Fund Transfer Page").WebElement("message").CheckProperty("innerText", message) End Function '[Then ensure a transaction failure message "message" is displayed] Function ThenFundTransferIsFailed(message) ThenFundTransferIsFailed = Browser("Browser").Page("Fund Transfer Page").WebElement("message").CheckProperty("innerText", message) End Function
For each unique step in the feature file, a function is written which calls QTP code as per the behaviour specified in the step. This file is implemented as a function library file. These functions are mapped to steps using a special commenting standard. For example there is a step When he enters “Jim” as payee name in feature file. For this step a WhenUserEneteredIntoThePayeeNameField() function is defined in the step definition file . A special comment is written on top of the function declaration. This will tell driver script to map the matching step from feature file to this function.
'[When he enters "name" as payee name] Function WhenUserEneteredIntoThePayeeNameField(strPayeeName) Browser("Browser").Page("Fund Transfer Page").WebEdit("payeeTextBox").Set strPayeeName End Function
Step 3 – Creating Driver
The BDD_Driver script is core of this framework. This script reads feature file and step definition file and creates a map of steps along with their corresponding functions. It then executes scenarios from feature file using this map.
BDD_Driver script also provides test data embedded in a step from feature file to the function called from step definition file. A step definition function internally calls QTP code which runs the automated step on the application emulating user behaviour.
'////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// ' Script Name: BDD_Driver ' Author: Unmesh Gundecha ' ' Description: This is the main script of QTP-BDD Framework. This drives the execution of features and steps ' Script first reads Step Definition File and creates a dictionary of available steps and functions. ' ' For executing a Feature file it refer the dictionary matching steps from feature file and calling the respective function ' While calling the step function it passes the test data embedded in Feature file steps to the function '////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// Option Explicit Dim arrStepDefs, objStepDict, intLineIdx, strLine, strNextLine, arrFeatureFile, intScenarios, strArgs, varResult Dim strKey, arrArgs, strTestStep, strStepStatus, intCnt 'Open the Test Application, by directly providing URL, QTP will launch the Browser and navigate to the specified URL SystemUtil.Run "http://dl.dropbox.com/u/55228056/fundTransfer.html" 'Go through the Step Definition file and create dictionary of step definition and corresponding functions arrStepDefs = Split(DotNetFactory.CreateInstance("System.IO.File").ReadAllText(PathFinder.Locate("FundTransferStepDefs.vbs")),vbCrLf) Set objStepDict = CreateObject("Scripting.Dictionary") For intLineIdx = 0 To UBound(arrStepDefs) strLine = arrStepDefs(intLineIdx) If strLine <> "" Then If Left(strLine,2) = "'[" And Right(strLine,1) = "]" Then strLine = Replace(Replace(strLine,"'[",""),"]","") strLine = ParseLine("Step", strLine) strNextLine = Trim(arrStepDefs(intLineIdx+1)) strNextLine = Replace(Replace(strNextLine,"Function",""),"Sub","") strNextLine = Mid(strNextLine,1,InStr(strNextLine,"(")-1) objStepDict.Add strLine, strNextLine End If End If Next 'Read Feature file and match step definitions in objStepDict arrFeatureFile = Split(DotNetFactory.CreateInstance("System.IO.File").ReadAllText(PathFinder.Locate("FundTransfer.feature")),vbCrLf) For intLineIdx = 0 To UBound(arrFeatureFile) If InStr(arrFeatureFile(intLineIdx),"Feature:") <> 0 Then Print "Feature: " & Trim(Replace(arrFeatureFile(intLineIdx),"Feature:","")) intScenarios = 0 ElseIf InStr(arrFeatureFile(intLineIdx),"Scenario:") <> 0 Then intScenarios = intScenarios + 1 Print vbTab & "Scenario: " & Trim(Replace(arrFeatureFile(intLineIdx),"Scenario:","")) Else strArgs = "" varResult = "" strLine = Replace(Trim(arrFeatureFile(intLineIdx)),vbTab,"") If strLine <> "" Then strKey = ParseLine("Step",strLine) 'After a step is found in dictionary generate the function call using the test data embedded in step from feature file If objStepDict.Exists(strKey) Then 'Get Test Data Values from Feature File Steps and create an Argument Array If ParseLine("Args",strLine) <> "" Then arrArgs = Split(ParseLine("Args",strLine),";") For intCnt = 0 to UBound(arrArgs) strArgs = strArgs & "arrArgs(" & intCnt & ")," Next strArgs = Mid(strArgs,1,Len(strArgs)-1) strTestStep = "varResult = " & objStepDict(strKey) & "(" & strArgs & ")" Else strTestStep = "varResult = " & objStepDict(strKey) End If 'Execute the function call Execute(strTestStep) If varResult <> "" Then If varResult Then strStepStatus = " [PASS]" Else strStepStatus = " [FAIL]" End If End If Print vbTab & vbTab & strLine & strStepStatus Else If InStr(strLine,"Given") <> 0 Or InStr(strLine,"When") <> 0 Or InStr(strLine,"Then") <> 0 Or InStr(strLine,"And") <> 0 Then Print vbTab & vbTab & strLine & " [PENDING]" Else Print vbTab & vbTab & strLine End If End If End If End If Next Print vbCrLf & "Total " & intScenarios & " Scenarios Executed!!" Function ParseLine(ByVal strWhatToParse, ByVal strLine) Dim objRegEx, objMatches, objMatch, strResult Set objRegEx = New RegExp objRegEx.Pattern = "\""(.*?)\""" objRegEx.IgnoreCase = True objRegEx.Global = True Set objMatches = objRegEx.Execute(strLine) For Each objMatch in objMatches If strWhatToParse = "Step" Then strResult = strResult & Replace(strLine,objMatch,vbNullString) ElseIf strWhatToParse = "Args" Then strResult = strResult & Replace(objMatch,"""","") & ";" End If Next If strResult <> "" Then If strWhatToParse = "Step" Then ParseLine = strResult If strWhatToParse = "Args" Then ParseLine = Mid(strResult,1,Len(strResult)-1) Else If strWhatToParse = "Step" Then ParseLine = strLine If strWhatToParse = "Args" Then ParseLine = "" End If End Function
This code might look little odd due to the language constraints that we have with VBScript. However it proves that BDD is possible with tools like QTP, however a better solution can be built in detail. Here is a sample report generated by the BDD_Driver
This is not complete framework and needs to be enhanced further as a mature BDD framework. Here are some key items that can be added/improved in next iteration:
- Support multiple Feature and Step Definition files
- 100% Gherkin syntax support
- Suggest step definitions where functions are not implemented for steps in log/report
- Support for ignoring steps where step definition is not implemented
- Support for reading tabular data from steps
- Report Generation like SpecFlow
Comparison with Keyword Driven testing
Implementing tests with BDD is pretty much similar to Keyword driven testing. In Keyword driven testing, Keywords or Actions are defined and mapped to functions which execute code related to the Keyword. Tests are written in tabular format which is easy to understand for users without having knowledge of the test tool or automation code. The driver script orchestrates the entire test execution.
In BDD, steps are written in a feature file can be compared to Keywords or actions and step definitions are the underlying functions created for the Keywords. However tests are expressed in a non tabular format, in a more natural plain text or domain specific language.
I’ll be happy to know your comments or suggestion to improve this further.
I’ll be running this post on a regular interval, sharing with you four best links on testing, test automation and emerging technologies. This is first in the series and hope you like it.
Four Picks #1
- StarEast 2011 Making Test Automation Work in Agile Projects presentation from Lisa Crispin. This presentation talks about applying whole team approach for test automation in Agile Projects. There is a great wisdom written in these slides.
- I.M.Testy (a.k.a Bj Rollison) has written interesting post Automation isn’t bad; bad automation is bad!!
- Spring 2011 issue of Methods and Tools is out with some interesting articles. I particularly liked
- Automated Acceptance Tests & Requirements Traceability – This article explores an approach for automated acceptance testing of a Java application using Concordian.
- RSpec Best Practices – RSpec is a widely used tool for BDD in Ruby. Author explains various best practices while using RSpec in BDD/TDD approach.
- Design Patterns In Python – I came across this interesting online book through a Twitter link. This book explains commonly used design patterns in Python and how these can be used in Test Automation context. Great read!!