Title: Susan Windsor
1Strategic Direction for Functional Test
Automation SoftTest 2005
- Susan Windsor
- Insight Through Intelligence
- WMHL Consulting Limited, MD
Title slide
2AGENDA
- Future for Functional Testing
- History of Functional Automation
- Strategic Direction
- Automation Frameworks in Action
- Impact upon You?
3Future for Functional Testing
4Increased Demand..?
- Global development, with large projects having
multi site, multi geography and multi suppliers
to contend with. - Corporate and regulatory requirements growing all
the time - Business demands new applications, faster and
cheaper to obtain competitive advantage - Closer alignment of IT and business to repair
lack of business confidence
5..Or, increased pressures?
- Automation frameworks and greater business
knowledge requirements, mean reduced numbers of
traditional functional testers - Agile development methods mean developers
undertake more unit and component testing - Growth of outsourced testing to different
geographies, leading to greater competition for
the roles
Is the role of the functional tester (who is
neither technical or business specialist) dead?
6History of Functional Automation
7Record and Playback
- all you need to do.!
- Easy to record the scripts
- Extremely fragile
- Expensive to maintain
- On average, each test run requires at least 50
of the scripts to be recorded again
Please tell me no one does this anymore!!!
8Scripting
- Use the scripting language to write scripts that
do what you want - Build in as much robustness as possible
- But, youre building an application to test an
application, which also needs testing! - Maintenance costs can be very high
- Needs programmer skills
- Cost of implementation not affordable at project
level
9Table Driven
- Like scripting but more flexible and greater
re-use - Remove items from the script that change, e.g.
data - Reduces maintenance costs a bit
- Implementation costs about the same
- Still requires specialist skills to implement
10Functional Test Automation is Broken!
- Focus on technology rather than business needs
- 80 of functional testing still manual
- 60 to 70 of automation tools used for
non-functional testing - Typically, traditional functional automation
stops at 100 scripts, regardless of test coverage
requirement - Critical factors cost of implementation and
maintenance, skills required and asset sharing
over different technologies
11Strategic Direction
12Automation Frameworks
- Wouldnt it be good if..
- Tests could be documented in common format,
regardless of whether they are manual or
automated - The format for the tests resulted in quicker
preparation time than traditional manual tests - Both developers and business testers could
understand and use the same test format - Script maintenance was no longer a requirement
- Tests could use any of the test execution tools
required by the underlying technology
Faster planning, faster execution, and far less
technical skill required
13Business Analysts already using them, and use
will grow
- Home grown frameworks built within organisations
to meet business demands - Niche suppliers providing frameworks try a
Google search 512,000 hits this week - Seen a handful that appear mature
- Latest review by Paul Herzlich (OVUM analyst)
- Market Leaders such as Mercury developing
Business Process Tester (BPT)
This is the industry direction now
14Automation Frameworks in Action
15The project
- Lloyds TSB Offshore in Channel Islands
- Migration from current banking platform IBIS to
Temenos Globus G13 - Tight deadlines due to EU Savings Tax Directive
coming into force on 1st July 2005 - Business processes required additional Globus
modules - Multiple releases leading up to deadline required
substantial testing - SQS-UK contacted to provide consultancy
16The Team
- SQS-UK recognised the requirement for test
automation - Joint team from Odin Technology and SQS-UK
assembled - Working alongside Lloyds TSB staff
- Odin Technology provided testing technology
- SQS-UK provided implementation resource and
testing know-how
17Project Approach
- Split project roles for Automation Technicians
and Testers - Three phase technical approach
- Tool Evaluation Technical Feasibility
- Technical Implementation
- Support
- Test Analysis and Design throughout
18Project Phase 1Tool Evaluation and Technical
Feasibility
19Tool Evaluation
- Automation isnt always straightforward!
- GUI Tools dont work out of the box for all
environments - They will almost certainly require some custom
functionality to interact with the SUT - The Evaluation process
- Assess what custom functionality is required
- Assess what involvement is required from an
Automation Technician to provide the custom
functionality - Assess if there are any other efficiency
improvements that can be made through application
of technology - This provides the scope of technical
implementation phase
20Project Phase 2 What would normally happen...
21Typical Automation Project
- Take existing Manual Scripts
- Rework by hand into automated scripts function
libraries - The bulk of the work
- Building the Object Map
- Providing the Mapping between the business terms
used to describe the SUT Interface and the test
tools requirements - Coding Test Scripts
- Providing the instructions and associated data
for the test tool to interact with the SUT and
perform testing
22Object Mapping
- Providing the Mapping between the business terms
used to describe the SUT Interface and the test
tools requirements
23Example
Manual Tester
The Login Window
Test Tool
class MSWDialog label Login.
24Example 2
Manual Tester
Username
Test Tool
class HTMLEdit id Lgn_Uid
25Object Mapping in a typical Automation Tool
- Usually implemented under different names in
different tools - GUI Map
- Object Repository
- Object Map
- Mapping an Object
- Start the Object recognition tool
- Point at the Object with the Mouse Pointer
- Select the Object
- Give the Object a meaningful business name
- Make manual amendments to the properties if
required
26Test Script Creation
- Providing the instructions and associated data
for the test tool to interact with the SUT and
perform testing
27Scripting Approach
- Use the Tools Scripting Language to construct
business processes
28(No Transcript)
29Project Phase 2 What was done differently
- Taught an old dog some new tricks
30Test Design Model
- A simple structured way of defining tests that is
independent of the interface and the means of
execution
31Features of the Test Design Model
- Easy to design tests without technical knowledge
- Well structured
- Can be used to describe tests for any interface
- Graphical User Interfaces (GUI)
- Non-UI Components (WebServices, APIs etc.)
- Independent of Execution Mechanism
- Manual testing
- Automated testing
- Not theory - proven 7 year history of
implementations - In use in UK and Europe at many organisations for
defining testing
32Test Design Model
33Test Model GUI Example
34A test step
- Object Action Data
- Action
- SET enter data, navigation
- GET retrieve data
- VAL compare expected/actual data
35How tests are designed
- Using Microsoft Excel
- Business processes are modelled
- Sequences of steps and data are built up in
Sub-Test Tables - Sequences of Sub-Tests model business processes
36Interpreting the Model
- There are products that can process the model
- To generate Test Automation Scripts
- For UI Testing using GUI Automation tools
- For Non-UI Testing using Harnesses
- To generate documentation for manual testing
37Interpreting the Model
38Globus A Self Descriptive Application
39What is a Self-Descriptive Application?
- An application that, through an automated
process, can provide a description of its
interfaces
40The Globus User Interface How it works
41How Globus Describes its Interface
42A Sample of Globus Information
APP CUSTOMER_INST FIELD CUST_SNAME TYPE INPUT
LABEL Surname LENGTH 20 SEQ 1 FIELD CUST_GEN T
YPE OPTION LABEL Gender LENGTH 1 SEQ 7
43How this Information was used
- The Description gives us 3 things
- The business names used for the objects
- The properties the test tool required for object
identification - The sequence of the objects in the window
- This could provide us with two things
automatically - The Object Map
- The sequence of steps for our TDM sub-tests
44More on Self-Descriptive Apps
- Requirement on development to name objects in
business terms - Microsoft .NET Applications
- Windows Forms and Web Applications can provide us
with - Object Maps
- Sub-tests for the Test Design Model
- This can drastically reduce the cost of
automation
45Overall - The Test System Design
46Project Phase 2Implementation
47The Implementation
- Technical Implementation
- 9 days of Test Tool Technician
- Using Automatic Interface Description in Globus
- 1800 Objects Mapped automatically
- 90 Sub-test components generated
- 30-75 steps per Sub-test
- Time to Generate 42 seconds
48What was left for the Testers?
- Test Design!
- Designing scenarios for testing the application
- Providing Test Data
- Sequencing Sub-Tests to form business processes
49Designing the Tests
- 4 Testers Using the Test Design Model
- No previous automation experience
- 40 Days Elapsed time
- 1012 Complex Banking Scenarios Automated
50Raw Statistics Test Execution
- Single Script
- Automated 80secs
- Manual 480secs
- Automated is 6x Faster
- Full Regression Pack
- Automated 22 Hrs (1 Workstation)
- 5.5 Hrs (4 Workstations)
- Manual 132 Hrs
- 22 Man/Days (using 6hrs productivity per
day) - 22 Man/Days Testing in 5.5 Hrs.
51Impact Upon You
52Recommendations
- Find out your organisations direction
- Decide upon your own future
- Increased management skills
- Increased non-functional skills
- Increased business skills
- Ensure you understand how Automation Frameworks
support testing so you have a view when asked,
because you will be.!
53Thank You
- Susan Windsor
- WMHL Consulting Limited, MD
- susan_at_wmhl.co.uk
- www.wmhl.co.uk
Case Study provided by Odin Technology Ltd
www.odin.co.uk
Closing slide