Business Case for Test Automation

1
Business Case for Test Automation
Business Case for Test Automation

• S.Janardhanan
• Chief Technology Officer
• ITC Infotech India Limited

2
Agenda

• Test Automation Myths and Reality
• Return on Investments
• Key take outs and summary

3

• Test Automation Myths and Reality

4
Test Automation Myths

• Test Automation is simple, every tester can do it
• This myth is promoted by the tool sales people.
  They are trying to promote the following test
  automation process
• Record the script
• Enhance the script by adding functions and data
  driving
• Run the scripts
• Report results
• Under the influence of this myth a QA manager can
  proudly report All our testers are developing
  test automation.

5
Reality - Test automation is a software
development task

• Automation should be designed, developed and
  tested
• You need to have some kind of a programming
  background to implement test automation. Test
  Automation is not as complex as C/C/Java
  development.
• Test automation standards should be developed
• Automated test components are assets that should
  be treated like application source code, unit
  tested, integration tested and performance
  aspects considered.

6
Myth 2 Commercial test tools are expensive

• Under the influence of this myth some companies,
  especially the small ones
• Try to develop their own test automation tools
• Use scripting languages like Perl and Ruby
• Use shareware test tools
• Do not consider test automation at all

7
Reality Commercial tools are cheap

• Per seat license for most expensive automation
  tool is 8K
• This tool will be used for 5 years.
• Maintenance/Support fees are 20 of tool cost or
  1,800 per year
• The cost of this tool is 8K/51,800 3,100
  per year
• The automation developer cost with overhead is
  100K per year
• The cost of this tool is just 3 of the person
  who uses it, but productivity gain can be very
  significant

8
Commercial Tool Benefits

• Customer support. Many of the open source tools
  come and go with little to no support
• Most commercial tools are constantly being
  updated as technologies change
• Most commercial tools usually have more
  functionality (QTP can test various GUI
  applications Web, .Net, Java, VB, C/C,
  PowerBuilder, etc. vs. WATIR Web only)
• Commercial tools usually have a large community
  of users, which translates into better
  availability of qualified resources
• Commercial tools require less advanced
  programming
• More test automation frameworks are available for
  commercial tools
• Commercial tools are integrated with Test
  Management tools which make reporting and
  execution much simpler.

9
Return On Investment (ROI)
10
Classic ROI Calculation
ROI BENEFIT/COST Automation Cost Price Of HW
Price of SW Development Cost Maintenance
Cost Execution Cost Manual Testing Cost
Development Cost Maintenance Cost Execution
Cost ROI (Manual Testing Cost - Automation
Cost)/Automation Cost Looks right, doesnt it?
11
Problems with Classic ROI Calculation

• You cant compare Automated Testing and Manual
  Testing. They are not the same and they provide
  different information about the AUT.
• You cant compare cost of multiple execution of
  automated tests vs. manual tests. You would never
  dream of executing that many test cases manually.

12
Automation Real ROI

• ROI value IS NOT the value of Automation vs. Cost
  of executing these tests manually
• Automation ROI value IS the benefit of this type
  of testing, and it can be
• Reducing Time to Market / Execution Time
• Increased Test Efficiency (Productivity)
• Increased Test Effectiveness test coverage
• Decrease in test defects escaping to production
• Improved test repeatability

13
Reduced Time to Market

• Can get a greater market share
• Makes people available to work on other projects
• Higher margins, if no competitive products are
  currently available

14
Productivity and Effectiveness

• More testing gets done faster, increasing the
  odds of finding defects
• Defects found early have better chances to be
  fixed
• Manual Testers can concentrate on clever ways to
  finding defects, instead of typing test inputs
  and verify output.
• About 7 of bug fixes create new bugs, sometimes
  in already tested parts of the system. With
  automation you can rerun tests for those modules.
  This almost never happened when testing done
  manually.

15
Manual VS Automated Testing Example

• There is a suite of manual test cases that have
  been deemed as candidates for automation. When
  run manually, the execution can start immediately
  (ITC 0 days). It takes the manual testing team
  seven business days to execute all of the test
  cases (TEC 7 days). The Total Testing Time will
  always be a linear factor of seven days.
• If the test cases are to be automated it will
  take 30 business days to complete (ITC 30
  days). Once the test scripts have been written it
  takes half of a business day to complete the
  tests with the necessary hardware resources (TEC
  0.5 days).

Test Cycle Manual Execution Time (days) Automation Execution Time (days)
1 7 30.5
3 21 31.5
5 35 32.5
7 49 33.5
9 63 34.5
16
ROI summary

• Each project requires different types of
  automation - there is no easy formula available
  to calculate ROI
• Performing ROI calculation can help to determine
  upfront what type of automation, what level of
  skills, what tools will be required.

17
Thank You