Economic Perspectives in Test Automation: Balancing Automated and Manual Testing with Opportunity Cost

1
Economic Perspectives in Test AutomationBalancin
g Automated and Manual Testing with Opportunity
Cost

• Paper By Rudolf Ramler and Klaus Wolfmaier
• Presented By Pallavi Gulati

2
Topics

• Why Automated Testing?
• Simplistic Cost Model for deciding between
  Automated vs. Manual Testing
• Flaws in the Simplistic Cost Model
• New Model based on Opportunity Cost
• Benefits offered by the new model
• Future Plans

3
Why the need to Automate Tests?

• Testing accounts for at least 50 of total
  project cost
• Fast-paced iterative development
• Shrinking schedules and budgets
• Test Automation proposed to reduce testing costs

4
Advantages - Automated Tests

• Reduces overall cost of RUNNING tests
• Though some initial cost required for writing
  automated tests
• Increases the number of tests run
• Increases frequency at which tests are run
• Incremental cost of test running is less
• Major concern in deciding whether to fix a
  particular bug or not

5
Disadvantages - Automated Tests

• Can not automate every test case
  timing-related, appearance, user-friendly, some
  system integration test cases, involving
  different hardware setups
• May not catch new bugs, targeted towards
  regression bugs
• Higher latency as compared to manual testing
  which can start immediately

6
When to go for automation?

• How do managers generally decide when to
  automate?
• Decision is based on factors like
• Efforts required to run manual test
• Frequency of testing
• Cost of third-party / in-house developed tool
• Efforts required to write automated tests

7
Simplistic Cost Model
Vm Estimated Cost of designing manual tests Am
Overall cost of manual testing for some of
times Va Estimated Cost of designing automated
tests (may also include cost of test tool) Aa
Overall cost of automated testing for some of
times
8
Flaws - Simplistic Cost Model

• Manual Automated Testing are not comparable
• Exploring new functionality vs regression testing
• Only costs are analyzed not benefits
• Manual testing starts early in the process but
  some latency in automated testing
• All test cases executions are considered equal
• Project Context (budget, resources) not
  considered
• Misses additional cost factors like training on
  the testing tool, new process adaptation

9
Why automated tests fail to provide expected
benefits?

• Due to flaws in estimation using the Simplistic
  Cost Model and other factors like
• Failure to understand that automated tests are
  targeted for regression bugs
• Gross underestimation of the effort required to
  develop automated tests
• Practically no allocation of efforts for
  maintaining automated tests

10
When to automate? vs. What to automate?

• It does not help to automate every test
• Need to decide trade-off between automated and
  manual testing
• Alternative - Opportunity Cost Model

11
Opportunity Cost Model

• Consider manual automated test cost
• Decide how many manual test cases should be
  automated
• Pick those which will give maximum benefit / risk
  mitigation / coverage
• Project budget deadlines considered
• Looks something like...

12
Opportunity Cost Model

• Va 1 hour to automate 1 test
• 0 min to run 1 automated test
• 0 min to come up with 1 manual test
• Dm 15min to execute 1 manual test

13
Restriction 1
14
Restrictions 2 and 3
15
Target Function
Ra Risk Mitigation for automated testing Rm
Risk Mitigation for manual testing This allows
us to include the benefit obtained by automating
a test
16
Scenarios for Analysis
17
Scenarios for Analysis
18
Model can be extended to include Additional
Influencing Factors

• Tests that rely solely on automation
• e.g. 100 concurrent transactions
• Productivity changes over time
• Causes an outward curved graph
• Growing Test Effort in iterative development
• Improves risk mitigation of automated test
• Maintenance cost
• Early / Late return on investment
• Developing automated test early using data driven
  technique
• Defect detection capability
• random input invoking crash

19
Advantages Opportunity Cost Model

• Draws attention to balancing automated and manual
  testing
• Benefits as well as costs
• Risk mitigation capability of tests
• Budget and deadlines
• More realistic comparison between automated and
  manual testing

20
Future Plans

• Reduce complexity make it more practical to use
• Evaluate preciseness and completeness
• Identify add more influencing factors
• Analyze need to add more restrictions

21
Summary

• Advantages of Automated Test
• Simplistic Cost Model
• Flaws
• Proposed Opportunity Cost Model
• Benefits offered
• Future Plans for expanding the model

22

• Wake up !!!