Using Metrics to Reduce Cost of Re-work

1
Using Metrics to Reduce Cost of Re-work

• Dwight Lamppert
• Senior Test Manager
• Franklin Templeton

2
Opening Question

• What is the job of testing?

3
Potential Response

• The job of testing is to find defects.

4
Myth Exposed

• Myth The Job of Testing Is to Find Defects1
• 1The Seven Principles of Lean Software
  Development by Mary Poppendieck

5
Quality is everyones responsibility

• Myth The Job of Testing Is to Find Defects
• The job of tests, and the people that develop
  and run tests, is to prevent defects, not find
  them. A quality assurance organization should
  champion processes that build quality into the
  code from the start rather than test quality in
  later.1
• 1The Seven Principles of Lean Software
  Development by Mary Poppendieck

6
At a recent software seminar

• The common attributes of successful software
  projects are
• Heroes
• Chaos
• Re-work
• Untracked OT

7
Todays Discussion

• Case Study 4 projects
• 2 Definitions
• Testing Metrics (what to collect analyze)
• Metrics from first 2 projects
• Process Improvement Steps
• Success Story Projects 3 4
• Some trends conclusions

8
Project Background

• Legacy mature system (mainframe)
• Industry regulatory enhancements
• Time-boxed 3 to 4 months
• Waterfall SDLC
• First 2 Projects
• Project 1 Dec09 1X
• Project 2 June10 2.3X

9
Definition Re-work

• Re-work AD time to fix and testing time to
  report and re-test System Testing and Production
  defects

10
Definition Static Testing

• Static Testing - Testing of a component or system
  at specification or implementation level without
  execution of that software, e.g., reviews or
  static code analysis.

11
Static vs. Dynamic Testing
12
Testing Defect Analysis

• When was the defect introduced (SDLC phase)
• When was the defect discovered (testing phase)

13
Points of Origin for Defects2

• Requirements
• Design
• Code
• User documents
• Bad fixes (new defects due to repairs of older
  defects)
• 2Software Engineering Best Practices Lessons
  from Successful Projects in the Top Companies by
  Capers Jones

14
Defect Discovery Phases

• Static Testing
• Document walk-throughs reviews
• Pre-testing
• Test Plan review
• All before Build is complete
• System Testing
• Production

15
Tracking the Cost of Re-work

• Defect Fix Time
• Re-test Time
• COR (Cost of Re-work) calculated by multiplying
  time x rate cost per defect

16
Testing Rework Workflow

• Re-work time during Test Execution is tracked as
  2 Components
• Time to Fix a Defect
• Time to Re-test Defect Fixes

17
Project 1 - Defect Fix Cost
18
Issues from first 2 projects

• High of project Cost of Re-work

19
COR 23 to 29 Project
20
Issues from first 2 projects

• High of project Cost of Re-work
• High of Requirements Design (RD) defects
  found later

21
20-30 RD Defects found later
22
Issues from first 2 projects

• High of project Cost of Re-work
• High of Requirements Design (RD) defects
  found later
• DDP (Defect Discovery ) slightly better than
  average

23
Defect Discovery Percentage (DDP)

• DDP of Total Defects discovered before
  implementation
• Provides a metric to evaluate the overall
  effectiveness of testing.
• 90 - Good
• 95 - Very Good

24
DDP average
25
Process Improvement Steps

• Developed framework in Quality Center to track
  COR
• Presentations to business partners executives
• Lively discussion during presentations
• AD appointed a new AD Lead
• AD improved their internal code review
• More emphasis on and participation during
  reviews, walk-throughs Static Testing

26
Process Improvement Steps

• More collaboration between AD and testing teams
• Increased emphasis on unit testing
• Risk-based testing approach
• Improved regression work
• Staff increased in domain knowledge

27
Projects 3 4 Process Improved

• Project 1 Dec09 1x
• Project 2 June10 2.3x
• Project 3 May11 3.4x
• Project 4 June11 2.3x

28
Static Testing emphasized
29
System Testing Defects
30
RD Defects found later
31
DDP (Defect Discovery )
32
COR as Project Cost
33
Defect Discovery Rate

• DDR of Total Defects discovered during an
  SDLC or Test Phase
• Static Testing
• System Testing
• Transition
• Goal - Trying to find out if we are finding
  defects earlier.

34
DDR Trends
As DDR for Static Testing increases, DDR for
System Testing and Production Defects decreases
----------------Coding-----------------
35
When is Unit Testing complete?

• A necessary criterion for completion of
  developer testing is that the developers believe
  that the software is ready to be delivered to the
  users on the day it is delivered to the
  system-test team.
• From System Testing with an Attitude, p. 33, by
  Nathan Petschenik

36
Todays Discussion

• Defect metrics powerful story
• Static Testing works
• It is a collaborative effort
• The positive effect ripples into Unit Testing,
  System Testing, UAT, and implementation
• No shortcuts here influence your project to do
  it right

37
Some Final Thoughts

• Training needed for our staff (testing, AD,
  business, PM community) - how to be more
  effective in document reviews and walk-throughs.
• Perspectives
• Tester static testing gt a paradise (a new
  maturity level)

38
Questions Answers

• ???