Estimating the number of components with defects post-release that showed no defects in testing

1
Estimating the number of components with defects
post-release that showed no defects in
testing C. Stringfellow A. Andrews C. Wohlin H.
Peterson Jeremy Mange
2
Motivation

• For nearly any product, defects will appear after
  release
• The defects that were not detected in testing
  point to problems in the test process
• We would like to know how many of these
  post-release defects to expect

3
Motivation

• Using these estimates, release / re-test
  decisions can be made
• The authors wish to compare methods of
  post-release defect estimation

4
Defect Estimation

• The paper compares three methods of this type
  of estimation
• Experience-based
• Capture-recapture
• Curve-fitting
• Experience-based requires historical data,
  Capture-Recapture and Curve-fitting do not

5
Experience-based Estimation

• Requires some sort of historical data
• Defect data
• Change data
• Historical data can be from previous products or
  prior releases of the same product
• If from previous products, they must be similar
  for these estimations to work

6
Experience-based Estimation

• Defect data
• Number of faults in a module can be estimated
  using the defect history of that module
• Change data
• Number of faults in a module can be estimated
  using the number of file changes
• Recent changes can be weighted more heavily

7
Capture-recapture Estimation

• A type of model originally designed for reviews
  and inspections
• Compares the number and types of defects found
  from multiple test sites
• Applies statistical estimation to the data based
  on assumptions about test sites

8
Capture-recapture Estimation

• Differences between test sites are accounted for
  in two areas
• Ability to find defects
• Probabilities of finding specific defects
• This yields four models
• Same ability, same probabilities
• Same ability, different probabilities
• Different ability, same probabilities
• Different ability, different probabilities

9
Capture-recapture Estimation
10
Curve-fitting Estimation

• Fits a mathematical curve to the data points to
  estimated remaining defects
• Two basic types
• Decreasing plot number of test sites that found
  each defect, sorted in decreasing order
• Increasing plot cumulative number of defects
  found by each testing event

11
Curve-fitting Estimation

• With both increasing and decreasing models, the
  fitted curve is used to predict the number of
  remaining defects
• Both exponential and linear prediction models
  exist for each type

12
Approach

• 1. Collect the data
• For each discovered defect, count the number of
  test sites that discovered it

13
Approach

• 2. Apply non-historical methods
• Both capture-recapture and curve-fitting models
  are applied
• These provide estimates of the number of total
  defects

14
Approach

• 3. Apply experience-based method
• Use historical data to estimate the number of
  post-release defects
• For each past project (or release), calculate

15
Approach

• 4. Using the calculations, estimate post-
  release defects
• For non-historical models, subtract the number of
  defects found in testing from the calculated
  total defects estimate

16
Approach

• 5. Compare this estimate to a decision
  threshold value
• If the number of expected post-release defects is
  acceptable, the product can be released
• If not, further testing must be performed
• Of course, the actual release decision will be
  based on many criteria, this is just one of those

17
Results

• This approach was carried out in a case study on
  a medical record system
• Results
• Capture-recapture and curve-fitting 5-20
  error
• Experience-based3-5 error

18
Conclusion

• If data for similar projects is available,
  experience-based models provide the best
  estimations for post-release defects
• However, non-historical (capture-recapture and
  curve-fitting) models also provide fairly
  accurate estimates independent of past data
• Post-release defect estimates should be used in
  release decisions

19
Questions?