What Do We Know about Defect Detection Methods

1
What Do We Know about Defect Detection Methods

• P. Runeson et al. "What Do We Know about Defect
  Detection Methods?", IEEE Software, May/June.
  2006 Page(s) 82-90.

2
Summary

• A survey of defect detection studies comparing
  inspection and testing techniques
• Yields practical recommendations
• Use inspections for requirements and design
  defects
• Use testing for code

3
Article Vocab Review

• Evidence-based software engineering
• Determines which methods to use and for what
  purpose
• EBSE involves relevant questions, surveying and
  appraising available evidence
• Integrates and evaluates new practices in a
  target environment
• Used as basis of article to help determine most
  efficient and effective methods for defect
  detection

4
Article Vocab Review (cont)

• Defect
• Relates to one or more underlying faults in an
  artifact such as code
• Maps to single faults (faults and defect are used
  interchangeably)
• Artifact
• Requirements
• Design
• Code
• Using industrial artifacts improves the studies
  generalizabilty

5
Types of Defects

• Grammatical defects are much different from
  missing requirements in a requirements
  specification
• Testing and inspection methods may deem better or
  worse depending on the type of defect
• Omission (missing) vs. Commission (incorrect)
• Unimportant, unimportant, crucial
• Classify defects on origins
• Requirements
• Design
• Code

6
Actors Technique

• Who is the reviewer or tester
• Freshman student in an experiment
• Experienced software engineer
• Technique
• Relative to inspection and testing
• Inspection families of verification techniques
• Testing structural (white box) functional
  (black box)

7
Purpose

• What's the purpose of the inspection and testing
  activity?
• Validation assuring the correct system is
  developed
• Verification assuring the system meets its
  specifications
• Both?
• Detection (for later isolation) vs. Isolation
  (isolate the underlying fault)

8
Defect detection activities

• Consider defect origins
• Primary defect detection (1)
• Secondary defect detection (2)

9
Evaluation criteria

• Efficiency - of defects found per time unit
  spent on verification
• Effectiveness the share of the existing defects
  found

10
The Survey (finally)!!!!!
11
Data from the studies
12
The issues covered from Table 23

• Requirements defects
• Design defects
• Code defects
• Different defect types
• Efficiency versus effectiveness

13
Requirements defects Thank god for the experts!

• Different requirements inspection methods same
  conclusion
• Spending effort up front to establish a good set
  of requirements is more efficient than developing
  a system on basis of incorrect requirements and
  then reworking it

14
Design defects

• Is inspecting design documents or testing the
  implemented function more efficient?
• Inspections were significantly more effective and
  efficient than testing
• More than half of the defects (53.5 ) found
  during inspection
• Less found during testing (41.8 )
• Bottom line design inspection is the winner!

15
Code defects

• Which is better? Code inspection or testing
  (functional or structural)?
• Which is more effective?
• Which is more efficient?
• No clear winner
• Well see why next slide

16
Average effectiveness for code defect detection
17
Average efficiency for code defect detection
18
Survey issues

• Different defect types
• Absence or presence of different types of defects
  might affect efficiency and effectiveness
• Studies report defects by differing types and
  cannot be used to investigate whether the
  differences depend on fault type, not only the
  technique
• Frequency of defects vary widely
• Only a fraction of defects were found among
  studies affects defect type frequency
• Classification schemes involve subjective
  judgment
• Conclusion jury is out as to the effect of
  defect types on the performance of inspection
  versus testing

19
Survey issues (cont)

• Effectiveness and efficiency
• Only 25 to 50 percent of defects found during
  inspection
• 30 to 60 percent found during testing
• More than HALF defects remain!!
• Conclusion Primary defect detection methods low
  effectiveness and efficiency values is that
  secondary detection methods play a larger role
  that the surveys conclude

20
The results are in!

• For requirements defects, no empirical evidence
  exists at all, but the fact that costs for
  requirements inspections are low compared to
  implementing incorrect requirements indicates
  that reviewers should look for requirements
  defects through inspection.

21
The results are in!

• For design specification defects, the case
  studies and one experiment indicate that
  inspections are both more efficient and more
  effective than functional testing.

22
The results are in!

• For code, functional or structural testing is
  ranked more effective or efficient than
  inspection in most studies. Some studies conclude
  that testing and inspection find different kinds
  of defects, so theyre complementary. Results
  differ when studying fault isolation and not just
  defect detection.

23
The results are in!

• Verifications effectiveness is low reviewers
  find only 25 to 50 percent of an artifacts
  defects using inspection, and testers find 30 to
  60 percent using testing. This makes secondary
  defect detection important. The efficiency is in
  the magnitude of 1 to 2.5 defects per hour spent
  on inspection or testing.

24
Results Summary Table
25
Questions ???