Introduction to Software Inspection EE599 Software V - PowerPoint PPT Presentation


PPT – Introduction to Software Inspection EE599 Software V PowerPoint presentation | free to view - id: 21d17-YzhiY


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation

Introduction to Software Inspection EE599 Software V


according to work done by Glen Russell at BNR (now Nortel Technologies) in 1991, ... 15 Kent Beck, Extreme Programming Explained: Culture Change, Addison Wesley, 1999 ... – PowerPoint PPT presentation

Number of Views:401
Avg rating:3.0/5.0
Slides: 47
Provided by: dian183


Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Introduction to Software Inspection EE599 Software V

Introduction to Software Inspection EE599
Software VV Winter 2006
  • Diane Kelly and Terry Shepard
  • Royal Military College of Canada
  • Electrical and Computer Engineering
  • ext. 6031

  • Rationale for inspection
  • Fagan style inspections
  • What makes inspections hard
  • New approaches
  • Mechanics of inspections
  • process
  • techniques
  • Experiments with TDI at RMC

Rationale for Inspections
  • The original argument payback on cost of fixes
    (e.g. debugging) leading to improved productivity
  • according to work done by Glen Russell at BNR
    (now Nortel Technologies) in 1991, 65 to 90 of
    operational (execution-time) defects are detected
    by inspection at 1/4 to 2/3 the cost of testing
  • Other arguments for inspection
  • better control of process
  • higher quality
  • reduced defect rates
  • reduced cost of finding defects
  • payback on fixing soft issues (Votta 3)

Industry Standard Inspection
  • First described by Michael Fagan 1976 30
  • subsequent paper in 1986 31
  • Fagan defines inspection
  • defines a process
  • defines roles for the participants
  • suggests feedback to improve the development
  • Fagan approach to inspections has become an
    industry standard
  • Several variations
  • eg. Gilb and Graham - next slide

Inspection process from Gilb and Graham -
variation on Fagan
Fagan-style inspections
  • Entry and exit criteria
  • Preparation
  • Moderator
  • Meeting 3-6 people
  • Controlled rate typically 100 to 200 loc/hr
  • rates that are too fast or too slow reduce
  • Controlled meeting length max 2 hours
  • Controlled number of meetings per day max 2
  • Track and report defects found, time spent
  • rate of finding defects per hour, rate per kloc
  • Classify defects

What makes inspections hard?
  • Mentally demanding
  • Specific skills needed may be in short supply
  • Role of moderator requires special people skills
  • meeting dynamics
  • possible tension between inspectors and product
  • Time pressures
  • squeezed by tight schedules
  • inspection interval problem
  • Attitudes
  • perceived as uninteresting work
  • not glamorous, support role only
  • Lack of ownership
  • who owns the inspection?
  • Unclear goals, multiple goals

New Approaches
  • Research and industry practice suggest variations
    on …
  • goals
  • meetings
  • techniques
  • size of teams
  • entry criteria
  • exit criteria
  • when to inspect
  • activities to include/exclude
  • ...
  • There are better approaches than the commonly
    used Fagan method. 25

Mechanics Inspect for Specific Goals - Examples
  • Set an inspection goal to identify usability
  • Ask questions such as
  • What are these identifiers used for?
  • What is the default choice in these switch
  • Have a numerical techniques expert check one
    module that does a matrix inversion
  • Have a junior programmer check the entire product
    for meaningful identifiers
  • Have a meeting focused solely on consistency
    issues of the user interface

Mechanics Inspect for Specific Goals Particular
Approaches 6,7
  • Set objectives of inspection ahead
  • Inspector skills are matched to specific aspects
    of the inspection
  • Each inspector focuses on a specific purpose
  • inspectors actively work with the document
  • questionnaires dont have passive questions
    requiring simple yes/no answers
  • Inspectors may either
  • examine specific modules -or-
  • examine entire product for one property
  • Small meetings focused on specific aspects
  • meetings may not raise new findings

Mechanics Types of Inspectors 6
  • Suggestions for finding good inspectors
  • specialists
  • application area
  • hardware
  • operating system
  • …
  • potential users of system
  • those familiar with design methodology used
  • those who enjoy finding logical inconsistencies
    and are skilled at doing so

Mechanics what to inspect
  • All software artifacts can be inspected
  • In some cases, code inspections can be replaced
    by design, architecture or requirements
  • In other cases, code inspections can replace unit
  • Inspection and testing are complementary
  • Architecture and design inspections have high
    payback (XP challenges this current debate)
  • Start small even one page can contain 50 or more
    defects (eg. early requirements inspection)

Mechanics when to inspect (1)
  • Inspection is possible at any time
  • Based on your goals
  • Fit inspections into your existing process
  • even an immature organization can implement an
    effective inspection program (eg. 24)
  • Dont allow the inspection to become a bottleneck
  • do what you can in parallel with other activities
  • Look at what resources are available

Mechanics when to inspect (2)
  • Determine the criteria
  • entry criteria
  • code after clean compile, before testing?
  • documents after spell checking, after syntax
  • anything can I comprehend what youve given me?
  • re-inspection criteria
  • more than 5 errors per page?
  • exit criteria
  • all findings addressed?
  • inspection tasks completed?
  • metrics at acceptable levels?

Mechanics tools
  • Comments about tools here are focused on code
  • use tools to make inspection more productive
  • reduce tedious trivia burden
  • for code clean up, restructuring
  • enforce code styles
  • e.g. enforce comment style consistency
  • some style errors slip past
  • automatic generation of documentation
  • e.g. call trees, flowcharts, UML diagrams, ...
  • find some syntactic errors
  • tools cannot find errors that result in correct
  • tools can get confused in complex code
  • report false positives
  • miss real issues
  • still need someone to read the code
  • asynchronous inspections virtual meetings
  • can reduce inspection intervals, but no synergy
  • current research on Computer Supported
    Collaborative Work (CSCW) may improve inspections

Does Every Inspection Need a Fagan-style Meeting?
  • Report on an experiment Votta 5
  • investigates the need for large inspection
  • two-person meetings are enough?
  • Report on an experiment Porter Votta 26
  • collection meetings produce no net improvement
    in terms of meeting losses and meeting gains
  • Report on an experiment Johnson Tjanjono 17
  • shows that the number of defects found does not
    increase as a result of meetings, but that
    meetings help to detect false positives

Is there a case for more than one meeting?
  • An Experiment to Assess the Cost-Benefits of Code
    Inspections in Large Scale Software Development
    (Porter, Siy, Toman, Votta 8)
  • varied the the number of reviewers, the number of
    teams inspecting the code unit, and the
    requirement of fixing defects between first and
    second teams inspections
  • two teams per inspection with fix was dropped
    part way through experiment as infeasible
  • results on two teams in parallel are
    inconclusive, where the total number of people is
    the same (eg. 2 teams of 1 vs. 1 team of 2)
  • one of the conclusions
  • structural changes to the inspection process do
    not always have the intended effect significant
    improvements to the inspection process will
    depend on the development of defect detection

Unstructured Techniques (1)
  • Ad-hoc is common
  • often based on paraphrasing
  • no focus on particular issues
  • no guidance on how to consider the product
  • no explicit use of inspectors expertise
  • not repeatable
  • no record of inspectors thought processes and
  • does the beginning of the product get more
    attention than the end?
  • are conceptually difficult parts ignored?

Unstructured Techniques (2)
  • Checklists are popular but have several
    shortcomings 2, 9
  • checklists are based on past experience
  • some categories will not have been found yet
  • depends on inspectors using individual expertise
    to find issues in missing categories
  • inspectors may not look for anything beyond what
    is in checklist
  • checklists may be lengthy and hard to use
  • checklist categories may not be effective or
  • categories need to be prioritized
  • some categories may not provide enough guidance
  • checklist for one project may not work on another
  • some checklists should be personalized
  • checklist effectiveness needs to be evaluated
  • constant improvement needed

Unstructured Techniques (3)
  • Checklists are effective when 12, 33
  • well established history of the product exists
  • product is predictable
  • inspection goals are accompanied by a well
    documented method of operationalization
  • eg. style manual, standards
  • providing a source of accumulated wisdom to a
    junior member of the team
  • an inspector is working independently and
  • the inspector is not experienced and
  • the checklist provides a relatively complete tour
    of the product under inspection

Overview of Structured Inspection Techniques
(University of Maryland 10)
  • Systematic
  • inspector knows how to inspect the document
  • Specific roles
  • inspector is responsible for a specific role with
    a specific focus
  • Distinct
  • each role is distinct in the sense of minimizing
  • coverage of different defect classes is achieved
    by having multiple roles

Scenario Based Reading
  • A technique that is used individually in order
    to analyze a product or a set of products. Some
    concrete instructions are given to the reader on
    how to read or what to look for in a document.
    (University of Maryland, Notes on Perspective
    based Scenarios 10)
  • defect-based and perspective-based scenarios
    (Porter, Votta, Basili 11)
  • focus on defect classes and use questionnaire to
    guide inspectors
  • focus on a role (perspective) and use
    questionnaire to guide inspectors (may also
    produce a product associated with that role)
  • can prepare a scenario for any inspection

Task Directed Inspection (TDI)
  • What is TDI?
  • inspectors produce a usable product for future
    evolution of the software system
  • inspectors expertise is matched to the task
  • used with a light-weight inspection process
  • focus on the work of the individual rather than
    that of the team
  • based on observation that creating careful
    documentation often shows up defects in product
    being documented

Experiments with TDI at RMC
  • Experiments in 2000 to 2002
  • Purpose of the experiments
  • Concerns to be addressed in experimentation
  • Elements of the experiments
  • Design of the experiments
  • Results and Conclusions
  • Experiments in 2003/2004

Purpose of TDI Experiments at RMC (2000 to 2002)
  • evaluate the effectiveness of TDI
  • can it guide the inspector to identify subtle
    defects in the work product?
  • different levels of understanding involved in
    finding defects
  • low level style issues
  • high level logic errors in a complex calculation
  • compare use of TDI to an ad hoc technique
  • observe the use of TDI in a situation very
    different from where it was first developed

Concerns to be Addressed by Design of
Experiment (Confounding factors)
  • natural variation in participants performance
  • maturation of subjects as experiment progresses
  • one detection technique biasing the use of
    another detection technique
  • variations in the instrument, i.e., the pieces of
    code selected for inspection
  • outside influences

Elements of the Experiment (2000 to 2002)
  • inspection techniques
  • paraphrasing
  • TDI - method description
  • TDI - test plan
  • code to be inspected
  • CAVMLC - civil engineering code
  • three pieces
  • equivalent length
  • equivalent complexity
  • equivalent application content
  • participants
  • experimental design

Design of Experiment (2000)
Results from Experiment 2000 - Report in CASCON
Proceedings 2000
Results from Experiment 2000 - Report in CASCON
Proceedings 2000
Analysis of Results for Experiments 2000 to 2002
  • for the experienced students
  • using TDI resulted in proportionally more
    findings in the Logical category
  • using TDI didnt make any difference in the
    proportion of findings in the Comparative
  • for the inexperienced students
  • using TDI may have helped increase the proportion
    of findings in the Comparative category
  • using TDI seemed to hinder identification of
    findings in the Logical category

Conclusions from all three Experiments
  • TDI provides positive results in level of
    understanding of the code for experienced
  • tasks must be appropriate for the inspectors
    background and experience
  • there is a strong interaction between inspection
    technique and inspectors preferred style of

Experiment 2003/2004
  • Assumption
  • TDI works!
  • Previous focus
  • work of individual
  • no effects of group meetings considered
  • Purpose for 2003/2004
  • what effects do meetings have when using TDI?

Elements of the Experiment 2003/2004
  • inspection technique
  • TDI - high level design
  • code to be inspected
  • chemical engineering - radiation dose on air crew
  • two pieces of code
  • content of both is computational based
  • participants
  • experimental design
  • partial factorial
  • repeated measures
  • individual versus groups
  • warm up in first week

References 1
  • Phillip M. Johnson, Reengineering Inspection,
    Communications of the ACM, Feb. 1998, Vol 41, No
    2, pp 49-52
  • Oliver Laitenberger, Khaled El Emam, Thomas
    Harbich, An Internally Replicated
    Quasi-Experimental Comparison of Checklist and
    Perspective-Base Reading of Code Documents, IEEE
    Transactions in Software Engineering, May 2001,
    Vol. 27, No. 5, pp.387-421
  • Lawrence G. Votta, Does the Modern Code
    Inspection Have Value? Presentation at the NRC
    Seminar on Measuring Success Empirical Studies
    of Software Engineering March 1999

References 2
  • Tom Gilb and Dorothy Graham, Software Inspection,
    Addison Wesley, 1993. see also http//www.result-p
  • Votta, Lawrence G. Does Every Inspection Need a
    Meeting?, SIGSOFT93 - Proceedings of 1st ACM
    SIGSOFT Symposium on Software Development
    Engineering, ACM Press, New York, 1993, pp
  • Parnas, David L., David M. Weiss Active Design
    Reviews Principles and Practice, Proceedings
    8th International Conference on Software
    Engineering, Aug. 1985

References 3
  • Knight, J.C., E.A. Myers An Improved Inspection
    Technique, Communications of the ACM, Nov. 1993,
    Vol 36, No 11, pp. 51-61
  • Porter, Adam A., Harvey P. Siy, Carol A. Toman,
    Lawrence G. Votta An experiment to Assess the
    Cost-Benefits of Code Inspctions in Large Scale
    Software Development, IEEE Transactions on
    Software Engineering, Vol 23, No 6, June 1997, pp
  • Chernak, Y. A Statistical Approach to the
    Inspection Checklist Formal Synthesis and
    Improvement IEEE Transactions on Software
    Engineering, 22(12)866-874, Dec. 1996

References 4
  • 10 University of Maryland, Notes on Perspective
    based Scenarios online http//
    Nov. 1999
  • 11 Porter, Adam A., Lawrence G. Votta, Victor R.
    Basili Comparing Detection Methods for Software
    Requirements Inspections A Replicated
    Experiment, IEEE Transactions on Software
    Engineering, Vol 21, No 6, June 1995, pp 563-575
  • 12 Kelly, Diane and Terry Shepard Task-Directed
    Software Inspection Technique An Experiment and
    Case Study, Proceedings IBM CASCON, Nov. 2000
  • 13 Glen W. Russell, "Experience with Inspection
    in Ultralarge-Scale Developments", IEEE Software,
    Jan. 1991, pp.25-31

References 5
  • 14 R. Chillarege, et al., "Orthogonal Defect
    Classification - A Concept for In-Process
    Management", IEEE Transactions on Software
    Engineering, v. 18 n.11, Nov 92, pp. 943-956
  • 15 Kent Beck, Extreme Programming Explained
    Culture Change, Addison Wesley, 1999
  • 16 Robert B. Grady, Successful Software Process
    Improvement, Prentice Hall, 1997
  • 17 Philip M. Johnson and Danu Tjahjono, Does
    Every Inspection Really Need A Meeting?,
    Journal of Empirical Software Engineering, 4, 1,
    pp 9-35, Jan. 1998

References 6
  • 18 David A. Wheeler, Bill Brykczynski, and
    Reginald N. Meeson Jr., Software Inspection An
    Industry Best Practice, IEEE CS Press, 1996
  • 19 Gregory Abowd, Len Bass, Paul Clement, Rick
    Kazman, Linda Northrop, Amy Zaremski
    Recommended Best Industrial Practice for
    Software Architecture Evaluation, Technical
    Report, CMU/SEI-96-TR-025, January 1997
  • 20 Stan Rifkin, Lionel Deimel Program
    Comprehension Techniques Improve Software
    Inspections A Case Study, Proceedings IWPC 00,
    IEEE 2000
  • 21 Steve McConnell, Quantifying Soft Factors,
    IEEE Software, Nov/Dec 2000, pp 9-11

References 7
  • 22 Terry Shepard, Margaret Lamb, and Diane
    Kelly, More Testing Should be Taught,
    Communications of the ACM, June 2001, 44 (6), pp.
  • 23 Michael Cusumano, Richard Selby, Microsoft
    Secrets, Simon Shuster Inc., 1995
  • 24 Edward Kit, Software Testing in the Real
    World - improving the process, Addison-Wesley,
  • 25 Robert L. Glass, Inspections - Some
    Surprising Findings, Communications of the ACM,
    April 1999, Vol. 42, no. 4, pp.17-19

References 8
  • 26 Adam Porter, Lawrence Votta, Comparing
    Detection Methods for Software Requirements
    Inspections A Replication Using Professional
    Subjects, Empirical Software Engineering
    Journal, 1997
  • 27 Victor Basili, Scott Green Oliver
    Laitenberger, Filippo Lanubile, Forrest Shull,
    Sivert Sorumgard, Marvin Zelkowitz, The
    Empirical Investigation of Perspective-Based
    Reading, Empirical Software Engineering An
    International Journal, 1(2), 1996, pp.133-164
  • 28 Oliver Laitenberger, Thomas Bell, An
    Industrial Case Study to examine a
    non-traditional Inspection Implementation for
    Requirements Specifications, Proceedings 8th
    IEEE Symposium on Software Metrics, June 2002,

References 9
  • 29 Michael Fredericks, Victor Basili, Using
    Defect Tracking and Analysis to Improve Software
    Quality A DACS State-of-the-Art Report, Rome
    NY, 1998
  • 30 M.E. Fagan, Design and Code Inspections to
    reduce Errors in Program development, IBM
    Systems Journal, Vol. 15, No. 3, 1976, pp.182-211
  • 31 M.E. Fagan, Advances in Software
    Inspections, IEEE Transactions on Software
    Engineering, Vol.12, No. 7, July 1986,
  • 32 Watts S. Humphrey, A Discipline for Software
    Engineering, Addison Wesley, 1995

References 10
  • 33 Stefan Biffl, Michael Halling, Investigating
    the Influence of Inspector Capability Factors
    with Four Inspection Techniques on Inspection
    Performance, Eight IEEE Symposium on Software
    Metrics, 2002, pp. 107-117
  • 34 Grady, Robert B. (1996). Software Failure
    Analysis for High-Return Process Improvement
    Decisions. Hewlett-Packard Journal, 47(4)
  • 35 Boris Beizer, Software Testing Techniques,
    2nd ed., Van Nostrand Reinhold, NY, 1990
  • 36 http//

References 11
  • 37 Robert Grady, Deborah Caswell Software
    Metrics Establishing a Company-Wide Program,
    Prentice-Hall, 1986
  • 38 IEEE Std 1044 Classification for Software
  • 39 David Card, Learning from our Mistakes with
    Defect Causal Analysis, IEEE Software Jan-Feb
    1998, pp.56-63
  • 40 http//
    HTM and FAQ.HTMconcepts

References 12
  • 41 Barry Boehm, Victor Basili Software Defect
    Reduction Top 10 List IEEE Computer, January
    2001, pp.135-137
  • 42 Ram Chillarege, Inderpal Bhandari, Jarir
    Chaar, Michael Halliday, Diane Moebus, Bonnie
    Ray, Man-Yuen Wong Orthogonal Defect
    Classification - A Concept for In-Process
    Measurements IEEE Transactions on Software
    Engineering, Vol.18, No.11, Nov. 1992, pp.943-956
  • 43 Jarir Chaar, Michael Halliday, Inderpal
    Bhandari, Ram Chillarege In-Process Evaluation
    for Software Inspection and Test, IEEE
    Transactions on Software Engineering, Vol.19,
    No.11, Nov. 1993, pp.1055-1070