Introduction to Software Inspection EE599 Software V - PowerPoint PPT Presentation

Loading...

PPT – Introduction to Software Inspection EE599 Software V PowerPoint presentation | free to view - id: 21d17-YzhiY



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

Introduction to Software Inspection EE599 Software V

Description:

according to work done by Glen Russell at BNR (now Nortel Technologies) in 1991, ... 15 Kent Beck, Extreme Programming Explained: Culture Change, Addison Wesley, 1999 ... – PowerPoint PPT presentation

Number of Views:401
Avg rating:3.0/5.0
Slides: 47
Provided by: dian183
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Introduction to Software Inspection EE599 Software V


1
Introduction to Software Inspection EE599
Software VV Winter 2006
  • Diane Kelly and Terry Shepard
  • Royal Military College of Canada
  • Electrical and Computer Engineering
  • kelly-d_at_rmc.ca
  • ext. 6031

2
Outline
  • Rationale for inspection
  • Fagan style inspections
  • What makes inspections hard
  • New approaches
  • Mechanics of inspections
  • process
  • techniques
  • Experiments with TDI at RMC

3
Rationale for Inspections
  • The original argument payback on cost of fixes
    (e.g. debugging) leading to improved productivity
  • according to work done by Glen Russell at BNR
    (now Nortel Technologies) in 1991, 65 to 90 of
    operational (execution-time) defects are detected
    by inspection at 1/4 to 2/3 the cost of testing
    13
  • Other arguments for inspection
  • better control of process
  • higher quality
  • reduced defect rates
  • reduced cost of finding defects
  • payback on fixing soft issues (Votta 3)

4
Industry Standard Inspection
  • First described by Michael Fagan 1976 30
  • subsequent paper in 1986 31
  • Fagan defines inspection
  • defines a process
  • defines roles for the participants
  • suggests feedback to improve the development
    process
  • Fagan approach to inspections has become an
    industry standard
  • Several variations
  • eg. Gilb and Graham - next slide

5
Inspection process from Gilb and Graham -
variation on Fagan
6
Fagan-style inspections
  • Entry and exit criteria
  • Preparation
  • Moderator
  • Meeting 3-6 people
  • Controlled rate typically 100 to 200 loc/hr
  • rates that are too fast or too slow reduce
    effectiveness
  • Controlled meeting length max 2 hours
  • Controlled number of meetings per day max 2
  • Track and report defects found, time spent
  • rate of finding defects per hour, rate per kloc
  • Classify defects

7
What makes inspections hard?
  • Mentally demanding
  • Specific skills needed may be in short supply
  • Role of moderator requires special people skills
  • meeting dynamics
  • possible tension between inspectors and product
    author
  • Time pressures
  • squeezed by tight schedules
  • inspection interval problem
  • Attitudes
  • perceived as uninteresting work
  • not glamorous, support role only
  • Lack of ownership
  • who owns the inspection?
  • Unclear goals, multiple goals

8
New Approaches
  • Research and industry practice suggest variations
    on …
  • goals
  • meetings
  • techniques
  • size of teams
  • entry criteria
  • exit criteria
  • when to inspect
  • activities to include/exclude
  • ...
  • There are better approaches than the commonly
    used Fagan method. 25

9
Mechanics Inspect for Specific Goals - Examples
  • Set an inspection goal to identify usability
    issues
  • Ask questions such as
  • What are these identifiers used for?
  • What is the default choice in these switch
    statements?
  • Have a numerical techniques expert check one
    module that does a matrix inversion
  • Have a junior programmer check the entire product
    for meaningful identifiers
  • Have a meeting focused solely on consistency
    issues of the user interface

10
Mechanics Inspect for Specific Goals Particular
Approaches 6,7
  • Set objectives of inspection ahead
  • Inspector skills are matched to specific aspects
    of the inspection
  • Each inspector focuses on a specific purpose
  • inspectors actively work with the document
  • questionnaires dont have passive questions
    requiring simple yes/no answers
  • Inspectors may either
  • examine specific modules -or-
  • examine entire product for one property
  • Small meetings focused on specific aspects
  • meetings may not raise new findings

11
Mechanics Types of Inspectors 6
  • Suggestions for finding good inspectors
  • specialists
  • application area
  • hardware
  • operating system
  • …
  • potential users of system
  • those familiar with design methodology used
  • those who enjoy finding logical inconsistencies
    and are skilled at doing so

12
Mechanics what to inspect
  • All software artifacts can be inspected
  • In some cases, code inspections can be replaced
    by design, architecture or requirements
    inspections
  • In other cases, code inspections can replace unit
    testing
  • Inspection and testing are complementary
    activities
  • Architecture and design inspections have high
    payback (XP challenges this current debate)
  • Start small even one page can contain 50 or more
    defects (eg. early requirements inspection)

13
Mechanics when to inspect (1)
  • Inspection is possible at any time
  • Based on your goals
  • Fit inspections into your existing process
  • even an immature organization can implement an
    effective inspection program (eg. 24)
  • Dont allow the inspection to become a bottleneck
  • do what you can in parallel with other activities
  • Look at what resources are available

14
Mechanics when to inspect (2)
  • Determine the criteria
  • entry criteria
  • code after clean compile, before testing?
  • documents after spell checking, after syntax
    checking?
  • anything can I comprehend what youve given me?
  • re-inspection criteria
  • more than 5 errors per page?
  • exit criteria
  • all findings addressed?
  • inspection tasks completed?
  • metrics at acceptable levels?

15
Mechanics tools
  • Comments about tools here are focused on code
  • use tools to make inspection more productive
  • reduce tedious trivia burden
  • for code clean up, restructuring
  • enforce code styles
  • e.g. enforce comment style consistency
  • some style errors slip past
  • automatic generation of documentation
  • e.g. call trees, flowcharts, UML diagrams, ...
  • find some syntactic errors
  • tools cannot find errors that result in correct
    syntax
  • tools can get confused in complex code
  • report false positives
  • miss real issues
  • still need someone to read the code
  • asynchronous inspections virtual meetings
  • can reduce inspection intervals, but no synergy
  • current research on Computer Supported
    Collaborative Work (CSCW) may improve inspections

16
Does Every Inspection Need a Fagan-style Meeting?
  • Report on an experiment Votta 5
  • investigates the need for large inspection
    meetings
  • two-person meetings are enough?
  • Report on an experiment Porter Votta 26
  • collection meetings produce no net improvement
    in terms of meeting losses and meeting gains
  • Report on an experiment Johnson Tjanjono 17
  • shows that the number of defects found does not
    increase as a result of meetings, but that
    meetings help to detect false positives

17
Is there a case for more than one meeting?
  • An Experiment to Assess the Cost-Benefits of Code
    Inspections in Large Scale Software Development
    (Porter, Siy, Toman, Votta 8)
  • varied the the number of reviewers, the number of
    teams inspecting the code unit, and the
    requirement of fixing defects between first and
    second teams inspections
  • two teams per inspection with fix was dropped
    part way through experiment as infeasible
  • results on two teams in parallel are
    inconclusive, where the total number of people is
    the same (eg. 2 teams of 1 vs. 1 team of 2)
  • one of the conclusions
  • structural changes to the inspection process do
    not always have the intended effect significant
    improvements to the inspection process will
    depend on the development of defect detection
    techniques

18
Unstructured Techniques (1)
  • Ad-hoc is common
  • often based on paraphrasing
  • no focus on particular issues
  • no guidance on how to consider the product
  • no explicit use of inspectors expertise
  • not repeatable
  • no record of inspectors thought processes and
    actions
  • does the beginning of the product get more
    attention than the end?
  • are conceptually difficult parts ignored?

19
Unstructured Techniques (2)
  • Checklists are popular but have several
    shortcomings 2, 9
  • checklists are based on past experience
  • some categories will not have been found yet
  • depends on inspectors using individual expertise
    to find issues in missing categories
  • inspectors may not look for anything beyond what
    is in checklist
  • checklists may be lengthy and hard to use
  • checklist categories may not be effective or
    productive
  • categories need to be prioritized
  • some categories may not provide enough guidance
  • checklist for one project may not work on another
  • some checklists should be personalized
  • checklist effectiveness needs to be evaluated
  • constant improvement needed

20
Unstructured Techniques (3)
  • Checklists are effective when 12, 33
  • well established history of the product exists
  • product is predictable
  • inspection goals are accompanied by a well
    documented method of operationalization
  • eg. style manual, standards
  • providing a source of accumulated wisdom to a
    junior member of the team
  • an inspector is working independently and
  • the inspector is not experienced and
  • the checklist provides a relatively complete tour
    of the product under inspection

21
Overview of Structured Inspection Techniques
(University of Maryland 10)
  • Systematic
  • inspector knows how to inspect the document
  • Specific roles
  • inspector is responsible for a specific role with
    a specific focus
  • Distinct
  • each role is distinct in the sense of minimizing
    overlap
  • coverage of different defect classes is achieved
    by having multiple roles

22
Scenario Based Reading
  • A technique that is used individually in order
    to analyze a product or a set of products. Some
    concrete instructions are given to the reader on
    how to read or what to look for in a document.
    (University of Maryland, Notes on Perspective
    based Scenarios 10)
  • defect-based and perspective-based scenarios
    (Porter, Votta, Basili 11)
  • focus on defect classes and use questionnaire to
    guide inspectors
  • focus on a role (perspective) and use
    questionnaire to guide inspectors (may also
    produce a product associated with that role)
  • can prepare a scenario for any inspection

23
Task Directed Inspection (TDI)
  • What is TDI?
  • inspectors produce a usable product for future
    evolution of the software system
  • inspectors expertise is matched to the task
  • used with a light-weight inspection process
  • focus on the work of the individual rather than
    that of the team
  • based on observation that creating careful
    documentation often shows up defects in product
    being documented

24
Experiments with TDI at RMC
  • Experiments in 2000 to 2002
  • Purpose of the experiments
  • Concerns to be addressed in experimentation
  • Elements of the experiments
  • Design of the experiments
  • Results and Conclusions
  • Experiments in 2003/2004

25
Purpose of TDI Experiments at RMC (2000 to 2002)
  • evaluate the effectiveness of TDI
  • can it guide the inspector to identify subtle
    defects in the work product?
  • different levels of understanding involved in
    finding defects
  • low level style issues
  • high level logic errors in a complex calculation
  • compare use of TDI to an ad hoc technique
  • observe the use of TDI in a situation very
    different from where it was first developed

26
Concerns to be Addressed by Design of
Experiment (Confounding factors)
  • natural variation in participants performance
  • maturation of subjects as experiment progresses
  • one detection technique biasing the use of
    another detection technique
  • variations in the instrument, i.e., the pieces of
    code selected for inspection
  • outside influences

27
Elements of the Experiment (2000 to 2002)
  • inspection techniques
  • paraphrasing
  • TDI - method description
  • TDI - test plan
  • code to be inspected
  • CAVMLC - civil engineering code
  • three pieces
  • equivalent length
  • equivalent complexity
  • equivalent application content
  • participants
  • experimental design

28
Design of Experiment (2000)
29
Results from Experiment 2000 - Report in CASCON
Proceedings 2000
30
Results from Experiment 2000 - Report in CASCON
Proceedings 2000
31
Analysis of Results for Experiments 2000 to 2002
  • for the experienced students
  • using TDI resulted in proportionally more
    findings in the Logical category
  • using TDI didnt make any difference in the
    proportion of findings in the Comparative
    category
  • for the inexperienced students
  • using TDI may have helped increase the proportion
    of findings in the Comparative category
  • using TDI seemed to hinder identification of
    findings in the Logical category

32
Conclusions from all three Experiments
  • TDI provides positive results in level of
    understanding of the code for experienced
    inspectors
  • tasks must be appropriate for the inspectors
    background and experience
  • there is a strong interaction between inspection
    technique and inspectors preferred style of
    working

33
Experiment 2003/2004
  • Assumption
  • TDI works!
  • Previous focus
  • work of individual
  • no effects of group meetings considered
  • Purpose for 2003/2004
  • what effects do meetings have when using TDI?

34
Elements of the Experiment 2003/2004
  • inspection technique
  • TDI - high level design
  • code to be inspected
  • chemical engineering - radiation dose on air crew
  • two pieces of code
  • content of both is computational based
  • participants
  • experimental design
  • partial factorial
  • repeated measures
  • individual versus groups
  • warm up in first week

35
References 1
  • Phillip M. Johnson, Reengineering Inspection,
    Communications of the ACM, Feb. 1998, Vol 41, No
    2, pp 49-52
  • Oliver Laitenberger, Khaled El Emam, Thomas
    Harbich, An Internally Replicated
    Quasi-Experimental Comparison of Checklist and
    Perspective-Base Reading of Code Documents, IEEE
    Transactions in Software Engineering, May 2001,
    Vol. 27, No. 5, pp.387-421
  • Lawrence G. Votta, Does the Modern Code
    Inspection Have Value? Presentation at the NRC
    Seminar on Measuring Success Empirical Studies
    of Software Engineering March 1999
    http//www.cser.ca/seminar/ESSE/slides/ESSE_Votta.
    pdf

36
References 2
  • Tom Gilb and Dorothy Graham, Software Inspection,
    Addison Wesley, 1993. see also http//www.result-p
    lanning.com/
  • Votta, Lawrence G. Does Every Inspection Need a
    Meeting?, SIGSOFT93 - Proceedings of 1st ACM
    SIGSOFT Symposium on Software Development
    Engineering, ACM Press, New York, 1993, pp
    107-114
  • Parnas, David L., David M. Weiss Active Design
    Reviews Principles and Practice, Proceedings
    8th International Conference on Software
    Engineering, Aug. 1985

37
References 3
  • Knight, J.C., E.A. Myers An Improved Inspection
    Technique, Communications of the ACM, Nov. 1993,
    Vol 36, No 11, pp. 51-61
  • Porter, Adam A., Harvey P. Siy, Carol A. Toman,
    Lawrence G. Votta An experiment to Assess the
    Cost-Benefits of Code Inspctions in Large Scale
    Software Development, IEEE Transactions on
    Software Engineering, Vol 23, No 6, June 1997, pp
    329-346
  • Chernak, Y. A Statistical Approach to the
    Inspection Checklist Formal Synthesis and
    Improvement IEEE Transactions on Software
    Engineering, 22(12)866-874, Dec. 1996

38
References 4
  • 10 University of Maryland, Notes on Perspective
    based Scenarios online http//www.cs.umd.edu/p
    rojects/SoftEng/ESEG/manual/pbr_package/node8.html
    Nov. 1999
  • 11 Porter, Adam A., Lawrence G. Votta, Victor R.
    Basili Comparing Detection Methods for Software
    Requirements Inspections A Replicated
    Experiment, IEEE Transactions on Software
    Engineering, Vol 21, No 6, June 1995, pp 563-575
  • 12 Kelly, Diane and Terry Shepard Task-Directed
    Software Inspection Technique An Experiment and
    Case Study, Proceedings IBM CASCON, Nov. 2000
  • 13 Glen W. Russell, "Experience with Inspection
    in Ultralarge-Scale Developments", IEEE Software,
    Jan. 1991, pp.25-31

39
References 5
  • 14 R. Chillarege, et al., "Orthogonal Defect
    Classification - A Concept for In-Process
    Management", IEEE Transactions on Software
    Engineering, v. 18 n.11, Nov 92, pp. 943-956
  • 15 Kent Beck, Extreme Programming Explained
    Culture Change, Addison Wesley, 1999
  • 16 Robert B. Grady, Successful Software Process
    Improvement, Prentice Hall, 1997
  • 17 Philip M. Johnson and Danu Tjahjono, Does
    Every Inspection Really Need A Meeting?,
    Journal of Empirical Software Engineering, 4, 1,
    pp 9-35, Jan. 1998

40
References 6
  • 18 David A. Wheeler, Bill Brykczynski, and
    Reginald N. Meeson Jr., Software Inspection An
    Industry Best Practice, IEEE CS Press, 1996
  • 19 Gregory Abowd, Len Bass, Paul Clement, Rick
    Kazman, Linda Northrop, Amy Zaremski
    Recommended Best Industrial Practice for
    Software Architecture Evaluation, Technical
    Report, CMU/SEI-96-TR-025, January 1997
  • 20 Stan Rifkin, Lionel Deimel Program
    Comprehension Techniques Improve Software
    Inspections A Case Study, Proceedings IWPC 00,
    IEEE 2000
  • 21 Steve McConnell, Quantifying Soft Factors,
    IEEE Software, Nov/Dec 2000, pp 9-11

41
References 7
  • 22 Terry Shepard, Margaret Lamb, and Diane
    Kelly, More Testing Should be Taught,
    Communications of the ACM, June 2001, 44 (6), pp.
    103-108
  • 23 Michael Cusumano, Richard Selby, Microsoft
    Secrets, Simon Shuster Inc., 1995
  • 24 Edward Kit, Software Testing in the Real
    World - improving the process, Addison-Wesley,
    1995
  • 25 Robert L. Glass, Inspections - Some
    Surprising Findings, Communications of the ACM,
    April 1999, Vol. 42, no. 4, pp.17-19

42
References 8
  • 26 Adam Porter, Lawrence Votta, Comparing
    Detection Methods for Software Requirements
    Inspections A Replication Using Professional
    Subjects, Empirical Software Engineering
    Journal, 1997
  • 27 Victor Basili, Scott Green Oliver
    Laitenberger, Filippo Lanubile, Forrest Shull,
    Sivert Sorumgard, Marvin Zelkowitz, The
    Empirical Investigation of Perspective-Based
    Reading, Empirical Software Engineering An
    International Journal, 1(2), 1996, pp.133-164
  • 28 Oliver Laitenberger, Thomas Bell, An
    Industrial Case Study to examine a
    non-traditional Inspection Implementation for
    Requirements Specifications, Proceedings 8th
    IEEE Symposium on Software Metrics, June 2002,
    pp.97-106

43
References 9
  • 29 Michael Fredericks, Victor Basili, Using
    Defect Tracking and Analysis to Improve Software
    Quality A DACS State-of-the-Art Report, Rome
    NY, 1998
  • 30 M.E. Fagan, Design and Code Inspections to
    reduce Errors in Program development, IBM
    Systems Journal, Vol. 15, No. 3, 1976, pp.182-211
  • 31 M.E. Fagan, Advances in Software
    Inspections, IEEE Transactions on Software
    Engineering, Vol.12, No. 7, July 1986,
    pp.744-751
  • 32 Watts S. Humphrey, A Discipline for Software
    Engineering, Addison Wesley, 1995

44
References 10
  • 33 Stefan Biffl, Michael Halling, Investigating
    the Influence of Inspector Capability Factors
    with Four Inspection Techniques on Inspection
    Performance, Eight IEEE Symposium on Software
    Metrics, 2002, pp. 107-117
  • 34 Grady, Robert B. (1996). Software Failure
    Analysis for High-Return Process Improvement
    Decisions. Hewlett-Packard Journal, 47(4)
    (August)
  • 35 Boris Beizer, Software Testing Techniques,
    2nd ed., Van Nostrand Reinhold, NY, 1990
  • 36 http//www.pmforum.org/library/glossary/
    PMG_D00.htm

45
References 11
  • 37 Robert Grady, Deborah Caswell Software
    Metrics Establishing a Company-Wide Program,
    Prentice-Hall, 1986
  • 38 IEEE Std 1044 Classification for Software
    Anomalies
  • 39 David Card, Learning from our Mistakes with
    Defect Causal Analysis, IEEE Software Jan-Feb
    1998, pp.56-63
  • 40 http//www.research.ibm.com/softeng/ODC/DETODC.
    HTM and FAQ.HTMconcepts

46
References 12
  • 41 Barry Boehm, Victor Basili Software Defect
    Reduction Top 10 List IEEE Computer, January
    2001, pp.135-137
  • 42 Ram Chillarege, Inderpal Bhandari, Jarir
    Chaar, Michael Halliday, Diane Moebus, Bonnie
    Ray, Man-Yuen Wong Orthogonal Defect
    Classification - A Concept for In-Process
    Measurements IEEE Transactions on Software
    Engineering, Vol.18, No.11, Nov. 1992, pp.943-956
  • 43 Jarir Chaar, Michael Halliday, Inderpal
    Bhandari, Ram Chillarege In-Process Evaluation
    for Software Inspection and Test, IEEE
    Transactions on Software Engineering, Vol.19,
    No.11, Nov. 1993, pp.1055-1070
About PowerShow.com