InProcess Metrics for Software Testing By Doddi Ravi

1
In-Process Metrics for Software Testing
By
Doddi Ravi

2
In-Process Metrics?

• the quality measurements that are taken while you
  are still in the midst of the process
• the measurements are taken from the testing
  process, not from the tested software
• Measurements are then used to predict how the
  testing will advance

3
Why to use In-Process metrics?

• help you predict the quality of the final
  completed product before you complete the
  process, thus giving you a chance to correct or
  improve quality before proceeding further
• Effective use of good in-process metrics can
  significantly enhance the success of the project,
  i.e., on-time delivery with desirable quality
• In-process tracking and measurements play a
  critical role in software development,
  particularly for software testing

4
In-Process metrics for
Software Testing

• Test progress S curve (plan, attempted, actual)
• PTR (test defects) arrivals over time

5
Metric implementation

• Metric implementation experiences at the IBM
  Rochester
• AS/400 software development laboratory

6
Test progress S curve (plan, attempted,
actual)

• the most important and basic tracking for
  managing software testing
• Purpose
• To track actual testing
  process against plan and therefore
  to be able to be proactive upon early
  indications that the testing is falling behind

7
Contd

• a test progress S curve over time with
• the x-axis representing
  the time unit
  (preferably in weeks)
• the y-axis representing
  the number of test
  cases or test points
• By S curve we mean that the data are cumulative
  over
• time and resemble an s shape as a result
  of the period
• of intense test activity, causing a steep
  planned test ramp-
• up

8
Contd

• For the metric to be useful, it should
  contain the following information on the same
  graph
• Planned progress over time (in terms of number
  of
• test cases or number of test points to be
  completed
• successfully by week)
• Number of test cases attempted by week
• Number of test cases completed successfully by
• week

9
Example using Test Cases
10
Example using Test Points
11
Pros of Test progress S curve

• Using of S curves
• reduces the risk of schedule slippage
• forces better planning.
• gives a quick visual status of the progress
  against the total plan and plan-to-date
• for release-to-release or project-to-project
  comparisions

12
S curves for release-to-release comparisions

• Uses weeks before product general availability
  (GA) as the time unit for the x-axis
• the testing execution plan needs to be laid out
  in terms of the weekly target, and actual data
  need to be tracked on a weekly basis

13
Example of release-to-release comparisions
14
Cons

• May be difficult to discern the exact amount of
  slippage, particularly for large testing efforts,
  where the number of test cases is in the hundreds
  of thousands

15
Solution

• display testing status in tabular form as below

16
PTR (test defects) arrivals over time

• tracking the test defect arrival pattern over
  time, in addition to tracking by test phase
• The pattern of defect arrivals over time gives
  more information
• different patterns of defect arrivals imply
  different quality levels in the field

17
PTR Tool

• Problem Tracking Report tool is used for defect
  tracking
• The unit for the x-axis is weeks before GA
• The unit for the y-axis is the number of PTR
  arrivals
• for the week, or related measures

18
Example for several AS/400 releases
19
Interpretation of the Metric

• If the defect rate is the same or lower than that
  of the previous release (or a model), then ask
  Is the testing for the current release less
  effective?
• If the answer is no, the quality
  perspective is positive
• If the answer is yes, extra
  testing is needed. process improvement in
  development and testing should be sought
• If the defect rate is substantially higher than
  that of the previous release (or a model), then
  ask Did we plan for and actually improve testing
  effectiveness significantly?
• If the answer is no, the quality is
  negative
• If the answer is yes, then the
  quality perspective
• is the same or positive

20
Conclusion

• a set of in-process metrics for the testing
  phases of the software development process with
  real-life examples based on implementation
  experiences of the IBM Rochester AS/400
  development laboratory have been discussed

21
References

• In-Process Metrics for Software Testing by S. H.
  Kan, J. Parrish, and D. Manlove
• Measuring when software is tested by Antti Pirilä
  SeminarWorkTietojenkäsittelytieteen
  laitosHelsingin yliopistoMarch 10, 2006, 9
  pages
• Software Metrics, SEI Curriculum Module
  SEI-CM-12-1.1 December 1988 Everald E. Mills,
  Seattle University
• Testing in Software Development ,Michael Poor,
  Larissa Winey, Miranda Steed, Takao Tanizawa,
  Jeffrey Dyer Computer Science Department at
  Tufts University 161 College Ave Medford, MA
  02155 michael.poor_at_tufts.edu
• http//www.research.ibm.com/journal/sj/401/kan.htm
  l
• http//www.awprofessional.com/articles/article.asp
  ?p30176rl1
• http//www.softwareqatest.com/qatfaq1.htmlFAQ1_2
• http//www.rspa.com/reflib/ProductMetrics.html
• http//en.wikipedia.org/wiki/Software_metrics

22

• Queries?

23
Applause!
24