Getting Business Value out of (Test) Metrics Richard Terry (UK) [e-mail] Rob Baarda (NL)

1
Getting Business Value out of (Test)
MetricsRichard Terry (UK)richard.terry_at_sogeti.c
omRob Baarda (NL)
2
What we will do / Structure

• Subjects
• Test Metric goals / stakeholders
• Which metrics some demo
• Process around metrics
• How to apply metrics / pittfalls
• Wrap-up
• For each subject intro, discussion
• Discussion
• hand raise
• hand raise with comments
• open

3
Metric Goals / Stakeholders

• Hand raise only
• Who is using project metrics?
• Who is using test metrics?
• Who would like to use test metrics?
• Some opinions please!
• Goals / Stakeholders for test metrics? OPEN!
• Your votes please

4
Which test metrics?
Test basis
Test object
Test Execution
Defects
Repair
5
Deductible metrics

• Effort Size / hours
• Productivity test cases / hours
• Efficiency defects / ( hours or test
  cases)
• Specification
• Test execution
• Retest of repaired defects
• DDP Defect Detection Percentage (Europe)DRE
  Defect Removal Efficiency (USA)
• Defect injection rate for rework
• Damage prevented?
• More metrics
• Availability of test environments
• OPEN

6
Dutch test metrics experiences

• Dutch initiative to gather test metrics
• Parties involved
• NESMANetherlands Software Metrics Association
• Testnet, including SogetiDutch Testing community
• LaQuSOLaboratory for Software QualityUniversitie
  s Eindhoven Nijmegen

Some slides with data from the database will
follow in the presentation, not in the hand-out,
due to a lack of statistical evidence
7
Processes around metrics

• Collection in a project
• Embedded in daily work
• Weekly summarisation
• Sanity checks
• Cost about 2 project budget
• Distribution
• For a benchmark on the level of
• Project releases
• Organisation
• Country
• International ISBSGInternational Software
  Benchmarking Standards Group

8
To what extent?

• OPEN Discussion about
• Accuracy of definitions
• Number of types of defects
• Is a batch test case the same as an online test
  case?
• Only testing of functionality or also security,
  performance, usability
• How to include regression testing?
• Predictive valueaverage (mean), median, standard
  deviation, correlations with?Prediction model
  needed?

9
To apply test metrics

• Project
• Estimating Testing New development
• Estimating Testing Maintenance release
• Controlling the test project
• Benchmark
• Measure the effectiveness of the testing, Improve
  inside / outside the test process?
• PITTFALLS?

10
Make a judgement based with metrics?

• Of person production/ quality of work
• OPEN Discussion
• Some Dutch effects
• Of Service Level Agreement
• What to measure and how to judge

11
Wrap-up

• Business value out of metricsHand raise
• Who would like to use test metrics?
• Changed opinions
• Comments please (- ? , ? -)
• Reading TMap Next
• Chapter 13 Goal-Question-Metrics for test
• Chapter 11 for data
• End of 2008 www.isbsg.orgParticipate? Leave
  your business card