Software Quality Metrics Overview - PowerPoint PPT Presentation


PPT – Software Quality Metrics Overview PowerPoint presentation | free to download - id: 1fefc0-ZDc1Z


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation

Software Quality Metrics Overview


Product metrics e.g., size, complexity, design features, performance, quality level ... Count executable lines plus data definitions ... – PowerPoint PPT presentation

Number of Views:320
Avg rating:3.0/5.0
Slides: 64
Provided by: robertw94
Learn more at:


Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Software Quality Metrics Overview

Software Quality Metrics Overview
Types of Software Metrics
  • Product metrics e.g., size, complexity, design
    features, performance, quality level
  • Process metrics e.g., effectiveness of defect
    removal, response time of the fix process
  • Project metrics e.g., number of software
    developers, cost, schedule, productivity

Software Quality Metrics
  • The subset of metrics that focus on quality
  • Software quality metrics can be divided into
  • End-product quality metrics
  • In-process quality metrics
  • The essence of software quality engineering is to
    investigate the relationships among in-process
    metric, project characteristics , and end-product
    quality, and, based on the findings, engineer
    improvements in quality to both the process and
    the product.

Three Groups of Software Quality Metrics
  • Product quality
  • In-process quality
  • Maintenance quality

Product Quality Metrics
  • Intrinsic product quality
  • Mean time to failure
  • Defect density
  • Customer related
  • Customer problems
  • Customer satisfaction

Intrinsic Product Quality
  • Intrinsic product quality is usually measured by
  • the number of bugs (functional defects) in the
    software (defect density), or
  • how long the software can run before crashing
    (MTTF mean time to failure)
  • The two metrics are correlated but different

Difference Between Errors, Defects, Faults, and
Failures (IEEE/ANSI)
  • An error is a human mistake that results in
    incorrect software.
  • The resulting fault is an accidental condition
    that causes a unit of the system to fail to
    function as required.
  • A defect is an anomaly in a product.
  • A failure occurs when a functional unit of a
    software-related system can no longer perform its
    required function or cannot perform it within
    specified limits

Whats the Difference between a Fault and a
The Defect Density Metric
  • This metric is the number of defects over the
    opportunities for error (OPE) during some
    specified time frame.
  • We can use the number of unique causes of
    observed failures (failures are just defects
    materialized) to approximate the number of
  • The size of the software in either lines of code
    or function points is used to approximate OPE.

Lines of Code
  • Possible variations
  • Count only executable lines
  • Count executable lines plus data definitions
  • Count executable lines, data definitions, and
  • Count executable lines, data definitions,
    comments, and job control language
  • Count lines as physical lines on an input screen
  • Count lines as terminated by logical delimiters

Lines of Code (Contd)
  • Other difficulties
  • LOC measures are language dependent
  • Cant make comparisons when different languages
    are used or different operational definitions of
    LOC are used
  • For productivity studies the problems in using
    LOC are greater since LOC is negatively
    correlated with design efficiency
  • Code enhancements and revisions complicates the
    situation must calculate defect rate of new and
    changed lines of code only

Defect Rate for New and Changed Lines of Code
  • Depends on the availability on having LOC counts
    for both the entire produce as well as the new
    and changed code
  • Depends on tracking defects to the release origin
    (the portion of code that contains the defects)
    and at what release that code was added, changed,
    or enhanced

Function Points
  • A function can be defined as a collection of
    executable statements that performs a certain
    task, together with declarations of the formal
    parameters and local variables manipulated by
    those statements.
  • In practice functions are measured indirectly.
  • Many of the problems associated with LOC counts
    are addressed.

Measuring Function Points
  • The number of function points is a weighted total
    of five major components that comprise an
  • Number of external inputs x 4
  • Number of external outputs x 5
  • Number of logical internal files x10
  • Number of external interface files x 7
  • Number of external inquiries x 4

Measuring Function Points (Contd)
  • The function count (FC) is a weighted total of
    five major components that comprise an
  • Number of external inputs x (3 to 6)
  • Number of external outputs x (4 to 7)
  • Number of logical internal files x (7 to 15)
  • Number of external interface files x (5 to 10)
  • Number of external inquiries x (3 to 6)
  • the weighting factor depends on complexity

Measuring Function Points (Contd)
  • Each number is multiplied by the weighting factor
    and then they are summed.
  • This weighted sum (FC) is further refined by
    multiplying it by the Value Adjustment Factor
  • Each of 14 general system characteristics are
    assessed on a scale of 0 to 5 as to their impact
    on (importance to) the application.

The 14 System Characteristics
  1. Data Communications
  2. Distributed functions
  3. Performance
  4. Heavily used configuration
  5. Transaction rate
  6. Online data entry
  7. End-user efficiency

The 14 System Characteristics (Contd)
  1. Online update
  2. Complex processing
  3. Reusability
  4. Installation ease
  5. Operational ease
  6. Multiple sites
  7. Facilitation of change

The 14 System Characteristics (Contd)
  • VAF is the sum of these 14 characteristics
    divided by 100 plus 0.65.
  • Notice that if an average rating is given each of
    the 14 factors, their sum is 35 and therefore VAF
  • The final function point total is then the
    function count multiplied by VAF
  • FP FC x VAF

Customer Problems Metric
  • Customer problems are all the difficulties
    customers encounter when using the product.
  • They include
  • Valid defects
  • Usability problems
  • Unclear documentation or information
  • Duplicates of valid defects (problems already
    fixed but not known to customer)
  • User errors
  • The problem metric is usually expressed in terms
    of problems per user month (PUM)

Customer Problems Metric (Contd)
  • PUM Total problems that customers reported for
    a time period ltdivided bygt Total number of
    license-months of the software during the period
  • where
  • Number of license-months Number of the
    install licenses of the software x Number of
    months in the calculation period

Approaches to Achieving a Low PUM
  • Improve the development process and reduce the
    product defects.
  • Reduce the non-defect-oriented problems by
    improving all aspects of the products (e.g.,
    usability, documentation), customer education,
    and support.
  • Increase the sale (number of installed licenses)
    of the product.

Defect Rate and Customer Problems Metrics
Defect Rate Problems per User-Month (PUM)
Numerator Valid and unique product defects All customer problems (defects and nondefects, first time and repeated)
Denominator Size of product (KLOC or function point) Customer usage of the product (user-months)
Measurement perspective Producersoftware development organization Customer
Scope Intrinsic product quality Intrinsic product quality plus other factors
Customer Satisfaction Metrics
Customer Satisfaction Issues
Customer Problems
Customer Satisfaction Metrics (Contd)
  • Customer satisfaction is often measured by
    customer survey data via the five-point scale
  • Very satisfied
  • Satisfied
  • Neutral
  • Dissatisfied
  • Very dissatisfied

IBM Parameters of Customer Satisfaction
  • Capability (functionality)
  • Usability
  • Performance
  • Reliability
  • Installability
  • Maintainability
  • Documentation
  • Service
  • Overall

HP Parameters of Customer Satisfaction
  • Functionality
  • Usability
  • Reliability
  • Performance
  • Service

Examples Metrics for Customer Satisfaction
  1. Percent of completely satisfied customers
  2. Percent of satisfied customers (satisfied and
    completely satisfied)
  3. Percent of dissatisfied customers (dissatisfied
    and completely dissatisfied)
  4. Percent of nonsatisfied customers (neutral,
    dissatisfied, and completely dissatisfied)

In-Process Quality Metrics
  • Defect density during machine testing
  • Defect arrival pattern during machine testing
  • Phase-based defect removal pattern
  • Defect removal effectiveness

Defect Density During Machine Testing
  • Defect rate during formal machine testing
    (testing after code is integrated into the system
    library) is usually positively correlated with
    the defect rate in the field.
  • The simple metric of defects per KLOC or function
    point is a good indicator of quality while the
    product is being tested.

Defect Density During Machine Testing (Contd)
  • Scenarios for judging release quality
  • If the defect rate during testing is the same or
    lower than that of the previous release, then
    ask Does the testing for the current release
  • If the answer is no, the quality perspective is
  • If the answer is yes, you need to do extra

Defect Density During Machine Testing (Contd)
  • Scenarios for judging release quality (contd)
  • If the defect rate during testing is
    substantially higher than that of the previous
    release, then ask Did we plan for and actually
    improve testing effectiveness?
  • If the answer is no, the quality perspective is
  • If the answer is yes, then the quality
    perspective is the same or positive.

Defect Arrival Pattern During Machine Testing
  • The pattern of defect arrivals gives more
    information than defect density during testing.
  • The objective is to look for defect arrivals that
    stabilize at a very low level, or times between
    failures that are far apart before ending the
    testing effort and releasing the software.

Two Contrasting Defect Arrival Patterns During
Three Metrics for Defect Arrival During Testing
  • The defect arrivals during the testing phase by
    time interval (e.g., week). These are raw
    arrivals, not all of which are valid.
  • The pattern of valid defect arrivals when
    problem determination is done on the reported
    problems. This is the true defect pattern.
  • The pattern of defect backlog over time. This is
    needed because development organizations cannot
    investigate and fix all reported problems
    immediately. This metric is a workload statement
    as well as a quality statement.

Phase-Based Defect Removal Pattern
  • This is an extension of the test defect density
  • It requires tracking defects in all phases of the
    development cycle.
  • The pattern of phase-based defect removal
    reflects the overall defect removal ability of
    the development process.

Defect Removal by Phase for Two Products
Defect Removal Effectiveness
  • DRE (Defects removed during a development phase
    ltdivided bygt Defects latent in the product) x
  • The denominator can only be approximated.
  • It is usually estimated as
  • Defects removed during the phase
  • Defects found later

Defect Removal Effectiveness (Contd)
  • When done for the front end of the process
    (before code integration), it is called early
    defect removal effectiveness.
  • When done for a specific phase, it is called
    phase effectiveness.

Phase Effectiveness of a Software Product
Metrics for Software Maintenance
  • The goal during maintenance is to fix the defects
    as soon as possible with excellent fix quality
  • The following metrics are important
  • Fix backlog and backlog management index
  • Fix response time and fix responsiveness
  • Percent delinquent fixes
  • Fix quality

Fix Backlog
  • Fix backlog is a workload statement for software
  • It is related to both the rate of defect arrivals
    and the rate at which fixes for reported problems
    become available.
  • It is a simple count of reported problems that
    remain at the end of each time period (week,
    month, etc.)

Backlog Management Index (BMI)
  • BMI (Number of problems closed during the month
    ltdivided bygt Number of problem arrivals during
    the month) x 100.
  • If BMI is larger than 100, it means the backlog
    is reduced.
  • If BMI is less than 100, then the backlog is

Opened Problems, Closed Problems, and Backlog
Management Index by Month
Fix Response Time and Fix Responsiveness
  • The fix response time metric is usually
    calculated as
  • Mean time of all problems from open to closed
  • Metric may be used for different defect severity
  • Fix response time relates to customer
  • But meeting agreed-to fix time is more than just
    achieving a short fix time.
  • A possible metric is the percentage of delivered
    fixes meeting committed dates to customers.

Percent Delinquent Fixes
  • The mean response time metric is a central
    tendency measure.
  • A more sensitive metric is the percentage of
    delinquent fixes (for each fix, if the turnaround
    time greatly exceeds the required response time,
    it is classified as delinquent).
  • Percent delinquent fixes (Number of fixes that
    exceeded the response time criteria by severity
    level ltdivided bygt Number of fixes delivered in a
    specified time) x 100

Percent Delinquent Fixes (Contd)
  • This is not a real-time metric because it is for
    closed problems only.
  • For a real-time metric we must factor in problems
    that are still open.
  • We can use the following metric
  • Real-Time Delinquency Index 100 x Delinquent
    / (Backlog Arrivals)

Real-Time Delinquency Index
Fix Quality
  • The number of defective fixes is another quality
    metric for maintenance.
  • The metric of percent defective fixes is simply
    the percentage of all fixes in a time interval
    that are defective.
  • Recording both the time the defective fix was
    discovered and the time the fix was made to be
    able to calculate the latent period of the
    defective fix.

Examples of Metrics Programs
  • Motorola
  • Follows the Goal/Question/Metric paradigm of
    Basili and Weiss
  • Goals
  • Improve project planning
  • Increase defect containment
  • Increase software reliability
  • Decrease software defect density
  • Improve customer service
  • Reduce the cost of nonconformance
  • Increase software productivity

Examples of Metrics Programs (Contd)
  • Motorola (contd)
  • Measurement Areas
  • Delivered defects and delivered defects per size
  • Total effectiveness throughout the process
  • Adherence to schedule
  • Accuracy of estimates
  • Number of open customer problems
  • Time that problems remain open
  • Cost of nonconformance
  • Software reliability

Examples of Metrics Programs (Contd)
  • Motorola (contd)
  • For each goal the questions to be asked and the
    corresponding metrics were formulated
  • Goal 1 Improve Project Planning
  • Question 1.1 What was the accuracy of estimating
    the actual value of project schedule?
  • Metric 1.1 Schedule Estimation Accuracy (SEA)
  • SEA (Actual project duration)/(Estimated
    project duration)

Examples of Metrics Programs (Contd)
  • Hewlett-Packard
  • The software metrics program includes both
    primitive and computed metrics.
  • Primitive metrics are directly measurable
  • Computed metrics are mathematical combinations of
    primitive metrics
  • (Average fixed defects)/(working day)
  • (Average engineering hours)/(fixed defect)
  • (Average reported defects)/(working day)
  • Bang A quantitative indicator of net usable
    function from the users point of view

Examples of Metrics Programs (Contd)
  • Hewlett-Packard (contd)
  • Computed metrics are mathematical combinations of
    primitive metrics (contd)
  • (Branches covered)/(total branches)
  • Defects/KNCSS (thousand noncomment source
  • Defects/LOD (lines of documentation not included
    in program source code)
  • Defects/(testing time)
  • Design weight sum of module weights (function
    of token and decision counts) over the set of all
    modules in the design

Examples of Metrics Programs (Contd)
  • Hewlett-Packard (contd)
  • Computed metrics are mathematical combinations of
    primitive metrics (contd)
  • NCSS/(engineering month)
  • Percent overtime (average overtime)/(40 hours
    per week)
  • Phase (engineering months)/(total engineering

Examples of Metrics Programs (Contd)
  • IBM Rochester
  • Selected quality metrics
  • Overall customer satisfaction
  • Postrelease defect rates
  • Customer problem calls per month
  • Fix response time
  • Number of defect fixes
  • Backlog management index
  • Postrelease arrival patterns for defects and

Examples of Metrics Programs (Contd)
  • IBM Rochester (contd)
  • Selected quality metrics (contd)
  • Defect removal model for the software development
  • Phase effectiveness
  • Inspection coverage and effort
  • Compile failures and build/integration defects
  • Weekly defect arrivals and backlog during testing
  • Defect severity

Examples of Metrics Programs (Contd)
  • IBM Rochester (contd)
  • Selected quality metrics (contd)
  • Defect cause and problem component analysis
  • Reliability (mean time to initial program loading
    during testing)
  • Stress level of the system during testing
  • Number of system crashes and hangs during stress
    testing and system testing
  • Various customer feedback metrics
  • S curves for project progress

Collecting Software Engineering Data
  • The challenge is to collect the necessary data
    without placing a significant burden on
    development teams.
  • Limit metrics to those necessary to avoid
    collecting unnecessary data.
  • Automate the data collection whenever possible.

Data Collection Methodology (Basili and Weiss)
  1. Establish the goal of the data collection
  2. Develop a list of questions of interest
  3. Establish data categories
  4. Design and test data collection forms
  5. Collect and validate data
  6. Analyze data

Reliability of Defect Data
  • Testing defects are generally more reliable than
    inspection defects since inspection defects are
    more subjective
  • An inspection defect is a problem found during
    the inspection process that, if not fixed, would
    cause one or more of the following to occur
  • A defect condition in a later inspection phase

Reliability of Defect Data
  • An inspection defect is one which would cause
  • A defect condition during testing
  • A field defect
  • Nonconformance to requirements and specifications
  • Nonconformance to established standards

An Inspection Summary Form