TQS Teste e Qualidade de Software Software Testing and Quality Software Quality Concepts

1
TQS - Teste e Qualidade de Software(Software
Testing and Quality) Software Quality Concepts
João Pascoal Faria jpf_at_fe.up.pt
www.fe.up.pt/jpf
2
Index

• Product and Process
• Product Quality
• Product Quality Attributes
• Bugs, Defects, Faults, Failures and Errors
• Quality Costs
• Quality Factors
• Process Quality
• Quality Related Activities
• Verification Validation
• Quality Assurance / Management
• Tests, inspections and reviews, measurements
• Process Management and Improvement

3
Product and process
Business System (Software Development
Organization)
goals
Business Process(Software Development Process)
Demand, Needs
Costumer or Market
(Software) Product or Service
software developer
software tester
software quality (assurance) engineer
Customer external or internal
resources
Product product or service
Test test and review
Development development and maintenance
4
What is a software product?

• Software product computer programs associated
  documentation
• Software products may be
• Custom (à medida) - developed for a particular
  customer, according to its specifications
• Generic (package) - developed for a general
  market, to be sold to a range of different
  customers
• Types of software products
• Business support software
• Supports the activities of a business or
  organization
• Usually multi-user
• Personal productivity software
• Spreadsheets, word processing tools,
• Usually single-user
• Embedded software

5
What is a software development process?

• Is the definition of a set of activities whose
  goal is the development or evolution of a
  software product
• Usually comprises the following generic
  activities
• Specification - what the system should do and its
  development constraints
• Development/Implementation - production of the
  software system
• Verification Validation - checking that the
  software meets its specification and is what the
  customer wants (source Ian Sommerville)

Software Development Process
New or changed requirements (problem)
New or changed software product (solution)
(source "Rational Unified Process")
6
Process and project
class
instance

• Project
• tasks
• artifacts (concrete)
• individuals
• schedule
• guidelines, procedures
• process performed, process instance

• Process
• activites
• artifacts (templates)
• actors, roles
• workflow
• guidelines, procedures
• process description

• what
• what
• who
• when
• how

aspects
7
The importance of software

• The economies of ALL developed nations are
  dependent on software
• More and more systems are software controlled
• Including an increasing number of safety-critical
  and mission-critical systems, with high demands
  on dependability
• More and more businesses depend on software for
  their success
• Software and Information Systems are critical
  success factors in an increasing number of
  businesses and organizations
• Software engineering expenditure (in the
  development and maintenance of software products)
  represents a significant fraction of GNP (Gross
  National Product) in all developed countries

8
What is product quality?

• Quality, simplistically, means that a product
  should meet its specification
• The software product should deliver the required
  functionality (functional requirements) with the
  required quality attributes (nonfunctional
  requirements)
• This is problematical for software systems
• Tension between customer quality requirements
  (efficiency, reliability, ...) and developer
  quality requirements (maintainability,
  reusability, ...)
• Some quality requirements are difficult to
  specify in an unambiguous way
• Software specifications are usually incomplete
  and often inconsistent
• The quality compromise we cannot wait for
  specifications to improve before paying attention
  to quality, and procedures must be put into place
  to improve quality in spite of imperfect
  specification
• More general notion (since specifications are
  imperfect) fitness for purpose, value to user
• Quality attributes are frequently conflicting and
  increase development costs, so there is a need
  for weighting and balancing
• Software engineering is concerned with the
  cost-effective development of good software
• Quality expectations - delivery

9
The current status of software quality

• Microsoft Windows XP End-User License Agreement
• 11. LIMITED WARRANTY FOR PRODUCT ACQUIRED IN
  THE US AND CANADA. Microsoft warrants that the
  Product will perform substantially in accordance
  with the accompanying materials for a period of
  ninety days from the date of receipt.() If an
  implied warranty or condition is created by your
  state/jurisdiction and federal or
  state/provincial law prohibits disclaimer of it,
  you also have an implied warranty or condition,
  BUT ONLY AS TO DEFECTS DISCOVERED DURING THE
  PERIOD OF THIS LIMITED WARRANTY (NINETY DAYS).
  ()Some states/jurisdictions do not allow
  limitations on how long an implied warranty or
  condition lasts, so the above limitation may not
  apply to you.()YOUR EXCLUSIVE REMEDY.
  Microsoft's and its suppliers' entire liability
  and your exclusive remedy shall be, at
  Microsoft's option from time to time exercised
  subject to applicable law, (a) return of the
  price paid (if any) for the Product, or (b)
  repair or replacement of the Product, that does
  not meet this Limited Warranty and that is
  returned to Microsoft with a copy of your
  receipt. (..)This Limited Warranty is void if
  failure of the Product has resulted from
  accident, abuse, misapplication, abnormal use or
  a virus.

10
Bugs, defects, faults, failures and errors (1)

• Multiple names for (almost) the same thing ...
• Bug - An unwanted and unintended property of a
  program or piece of hardware, especially one that
  causes it to malfunction. Antonym of feature.
  E.g. "There's a bug in the editor it writes
  things out backward." The identification and
  removal of bugs in a program is called
  "debugging".
• Defect same as bug.
• Fault (falha) - A manifestation of an error in
  software. A fault, if encountered, may cause a
  failure.
• Failure (falha) - The inability of a system or
  system component to perform a required function
  within specified limits. A failure may be
  produced when a fault is encountered.
• Error - A discrepancy between a computed,
  observed, or measured value or condition and the
  true, specified, or theoretically correct value
  or condition.
• (source http//computing-dictionary.thefreedicti
  onary.com)

11
Bugs, defects, faults, failures and errors (2)

• Useful distinction according to cause and effect
• bug/defect (construction-time) -gt fault/error
  (run-time) -gt failure (run-time, observable)
  (??)
• A fault does not necessarily lead to a failure
• Fault tolerant systems or components have the
  ability to continue normal operation despite the
  presence of hardware or software faults (this
  often involves some degree of redundancy)
• A system failure is not necessarily caused by a
  system fault
• system failure caused by user fault drive a car
  at 240 km/h
• Bugs and features are antonyms. Technically, if
  something is documented as a feature, it's not a
  bug.

12
When does a software bug occur?

• The software doesn't do something that the
  product specification says it should do
• The software does something that the product
  specification says it shouldn't do
• The software does something that the product
  specification doesn't mention
• The software doesn't do something that the
  product specification doesn't mention but should
• The software is difficult to understand, hard to
  use, slow, or in the software tester's eyes
  will be viewed by end users as just plain not
  right
• (source Ron Patton, "Software Testing")

13
What are the main sources of bugs?

• (source "Software Testing", Ron Patton)

14
Product quality attributes

• Attributes of good software (beyond delivering
  the required functionality)
• Efficiency
• Software should not make wasteful use of system
  resources (disk and memory space, CPU time, etc.)
  and should present appropriate response times
• Usability (ease of use and learning)
• Software must be usable by the users for which it
  was designed
• Dependability (reliability, availability,
  security, safety,)
• Software must be trustworthy
• Maintainability (ease of maintenance)
• Software must evolve to meet changing needs
• Software costs more to maintain than it does to
  develop. For systems with a long life,
  maintenance costs may be several times
  development costs

15
Software quality characteristics and attributes -
ISO 9126 1998 (1/3)

• Characteristics Subcharacteristics
• Functionality - Characteristics relating to
  achievement of the basic purpose for which the
  software is being engineered
• Suitability - The presence and appropriateness of
  a set of functions for specified tasks
• Accuracy - The provision of right or agreed
  results or effects
• Interoperability - Softwares ability to interact
  with specified systems
• Security - Ability to prevent unauthorized
  access, whether accidental or deliberate, to
  programs and data.
• Compliance - Adherence to application-related
  standards, conventions, regulations in laws and
  protocols
• Reliability - Characteristics relating to
  capability of software to maintain its level of
  performance under stated conditions for a stated
  period of time
• Maturity - Attributes of software that bear on
  the frequency of failure by faults in the
  software
• Fault tolerance - Ability to maintain a specified
  level of performance in cases of software faults
  or unexpected inputs
• Recoverability - Capability and effort needed to
  reestablish level of performance and recover
  affected data after possible failure
• Compliance - Adherence to application-related
  standards, conventions, regulations in laws and
  protocols

16
Software quality characteristics and attributes -
ISO 9126 1998 (2/3)

• Usability - Characteristics relating to the
  effort needed for use, and on the individual
  assessment of such use, by a stated or implied
  set of users
• Understandability - The effort required for a
  user to recognize the logical concept and its
  applicability
• Learnability - The effort required for a user to
  learn its application, operation, input, and
  output
• Operability - The ease of operation and control
  by users
• Attractiveness - The capability of the software
  to be attractive to the user
• Compliance - Adherence to application-related
  standards, conventions, regulations in laws and
  protocols
• Efficiency - Characteristic related to the
  relationship between the level of performance of
  the software and the amount of resources used,
  under stated conditions
• Time behavior - The speed of response and
  processing times and throughput rates in
  performing its function
• Resource utilization - The amount of resources
  used and the duration of such use in performing
  its function
• Compliance - Adherence to application-related
  standards, conventions, regulations in laws and
  protocols

17
Software quality characteristics and attributes -
ISO 9126 1998 (3/3)

• Maintainability - Characteristics related to the
  effort needed to make modifications, including
  corrections, improvements or adaptation of
  software to changes in environment, requirements
  and functional specifications
• Analyzability - The effort needed for diagnosis
  of deficiencies or causes of failures, or for
  identification parts to be modified
• Changeability - The effort needed for
  modification fault removal or for environmental
  change
• Stability - The risk of unexpected effect of
  modifications
• Testability - The effort needed for validating
  the modified software
• Compliance - Adherence to application-related
  standards, conventions, regulations in laws and
  protocols
• Portability - Characteristics related to the
  ability to transfer the software from one
  organization or hardware or software environment
  to another
• Adaptability - The opportunity for its adaptation
  to different specified environments
• Installability - The effort needed to install the
  software in a specified environment
• Co-existence - The capability of a software
  product to co-exist with other independent
  software in common environment
• Replaceability - The opportunity and effort of
  using it in the place of other software in a
  particular environment
• Compliance - Adherence to application-related
  standards, conventions, regulations in laws and
  protocols

18
Main dimensions of dependability

• Reliability - The probability of failure-free
  system operation over a specified time in a given
  environment for a given purpose
• Availability - The probability that a system, at
  a point in time, will be operational and able to
  deliver the requested services
• Its possibly to have high availability with low
  reliability if failures are repaired quickly
• Safety - The systems ability to operate,
  normally or abnormally, without danger of causing
  human injury or death and without damage to the
  systems environment
• Security The systems ability to protect itself
  from accidental or deliberate external attack

19
Systems with special quality needs - Critical
Systems

• Types of critical systems
• Safetycritical system a system whose failure
  may result in injury, loss of life or major
  environment damage
• e.g. an insulin delivery system
• Mission-critical system a system whose failure
  may result in the failure of some goal-directed
  activity
• e.g. a navigational system for a space aircraft
• Business-critical system a system whose failure
  may result in the failure of the business using
  the system
• e.g. a customer account system in a bank
• For critical systems, it is usually the case that
  the most important system property is the
  dependability of the system
• Dependability and reliability may be achieved by
  fault prevention, correction but also by fault
  tolerance

20
Does usage and time cause degradation of a
software product quality?

• By definition, should not, but
• A program running continuously for a long period
  of time may work increasingly slower or even
  crash
• e.g. because of memory leaks or memory
  fragmentation
• may require shutting down and restarting
• do you know products like this?
• Performance decreases with the number of
  concurrent users and the volume of the data
• may require hardware/software upgrade
• Maintainability decreases with time
• may require preventive maintenance (migration to
  new technologies, etc.)
• Software becomes obsolete very quickly
• because of fast evolution of technology,
  requirements or knowledge
• sometimes software is used for longer time than
  expected (remember Y2K bug)
• requires continuous innovation and evolution

21
Principal product quality factors (1)
Software development process
Budget and Schedule

• (source "Software Engineering", I. Sommervile)

22
Principal product quality factors (2)

• Process quality
• A good process is usually required to produce a
  good product
• For manufactured goods, process is the principal
  quality determinant
• For design-based activity (like software
  development), other factors are also involved
  especially the capabilities of the designers
• For large projects with average capabilities,
  the development process determines product
  quality
• People quality
• For small projects, the capabilities of the
  developers is the main determinant
• Corollary you need lower quality people (and
  higher quality process) in larger projects?
• Project Size x People Quality Constant ?
• Development technology
• Is particularly significant for small projects
• Budget and schedule
• In all projects, if an unrealistic schedule is
  imposed then product quality will suffer

23
Process quality attributes

• (source "Software Engineering", I. Sommervile)

24
Quality costs

• Costs of conformance
• All costs associated with planning and running
  tests (and revisions) just one time
• Costs of nonconformance
• Costs due to internal failures (before release)
• Cost of isolating, reporting and regression
  testing bugs (found before the product is
  released) to assure that they're fixed (left-hand
  side of fig. 1.2)
• Costs due to external failures (after release)
• If bugs are missed and make it through to the
  customers, the result will be costly product
  support calls, possibly fixing, retesting, and
  releasing the software, and in a worst
  case-scenario a product recall or lawsuits
  (right-hand side of fig. 1.2)
• (source "Software Testing", Ron Patton)

25
Quality costs Costs of nonconformance (1)

• (source "Software Project Survival Guide", Steve
  McConnell)

26
Quality costsCosts of nonconformance (2)

• (source "Software Testing", Ron Patton)

27
Quality costsQuality is free!?

• In his book "Quality is Free The Art of Making
  Quality Certain", Philip Crosby demonstrates that
  the costs of conformance plus the costs of
  nonconformance due to internal failures is less
  than the costs of nonconformance due to external
  failures (until hat limit?)
• (source Ron Patton, "Software testing")

costs of conformance
costs of nonconformance due to external failures
lt

costs of nonconformance due to internal failures
28
Quality costsThe optimal amount of testing

• (source "Software Testing", Ron Patton)

29
Quality-related activities (1)

• Software Verification and Validation (V V)
• Goals
• Establish the existence of defects in a product
• Assess whether or not the product is usable in an
  operational situation
• Verification
• Ensure that we are building the product right,
  i.e., according to its specification
• Validation
• Ensure that we are building the right product,
  i.e., according to user needs
• V V are integral part of the development
  process
• Concerned directly with product quality

30
Quality-related activities (2)

• Software Quality Assurance/Management (SQA/SQM)
• Goals Ensure that the required level of quality
  is achieved in software products, namely, that
  defined standards and procedures are followed
• SQA should aim to develop a quality culture
  where quality is seen as everyones
  responsibility
• Sub-activities
• (Organizationwide) Quality standards definition
• Establish organisational procedures and standards
  for quality. Produce a quality manual
• (Projectwide) Quality planning
• Select applicable procedures and standards for a
  particular project and modify these as required.
  Produce a quality plan.
• (Projectwide) Quality control
• Ensure that procedures and standards are followed
  by the software development team. Produce quality
  review reports
• Quality management should be separate from
  project management to ensure independence of
  budget and schedule pressures
• Concerned directly with process quality and,
  indirectly, product quality

31
Quality management and software development
deliverables

• (source Ian Sommerville)

32
Main approaches for VV and QA

• Tests
• Dynamic technique, concerned with exercising and
  observing product behaviour to discover defects
• The system is executed with test data (defined
  test cases) and its operational behaviour is
  observed to discover defects (differences between
  observed and expected)
• Used mainly for VV
• GUI testing difficult to automate API testing
  easier to automate
• Inspections and reviews
• Static technique - concerned with the analysis of
  the static system representation (source code,
  documentation, ) to discover problems
• May be supplement by tool-based document and code
  analysis
• Measurements
• The value of defined metrics is automatically
  measured on selected components of the product,
  for prediction or control purposes
• Used mainly for QA
• All involve planning, execution and result
  analysis and reporting

33
Typical tests and reviews

• (source "Software Project Survival Guide", Steve
  McConnell)

34
VV versus SQA

• "The goal of a software tester is to find bugs,
  find them as early as possible, and make sure
  that they get fixed"
• "A software quality assurance person's main
  responsibility is to create and enforce standards
  and methods to improve the development process
  and to prevent bugs from ever occurring"
• "A software quality assurance person's main
  responsibility is to examine and measure the
  current software development process and find
  ways to improve it with a goal of preventing bugs
  from ever occurring"
• (source Ron Patton)

35
Business Process Management and Improvement and
Quality Management

• A software development process is the main
  business process in a software development
  business
• A business process is an organized set of
  activities that produces a result with value
  (product or service) to a costumer (internal or
  external)
• Business process management (BPM) comprises
• process modeling and definition
• includes the definition of key performance
  indicators (KPI's), for process quality and
  performance evaluation
• process implementation, execution and operation
• process measuring and evaluation
• process improvement and re-engineering
• Business Process Management (BPM) and Quality
  Management (QM) are closely related
• Process improvement is essential to achieve
  product (and service) improvement

36
The Six Sigma Concept

• One of the primary quality initiatives of our
  time, pioneered by Motorola and popularized by
  General Electric
• The name, Six Sigma, is taken from the approachs
  statistical roots. If a product or process has a
  six sigma level of consistency, then it is
  experiencing only 3.4 defects per million. In
  other words, six sigma products and processes are
  99.99966 consistent
• Has been billed as a critical business tool for
  the 21st century
• In a fast changing business environment,
  companies have used Six Sigma initiatives to gain
  improvements in process and product quality
• The success that companies have enjoyed lies in
  the ability to tie project results directly to
  improved customer satisfaction ratings and bottom
  line savings.
• In order to reach these results, Six Sigma
  efforts pursue the following five objectives
• Define the problem area in objective terms
• Measure the performance of products and processes
• Analyze the problems to identify root causes
• Improve the results by redesigning processes and
  reducing variation
• Control the processes to ensure the improvements
  are permanent
• (source http//www.proformacorp.com/solutions/six
  sigma.asp )

37
The PDCA Cycle as a Framework for Improvement

• Originally conceived by Walter Shewhart in
  1930's, and later adopted by W. Edwards Deming
• Is a model that provides a framework for the
  improvement of a process, product or system
• Plan a change or a test, aimed at improvement
• Analyze what you intend to improve, looking for
  areas that hold opportunities for change
• Choose areas that offer the most return for the
  effort you put in
• Do - Carry out the change or test (preferably on
  a small scale)
• Check the results. What was learned? What went
  wrong?
• After you have implemented the change for a short
  time, you must determine how well it is working.
• Is it really leading to improvement in the way
  you had hoped?
• Decide on several measures with which you can
  monitor the level of improvement.
• Act - Adopt the change, abandon it, or run
  through the cycle again
• After planning a change, implementing and then
  monitoring it, you must decide whether it is
  worth continuing that particular change. If it
  consumed too much of your time, was difficult to
  adhere to, or even led to no improvement, you may
  consider aborting the change and planning a new
  one. However, if the change led to a desirable
  improvement or outcome, you may consider
  expanding the trial to a different area, or
  slightly increasing your complexity. This sends
  you back into the Plan phase and can be the
  beginning of the ramp of improvement.
• The PDCA cycle may be applied to all sorts of
  management activities
• project management plan do monitor - control

(source http//www.dartmouth.edu/ogehome/CQI/PDC
A.html )
38
References and further reading

• "Software Testing", Ron Patton, SAMS, 2001
• "Software Project Survival Guide", Steve
  McConnell, Microsoft Press, 1998
• "Testing Computer Software", 2nd Edition, Cem
  Kaner, Jack Falk, Hung Nguyen, John Wiley Sons,
  1999
• "Software Engineering", Ian Sommerville,
  6th Edition, Addison-Wesley, 2000 Ian Sommervile
• "Practical Software Testing", Ilene Burnstein,
  Springer-Verlag, 2003
• http//computing-dictionary.thefreedictionary.com

39
Notes after class (2 March 2004)

• Quality fitness for purpose / fitness for use
• PT adequação ao fim a que se destina
• Configuration and change management and
  problem/defect/bug tracking have critical
  importance for quality improvement
• bugs found have to be fixed and retested, and it
  is fundamental to keep track of the relationships
  between tests performed, versions, changes, bugs
  found, bugs fixed, etc.
• Requirements development and management have
  critical importance for quality improvement
• Most defects have their origin in requirements
• See CMMI description (TQS1.ppt) for terminology,
  quality related activities and process improvement

40
Notes after class (2 March 2004)

• How to apply the six sigma concept and defect
  rate (3.4 defects per million) to software?
• Defect rate in software number of bugs per KLOC
  (1000 Lines Of Code)
• Beizer's (1990) review estimates 10 to 30 bugs
  per 1000 executable statements (before testing)
• assuming 1 statement per LOC, this gives 10-30
  bugs/KLOC
• Typical defect rates in MS applications
• 10-20 defects/KLOC during unit testing
• 0.5 defects/KLOC after release
• What is your typical defect rate?
• According to Cem Kaner, "One common definition of
  a software error is a mismatch between the
  program and its specification. DON'T USE THIS
  DEFINITION. A mismatch between the program and
  its specification is an error in the program if
  and only if the specification exists and is
  correct. A program that follows a terrible
  specification perfectly is terrible, not perfect.
  A better definition is A software error is
  present when the program does not do what its end
  user reasonably expected it to do."