CSE 495

1
Advanced Topics

• Cleanroom Software Engineering
• Quality Assurance

2
Cleanroom Overview

• What is it?
• Motivations
• Activities
• Phased Implementation
• Concerns

3
Cleanroom - What is it?

• A method for developing software with known and
  predictable reliability.
• Combines rigorous methods of software
  specification, design, correctness verification,
  and statistical quality certification in a new
  life cycle based on incremental development.

4
Cleanroom--What is it? (cont)

• A software development philosophy which is based
  on static verification techniques. The objective
  of this approach to software development is
  zero-defect software.
• Based on the notion that defects in software
  should be avoided rather than detected and
  repaired. It relies on a strict, formalized
  inspection process to discover faults before the
  software is tested.

5
Cleanroom Motivations

• Software with known Mean Time to Failure
• Software with near zero defects
• Quality software which is cheaper to produce

Dyer 92
6
Cleanroom Activities

• Formal Specification
• Increment Planning
• Formal Design
• Correctness Verification
• Statistical Testing
• Software Reliability Measurement

7
Formal Specification

• Formal methods (box structuring techniques, Z,
  VDM, etc.) force a more exacting analysis of the
  requirements and tend to minimize ambiguity,
  inconsistency, and incompleteness in the
  resultant software specification.
• Two specifications
• Functional
• Usage

8
Increment Planning

• The software is partitioned into increments which
  are developed separately using the cleanroom
  process.
• An increment is a useful, albeit limited, system
  in its own right.
• Users can feed back reports of the system and
  propose changes that are required.
• Important because it minimizes the disruption
  caused to the development process by
  customer-requested requirements changes.

9
Formal Design

• A rigorous and formal design method is a
  necessary element for generating software whose
  correctness can be verified.
• A structured programming process, using only a
  limited number of control and data abstraction
  constructs, is recommended.
• The program development process is a process of
  stepwise refinement of the specification.

10
Correctness Verification

• Correctness is defined as the equivalence between
  a requirement and the design, documented with the
  design primitives, which supposedly implements
  the requirement.
• Designs are verified first by the designer when
  constructing a design and subsequently by
  independent inspectors when reviewing the design.
• With some algebraic manipulation, the question of
  correctness for a total software product can be
  reduced to the summation of the correctness
  proofs for the component parts.

11
Statistical Testing

• Statistical testing is a software testing process
  in which the objective is to measure the
  reliability of the software rather than to
  discover software faults.
• Determine the operational profile
• Select or generate a set of test data
  corresponding to the operational profile
• Apply these test cases to the program, recording
  the amount of execution time between each
  observed system failure
• After a statistically significant number of
  failures have been observed, the software
  reliability can be computed.
• Testing the software the way users intend to use
  it.

12
Statistical Testing - Difficulties

• Operational profile uncertainty
• Difficult to anticipate usage for new or
  innovative systems
• System usage can change over time
• Many systems have many improbable, but not
  impossible, conditions
• High costs of operational profile generation
• Statistical uncertainty when high reliability is
  specified
• It may be hard to generate (especially
  automatically) test data to create rare, but
  valid, inputs

13
Software Reliability Measurement

• Software quality is often represented in errors
  per thousand LOC (KSLOC)
• This is useful to the developer, but of limited
  value to the user
• Software reliability, in terms of mean time to
  failure (MTTF), is a more meaningful measure for
  the user, which gives a positive quality
  indicator (longer MTTF is better) and which can
  be estimated prior to software delivery.

Dyer 92
14
Phased Cleanroom Implementation
1.
Configuration Managed Software
Baseline Process
Statistical Process Control
Statistical Testing
Reliability Measurement
Formal Specifications
Dyer 92
15
Cleanroom Concerns

• Too mathematical and not usable in an environment
  of changing requirements
• Developer unit testing is an essential step in
  software development and can not be eliminated
• Randomized testing was neither a cost effective
  method nor could it provide adequate requirements
  coverage
• Distrust in the concept of a software MTTF and in
  its role in the software development process

Dyer 92
16
Cleanroom - Reasons for Slow Growth

• A belief that correctness verification is too
  theoretical to be usable in real software
  development
• It advocates no developer testing and replaces it
  with correctness verification and statistical
  quality control -- concepts opposite of how most
  software is developed today
• Maturity of the software development industry

Henderson 95
17
Summary

• What is it?
• Motivations
• Activities
• Phased Implementation
• Concerns