Software reliability

1
Software reliability
2
Software reliability(1)

• Formal specification and proof do not guarantee
  that the software will be reliable in practical
  use.
• The specification may not reflect the real
  requirements of system users.
• The proof may contain errors.
• The proof may assume a usage pattern which is
  incorrect.
• Increasing reliability can dramatically increase
  development costs.

3
Software reliability(2)

• Reliability should take precedence over
  efficiency.
• Computers are now cheap and fast
• Unreliable software is liable to be discarded by
  users
• System failure costs may be enormous
• Unreliable systems are difficult to improve
• Inefficiency is predictable
• Unreliable systems may cause information loss

4
Reliability metrics

• Metrics which have been used for software
  reliability specifications.
• POFOD(Probability of failure on demand) a
  measure of the likelihood that the system will
  fail when a service request is made.
• ROCOF(Rate of failure occurrence) a measure of
  the frequency of occurrence with which unexpected
  behavior is likely to occur.
• MTTF(Mean time to failure) a measure of the
  time between observed system failures.
• AVAIL(Availability) a measure of how likely the
  system is to be available for use.

5
Software reliability specification(1)

• The required level of reliability should be
  expressed quantitatively in the software
  requirement specification.
• The software shall be as reliable as possible.
• The software shall exhibit no more than N
  faults/1000 lines.
• Failure class
• Transient Occurs only with certain inputs
• Permanent Occurs with all inputs
• Recoverable System can recover without operator
  intervention
• Unrecoverable Operator intervention needed to
  recover from failure.

6
Software reliability specification(2)

• Non-corrupting Failures does not corrupt system
  state or data
• Corrupting Failure corrupts system state or
  data
• Steps for establishing a reliability
  specification
• For each identified sub-system, identify the
  different types of system failure which may occur
  and analyze the consequences of these failures.
• From the system failure analysis, partition
  failures into appropriate classes. A reasonable
  starting point is to use the failure types.
• For each failure class identified, define the
  reliability requirement using the appropriate
  reliability metric. It is not necessary to use
  the same metric for different classes of failure.

7
Software reliability specification(3)

• Example Auto-teller system in the network
• Failures that affect a single machine in the
  network
• Failures that affect the database and therefore
  all ATMs in the network
• Possible failure classes and reliability
  specifications
• Permanent, non-corrupting the system fails to
  operate with any card which is input. Software
  must be restarted to correct failure ROCOF
• Transient, non-corrupting the magnetic stripe
  data cannot be read on an undamaged card which is
  input PODOF
• Transient, corrupting a pattern of transactions
  across the network causes database corruption

8
Statistical testing(1)

• Software testing process in which the objective
  is to measure the reliability of the software.
• Steps in statistical testing
• Determine the operational profile of the
  software.
• Operational profile consists of a specification
  of classes of input and the probability of their
  occurrence.
• Select or generate a set of test data
  corresponding to the operational profile.
• Apply these test cases to the program, recording
  the amount of execution times between each
  observed system failure.
• After a statistically significant number of
  failures have been observed, the software
  reliability can then be computed.

9
Statistical testing(2)

• Difficulties? statistical test? ?? ??? ???.
• Operational profile uncertainty
• High costs of operational profile generation
• Statistical uncertainty when high reliability is
  specified

Number of inputs
Input classes
10
Reliability growth modeling(1)

• Reliability growth model
• mathematical model of software reliability
• predicts how software reliability should improve
  over time as faults are discovered and repaired.
• Step function model
• reliability increases by a constant increment
  each time a fault is discovered and repaired.
• assumes that software repairs are always correct
  and never increase the number of faults present
  in the software.
• also assumes that all faults contribute equally
  to reliability.

11
Reliability growth modeling(2)

• Random-step function model
• introduces a random element into the reliability
  growth improvement effected by a software repair.
  
• negative reliability growth
• The software is tested using a statistical
  approach and the reliability is measured.