Software Safety: An Oxymoron - PowerPoint PPT Presentation

Loading...

PPT – Software Safety: An Oxymoron PowerPoint presentation | free to download - id: 114f55-OWZmY



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

Software Safety: An Oxymoron

Description:

Software Safety: An Oxymoron? March 29, 2007. Ken Wong, Ph.D., Senior Systems Analyst ... Program testing can be used to show the presence of bugs, but never to ... – PowerPoint PPT presentation

Number of Views:166
Avg rating:3.0/5.0
Slides: 36
Provided by: hpcus673
Learn more at: http://www.vanq.org
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Software Safety: An Oxymoron


1
Software SafetyAn Oxymoron?
  • March 29, 2007
  • Ken Wong, Ph.D., Senior Systems Analyst
  • McKesson Medical Imaging Group

2
Points to Ponder
  • A system can be correct and reliable and yet
    unsafe
  • Software safety is not about bugs
  • Program testing can be used to show the presence
    of bugs, but never to show their absence
  • We will return to these statements in the
    discussion

3
Outline
  • Introduction to Software Safety
  • Software Meet System Safety
  • System Safety Meet Software
  • Verifying Software Safety

4
Introduction toSoftware Safety
5
Software In the Real World
  • Therac 25 accidents
  • Ariane 5 Flight 501 explosion
  • Titan 4 Centaur/Milstar failure
  • TCAS collision near Uberlingen, Germany

6
Ariane 501
7
Ariane 501 Events
  • Destruction of Ariane 501 on 4 June 1996 (from
    final report)
  • nominal behaviour of the launcher up to H0 36
    seconds
  • failure of the back-up Inertial Reference System
    (SRI) followed immediately by failure of the
    active SRI

8
Building Dependable Software
Quality
Safety
Correctness
Reliability
Security
9
Safety is a Distinct Property
  • Safety is a distinct part of the interlocking
    puzzle of how to build dependable software
  • A system can be correct and reliable and yet
    unsafe!
  • Improved software process alone does not mean a
    safer system
  • Note These can be a contentious claims even
    among safety engineers.

10
Safety is
avoiding mishaps!
11
Software Meet System Safety
12
Is it Safe?
Christian Szell Is it safe? Babe Yes, it's
safe, it's very safe, it's so safe you wouldn't
believe it. - Marathon Man 1976
13
System Safety
  • System Safety is a systematic approach to
    safety primarily developed in the US for the
    aerospace and defense industries
  • Spreading to other industries, e.g., health care
  • Focus on managing system hazards
  • E.g., FDA Quality System Regulation recommends
    risk analysis (A.K.A. hazard analysis)

14
System Safety
Risk Assessment
Safety Verification
15
Hazard
  • A hazard is the systems potential contribution
    to a mishap
  • E.g., brake failure, engine overheating
  • Key is understanding the system environment

16
Hazards and Mishaps
mishap
hazard
hazard causes
System
Environment
17
Ariane 501 SRI Bug?
  • Uncaught exception from floating point conversion
  • From high value of BH (Horizontal Bias)
  • Programming 101!
  • Conversion check deliberately removed for
    performance reasons
  • SRI reused from Ariane 4
  • Check not required for Ariane 4 trajectory

18
Safety is a System Property
  • SRI worked exactly as specified for Ariane 4!
  • Ariane 5 trajectory different from Ariane 4
  • SRI spec did NOT include Ariane 5 trajectory data
  • SRI NOT tested with Ariane 5 trajectory data
  • Safety cannot be understood without knowing the
    operational environment
  • FDA use-related vs device failure hazards
  • E.g., TCAS collision in Germany

19
When Software Met Safety
  • there was a definite risk in assuming that
    critical equipment such as the SRI had been
    validated by qualification on its own, or by
    previous use on Ariane 4.
  • ARIANE 5 Flight 501 Failure Report

20
System Safety Meet Software
21
In the beginning (or Europe)
  • Mechanical systems with well understood designs
  • Hazards caused by component failure from random
    hardware faults
  • Mitigation through integrity and redundancy
  • Myth, but there is underlying truth in all good
    myths

22
Fault Tree Analysis
Basic Event
Steering Fails
Intermediate Event
OR
Steering Assembly Fails
Driver Error
OR
OR
Steering Wheel Fails
Drive Shaft Fails
Steering Control Software Fails
23
Is Software Another Component?
  • What is the probability that the steering control
    software fails?
  • If software is just another component
  • Software cannot wear out or breakdown like a
    mechanical component
  • Only fault is a programming bug
  • Assuming programmers do their job, failure rate
    should be zero
  • Paraphrased from talk by a system safety engineer

24
Software Revealed
Basic Event
Steering Fails
Intermediate Event
OR
Steering Assembly Fails
Driver Error
OR
OR
Steering Wheel
Drive Shaft Fails
Steering Control Software Fails
25
The Software Werewolf
  • Of all the monsters that fill the nightmares
    of our folklore, none terrify more than
    werewolves, because they transform unexpectedly
    from the familiar into horrors The familiar
    software project, at least as seen by the
    nontechnical manager, has something of this
    character
  • Frederick P. Brooks, Jr. from No Silver Bullet
    Essence and Accidents of Software Engineering

26
Ariane 501 Safety in Numbers?
  • In response to fault, the Primary SRI was
    deliberately shutdown
  • Attempt made to switch to backup SRI
  • Typical strategy in face of random failures
  • However, BOTH SRIs shutdown!
  • Fault due to same design in both SRIs
  • Exception in non-essential component

27
Safety is an Emergent Property
  • Software safety is not about faults
  • Many potential faults but not all created equal
    most have no impact on safety
  • Correct behaviour can contribute to the hazard!
  • Hazards can emerge from complex interactions
    between correct components

28
When Safety Met Software
  • An underlying theme in the development of Ariane
    5 is the bias towards the mitigation of random
    failure.
  • Board wishes to point out that software is an
    expression of a highly detailed design and does
    not fail in the same sense as a mechanical
    system.
  • ARIANE 5 Flight 501 Failure Report

29
Verifying Software Safety
30
Software and Safety Process
Hazards
Requirements
Hazard ID, Analysis and Mitigation
Design
Safety Verification
Verification
Source Code
31
Limits of Testing
  • Program testing can be used to show the
    presence of bugs, but never to show their absence
  • E. Dijkstra in Structured Programming

32
Hazard-Driven Testing
  • Focus on hazard force it to occur
  • Consider
  • Hazard risk (risk-based testing)
  • Mishap scenarios
  • Hazard causes identified during hazard analysis
  • Problem reports/issues with safety implications
  • See Jeffrey J. Joyce and Ken Wong, Hazard-driven
    Testing of Safety-Related Software

33
Summary and Conclusions
  • Safety is a distinct property
  • Safety is a system property
  • Operational and development environment factors
  • Safety is an emergent property
  • Hazards can emerge from complex interactions
    between correct components

34
Safety and Software Happy Together?
35
References
  • ARIANE 5 Flight 501 Failure Report by the Inquiry
    Board, Paris, July 1996
  • Frederick P. Brooks, Jr., No Silver Bullet
    Essence and Accidents of Software Engineering,
    Computer Magazine, April 1987
  • Jeffrey J. Joyce and Ken Wong, Hazard-driven
    Testing of Safety-Related Software, 21st
    International System Safety Conference, Ottawa,
    Ontario, August 4-8, 2003
  • All available on-line
About PowerShow.com