Developing Dependable Systems by Maximizing Component Diversity and Fault Tolerance - PowerPoint PPT Presentation

Loading...

PPT – Developing Dependable Systems by Maximizing Component Diversity and Fault Tolerance PowerPoint presentation | free to download - id: 83c081-OGQ0O



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

Developing Dependable Systems by Maximizing Component Diversity and Fault Tolerance

Description:

Developing Dependable Systems by Maximizing Component Diversity and Fault Tolerance Jeff Tian, Suku Nair, LiGuo Huang, Nasser Alaeddine and Michael Siok – PowerPoint PPT presentation

Number of Views:72
Avg rating:3.0/5.0
Slides: 33
Provided by: smue2
Learn more at: http://lyle.smu.edu
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Developing Dependable Systems by Maximizing Component Diversity and Fault Tolerance


1
Developing Dependable Systems by Maximizing
Component Diversity and Fault Tolerance
  • Jeff Tian, Suku Nair, LiGuo Huang,
  • Nasser Alaeddine and Michael Siok
  • Southern Methodist University
  • US/UK Workshop on Network-Centric Operation and
    Network Enabled Capability, Washington, D.C.,
    July 24-25, 2008

2
Outline
  • Overall Framework
  • External Environment Profiling
  • Component Dependability
  • Direct Measurement and Assessment
  • Indirect Assessment via Internal Contributor
    Mapping
  • Value Perspective
  • Experimental Evaluation
  • Fault Injection for Reliability and Fault
    Tolerance
  • Security Threat Simulation
  • Summary and Future Work

3
Overall Framework
  • Systems made up of different components
  • Many factors contribute to system dependability
  • Our focus Diversity of individual components
  • Component strength/weakness/diversity
  • Target Different dependability attributes and
    sub-attributes
  • External reference Operational profile (OP)
  • Internal assessment Contributors to
    dependability
  • Value perspective Relative importance and
    trade-off
  • Maximize diversity gt Maximize dependability
  • Combine strength
  • Avoid/complement/tolerate flaws/weaknesses

4
Overall Framework (2)
  • Diversity Four Perspectives
  • Environmental perspective Operational profile
    (OP)
  • Target perspective Goal, requirement
  • Internal contributor perspective Internal
    characteristics
  • Value perspective Customer
  • Achieving diversity and fault tolerance
  • Component evaluation matrix per target per OP
  • Multidimensional evaluation/composition via DEA
    (Data Envelopment Analysis)
  • Internal contributor to dependability mapping
  • Value-based evaluation using single objective
    function

5
Terminology
  • Quality and dependability are typically defined
    in terms of conformance to customers
    expectations and requirements
  • Key concepts defect, failure, fault, and error
  • Dependability the focus in this presentation
  • Key attributes reliability, security, etc.
  • Defect some problem with the software
  • either with its external behavior
  • or with its internal characteristics

6
Failure, Fault, Error
  • IEEE STD 610.12 terms related to defect
  • Failure The inability of a system or component
    to perform its required functions within
    specified requirements
  • Fault An incorrect step, process, or data
    definition in a computer program
  • Error A human action that produces an incorrect
    result
  • Errors may cause faults to be injected into the
    software
  • Faults may cause failures when the software is
    executed

7
Reliability and Other Dependability Attributes
  • Software reliability the probability for
    failure-free operation of a program for a
    specified time under a specified set of operating
    conditions (Lyu, 1995 Musa et al., 1987)
  • Estimated according to various model based on
    defect and time/input measurements
  • Standard definitions for other dependability
    attributes, such as security, fault tolerance,
    availability, etc.

8
Outline
  • Overall Framework
  • External Environment Profiling
  • Component Dependability
  • Direct Measurement and Assessment
  • Indirect Assessment via Internal Contributor
    Mapping
  • Value Perspective
  • Experimental Evaluation
  • Fault Injection for Reliability and Fault
    Tolerance
  • Security Threat Simulation
  • Summary and Future Work

9
Diversity Environmental Perspective
  • Dependability defined for a specific environment
  • Stationary vs dynamic usage environments
  • Static, uniform, or stationary (reached an
    equilibrium)
  • Dynamic, changing, evolving, with possible
    unanticipated changes or disturbances
  • Single/overall OP for former category
  • Musa or Markov variation
  • Single evaluation result possible per component
    per dependability attribute e.g., component
    reliability R(i)
  • Environment Profiling for Individual Components
  • Environmental snapshots captured in Musa or
    Markov Ops
  • Evaluation matrix (later)

10
Operational Profile (OP)
  • Operational profile (OP) is a list of disjoint
    set of operations and their associated
    probabilities of occurrence (Musa 1998)
  • OP describes how users use an application
  • Help guide the allocation of test cases in
    accordance with use
  • Ensure that the most frequent operations will
    receive more testing
  • As the context for realistic reliability
    evaluation
  • Other usages, including diversity and
    internal-external mapping in this presentation

11
Markov Chain Usage Model
  • Markov chain usage model is a set of states,
    transitions, and the transition probabilities
  • As an alternative to Musa (flat) OP
  • Each link has an associated probability of
    occurrence
  • Models complex and/or interactive systems better
  • Unified Markov Models (Kallepalli and Tian, 2001
    Tian et al., 2003)
  • Collection of Markov Ops in a hierarchy
  • Flexible application in testing and reliability
    improvement

12
Operational Profile DevelopmentStandard
Procedure
  • Musas steps (1998) for OP construction
  • Identify the initiators of operations
  • Choose a representation (tabular or graphical)
  • Create an operations list
  • Establish the occurrence rates of the individual
    operations
  • Establish the occurrence probabilities
  • Other variations
  • Original Musa (1993) 5 top-down refinement steps
  • Markov OP (Tian et al) FSM then probabilities
    based on log files

13
OPs for Composite Systems
  • Using standard procedure whenever possible
  • For overall stationary environment
  • For individual component usage gt component OP
  • For dynamic environment
  • Snapshot identification
  • Sets of OPs for each snapshot
  • System OP from individual component OPs
  • Special considerations
  • Existing test data or operational logs can be
    used to develop component OPs
  • Union of component OPs gt system OP

14
OP and Dependability Evaluation
  • Some dependability attributes defined with
    respect to a specific OP e.g., reliability
  • For overall stationary environment direct
    measurement and assessment possible
  • For dynamic environment OP-reliability pairs
  • Consequence of improper reuse due to different
    OPs (Weyuker 1998)
  • From component to system dependability
  • Customization/selection of best-fit OP for
    estimation
  • Compositional approach (Hamlet et al, 2001)

15
Outline
  • Overall Framework
  • External Environment Profiling
  • Component Dependability
  • Direct Measurement and Assessment
  • Indirect Assessment via Internal Contributor
    Mapping
  • Value Perspective
  • Experimental Evaluation
  • Fault Injection for Reliability and Fault
    Tolerance
  • Security Threat Simulation
  • Summary and Future Work

16
Diversity Target Perspective
  • Component Dependability
  • Component reliability, security, etc. to be
    scored/evaluated
  • Direct Measurement and Assessment
  • Indirect Assessment (later)
  • Under stationary environment
  • Dependability vector for each component
  • Diversity maximization via DEA (data envelopment
    analysis)
  • Under dynamic environment
  • Dependability matrix for each component
  • Diversity maximization via extended DEA by
    flattening out the matrix

17
Diversity Maximization via DEA
  • DEA (data envelopment analysis)
  • Non-parametric analysis
  • Establishes a multivariate frontier in a dataset
  • Basis linear programming
  • Applying DEA
  • Dependability attribute frontier
  • Illustrative example (right)
  • N-dimensional hyperplane

18
DEA Example
  • Lockheed-Martin software project performance with
    regard to selected metrics and production
    efficiency model
  • Measures efficiencies of decision making units
    (DMU) using weighted sums of inputs and weighted
    sums of outputs
  • Compares DMUs to each other
  • Sensitivity analysis affords study of
    non-efficient DMUs in comparison
  • BCC VRS Model used in initial study

19
DEA Example (2)
  • Using production efficiency model for
    Compute-Intensive dataset group
  • Ranked set of projects
  • Data showing distance and direction from
    efficiency frontier

20
Diversity Internal Perspective
  • Component Dependability
  • Direct Measurement and Assessment might not be
    available, feasible, or cost-effective
  • Indirect Assessment via Internal Contributor
    Mapping
  • Internal Contributors
  • System design, architecture
  • Component internal characteristics size,
    complexity, etc.
  • Process/people/other characteristics
  • Usually more readily available data/measurements
  • InternalgtExternal mapping
  • Procedure with OP as input too (e.g.,
    faultgtreliability)

21
Example Fault-Failure Mapping for Dynamic Web
Applications
22
Web Example Fault-Failure Mapping
  • Input to analysis (and fault-failure conversion)
  • Anomalies recorded in web server logs (failure
    view)
  • Faults recorded during development and
    maintenance
  • Defect impact scheme (weights)
  • Operational profile
  • Product A is an ordering web application for
    telecom services
  • Consists of hundreds of thousands of lines of
    code
  • Running on IIS 6.0 (Microsoft Internet
    Information Server),
  • Process couple of millions requests per day

23
Web Example Fault-Failure Mapping (Step 1)
  • Pareto chart for the defect classification of
    product A
  • The top three categories represent 66.26 of the
    total defect data

24
Web Example Fault-Failure Mapping (Steps 4 5)
Number of Hits with response code 200 and 300 235142
Average Number of hits per transaction 40
Number of transactions 5880
Operation Operation Probability Number of Transactions
New order 0.1 588
Change order 0.35 2058
Move order 0.1 588
Order Status 0.45 2646
  • OP for product A and the corresponding numbers
    of transactions.

25
Web Example Fault-Failure Mapping (Step 6)
  • Using the number of transactions calculated from
    OP and the defined fault impact schema, we
    calculated the fault exposure or corresponding
    potential failure frequencies

Application Aspect Impact Weight Number of transactions Failure Frequency
Order status Showstopper 100 2646 2646
Order status High 70 2646 1852
Order status Medium 50 2646 1323
Order status Low 20 2646 529
Order status Exception 5 2646 132
26
Web Example Fault-Failure Mapping (Step 7)
Rank Response Code Fault Failure Frequency
1 404 /images/dottedsep.gif 5805
2 404 /images/gnav_redbar_s_r.gif 3687
3 404 /images/gnav_redbar_s_l.gif 3537
4 200/300 Order status showstopper 2646
5 404 /includes/css/images/background.gif 2593
6 200/300 Change order- showstopper 2058
7 200/300 Order status high 1852
8 200/300 Change order high 1441
9 200/300 Order status medium 1323
10 200/300 Change order medium 1029
11 404 /includes/css/nc2004style.css 721
27
Web Example Fault-Failure Mapping (Result
Analysis)
  • A large number of failures were caused by a small
    number of errors with high usage frequencies
  • Fixing faults with a high usage frequency and a
    high impact could achieve better efficiency in
    reliability improvement
  • By fixing the top 6.8 faults, the total failures
    were reduced by about 57
  • Similarly, 10 -gt 66, 15-gt71, 20-gt75, for
    top-faults induced failure reduction
  • Defect data repository and web server log
    recorded failures have insignificant overlap gt
    both are needed for effective reliability
    improvement

28
Diversity Value Perspective
  • Component Dependability Attribute
  • Direct Measurement and Assessment might not
    capture what customers truly care about
  • Different value attached to different
    dependability attributes
  • Value-based software quality analysis
  • Quantitative model for software dependability ROI
    analysis
  • Avoid one-size-fits-all
  • Value-based process experience at NASA/USC
    (Huang and Boehm) extend to dependability
  • Mapping to value-based perspective more
    meaningful to target customers

29
Value Maximization
  • Single objective function
  • Relative importance
  • Trade-off possible
  • Quantification scheme
  • Gradient scale to selecte component(s)
  • Compare to DEA
  • General cases
  • Combination with DEA
  • Diversity as a separate dimension possible

30
Outline
  • Overall Framework
  • External Environment Profiling
  • Component Dependability
  • Direct Measurement and Assessment
  • Indirect Assessment via Internal Contributor
    Mapping
  • Value Perspective
  • Experimental Evaluation
  • Fault Injection for Reliability and Fault
    Tolerance
  • Security Threat Simulation
  • Summary and Future Work

31
Experimental Evaluation
  • Testbed
  • Basis OPs
  • Focus on problems and system behavior under
    injected or simulated problems
  • Fault Injection for Reliability and Fault
    Tolerance
  • Reliability mapping for injected faults
  • Use of fault seeding models
  • Direct fault tolerance evaluation
  • Security Threat Simulation
  • Focus 1 likely scenarios
  • Focus 2 coverage via diversity

32
Summary and Future Work
  • Overall Framework
  • External Environment Profiling
  • Component Dependability
  • Direct Measurement and Assessment
  • Indirect Assessment via Internal Contributor
    Mapping
  • Value Perspective
  • Experimental Evaluation
  • Fault Injection for Reliability and Fault
    Tolerance
  • Security Threat Simulation
  • Summary and Future Work
About PowerShow.com