CmpE 550 Advanced Software Engineering - PowerPoint PPT Presentation


PPT – CmpE 550 Advanced Software Engineering PowerPoint presentation | free to download - id: 62f8b-ZDc1Z


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation

CmpE 550 Advanced Software Engineering


'Scenarios are applied to the analysis of ... Slips: Attention based lapses and omissions in skilled behavior ... The Model Controller controls the BN models. ... – PowerPoint PPT presentation

Number of Views:23
Avg rating:3.0/5.0
Slides: 37
Provided by: selimoz


Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: CmpE 550 Advanced Software Engineering

CmpE 550 Advanced Software Engineering
Scenario-Based Assessment of Nonfunctional
Selim Özyilmaz Elif Sürer
  • Purpose Validation of Requirements Specification
    using Scenarios
  • Scenarios are applied to the analysis of
    non-functional requirements using dependency
    tables to assess the relationship between goals
    (functional and non-functional requirements) and
    the agents and tasks that achieve them in the i

  • Many NFRs are influenced by human properties,
    they inherit the diverse nature of human
    characteristics (System Reliability is influenced
    by human characteristics such as ability)
  • This work is a revised version of a previous one
    that prompted designers with questions about
    potential problems in a scenario event sequence.
    (Psychology-based taxonomy of failure)

  • The problem is that too many scenario variations
    were generated. To prevent, transform the
    taxonomy of human and system failures into a
    model to predict the errors in the system
  • Bayesian Belief Nets (BNs) are used to predict
    the reliability.

Related Work
  • Model-checking techniques have been used
    extensively to verify and validate requirements.
  • Communication problem occurs between
    user-stakeholders and the model developers.

Software Cost Reduction (SCR)
tabular representation
Related Work
Related Work
  • A combination of visualizations, examples, and
    simulations is necessary to explain complex
    requirements to end users.
  • Scenario-based representations and animated
    simulations help users see the implications of
    system behavior and, thereby, improve
    requirements validation.

Related Work
Related Work
  • Animation simulation tools -gt by Dubois et al. in
    the ALBERT II language
  • KAOS language and supporting GRAIL tool which
    enable formal reasoning about dependencies
    between goal model, system behavior and
  • Animator-validator tool, TROLL, uses a formal
    object-oriented language for modeling information

Related Work
Related Work
  • What is a sufficient set of scenarios to enable
    validation to be completed with confidence?
  • While we believe there is no quick answer to this
    vexing problem, one approach is to automate the
    process as far as possible so more scenarios can
    be tested.

Related Work
Related Work
  • Intent specifications provide a hierarchical
    model to facilitate reasoning about system goals
    and requirements in safety critical systems.
  • Goals are decomposed in a means-ends hierarchy,
    widely practiced in requirements engineering.
  • Automated support for reasoning about conflicting
    system states and behavior is provided by the
    SpecTRM-RL tool.

Related Work
Related Work
  • Assessment of nonfunctional system requirements,
    such as system reliability, has to use
    probabilistic reasoning since the range of
    potential system behaviors is either unknown, in
    the early requirements phase, or too large to
  • Bayesian Nets (BNs) have been developed to assess
    software quality from properties of the code and
    software engineering process.

Related Work
Related Work
  • BNs have been widely applied as a probabilistic
    reasoning technique in software engineering and
    other domains however, previous work used single
    nets to evaluate a set of discrete states
    pertaining to a software product or development
  • More automated tools for scenario analysis of NFR
    conformance for requirements specifications with
    multiple BN tests have been developed.

Modeling Uncertainity
Modeling Uncertainty
  • There are four different techniques to model
  • Bayesian Probability
  • Dempster-Shafer
  • Fuzzy Sets
  • Possibility Theory
  • Bayesian Probability offers an easier combination
    of multiple influences on probability than
    Dempster-Shafer and a sounder reasoning than
    Fuzzy sets.
  • Bayesian Probability provides a decision theory
    of how to act on the world in an optimal fashion
    under circumstances of uncertainity.

Bayesian Belief Nets
  • Directed acyclic graphs of causal influences,
    where the nodes represent the variables and the
    arcs represent the relationships between
  • Variables can have any number of states in a BN,
    so the choice of measurement is left to the
  • Network Probability Table (NPT)

BN (Contd)
  • Input evidence are propagated through the network
    updating the values of other nodes
    (Computationally complex but effective algorithms

BN (Contd)
  • Three possible ways to compute the probability of
    each node in the network
  • Input a posterior probability into a BN as a
    prior observation ?each run should be assumed to
    be independent
  • Combine output probabilities from a sequential
    run using a summarizer ?probabilities of
    particular run has to be set initially
  • Output probability of each event is compared with
    a threshold value, if surpassed success else
  • Pinpoint some steps in the scenario which are
    weak in reliability
  • Sensitiviy analysis can be carried out with
    multiple BN runs by varying environmental

BN Model of System Reliability
  • Influencing factors are divided into two
  • Slips Attention based lapses and omissions in
    skilled behavior
  • Mistakes failures in plans and knowledge of
  • System environmental variables have an indirect
    effect on an individuals ability whereas
    organizational factors (management culture) have
    a direct effect.
  • High Cognitive Complexity ? more prone to
  • High Physical Complexity ?more prone to slip

System Reliability
  • Remarks
  • Input variables are all discrete states
  • The BN is a run with a range of scenarios that
    stress-test system design against operational
  • Scenarios can be either taken from
    domain-specific operational procedures or by
    interviewing with users
  • Two different outputs slips and mistakes

Operational Performance Time
  • Same variables with reliability case, the
    difference is that except two output nodes only
    one output node
  • The output is used to increase the best case task
    completion time to reflect the less than ideal
    properties of human and machine agents.
  • ET estimated time BT best task completion time
  • To reflect the case of reverting to manual when
    an automated technology fails, highly automated
    tasks worst completion time is set greater than
    the those of manual tasks

SRA System Architecture
  • Analysis starts with the selection of the i
    model to be evaluated and creating the test
  • Scenarios are narratives taken from real life
    experience describing operation of similar
    systems from which event sequences are extracted.

SRA Tools Components
  • The Session Controller implements the user
    command interface for selecting designs and
    scenarios and executes the algorithm that
    assesses a set of scenarios with the BNs. It
    calls the system reliability or operational
    performance BN assessors to execute the BN runs
    with all possible environmental combinations.

SRA Tools Components
  • The i model editor allows interactive
    construction of i models with typical CASE
    tool-type functions.
  • The Interactive Scenario Constructor produces
    test scenarios from the system model based on
    user directions. Scenarios are stored in a
    database in an array of tuples.

SRA Tools Components
SRA Tools Components
  • The Model Controller controls the BN models. It
    selects the appropriate BN model for each task
    step, then populates the input nodes, runs the
    model, and receives the belief distributions of
    the output nodes.
  • The Model Controller also manages the back
    propagation of the BN model to identify required
    technology and agent characteristics.

SRA Tools Components
SRA Tools Components
  • The BN assessor modules run the net by calling
    the HUGIN algorithm for each task step and for
    each set of environmental variable combinations.
  • The output from each run is compared with the
    desired NFR threshold and the survivor runs are
    passed to the results visualizer.

SRA Tools Components
SRA Tools Components
  • The Visualizer provides a visual summary of all
    qualified BN runs for a set of scenarios for one
    or more system designs.
  • This enables different designs to be compared and
    problem areas in the requirements to be
    identified, i.e., task/technical component
    combinations which show low potential NFR
    assessments. The Visualizer displays results at
    three levels System, Scenario, and Phase views.

NFR Analysis Method
NFR Analysis Method
  • Remarks
  • Scenarios are composed of a number of phases and
    each phase is composed of a number of task-steps
    each modeled as ltAgent,Task,Technologygt
  • Phases are used to structure task sequences that
    fulfill a higher order goal.
  • The best design is generally the one who has more
    surviving BN runs.
  • The best design also needs to be resilient to
    environmental conditions.

NFR Analysis Method
NFR Analysis Method
  • Impact of Environmental Variables
  • (1) Survivor runs with ER x set to best case
  • (2) Total survivor runs for all settings
  • If an overall design or a particular task fails
    to meet the threshold back propagation alaysis is
    used to discover the necessary settingto achieve
    the NFR value.
  • All input nodes are unconstrained (calculation
    for all)
  • One/few inputs are unconstrained (calculation for

Case Study
  • The application of the SRA tool in validating the
    operational performance and system reliability of
    a complex socio-technical system.
  • The requirements question is to assess the impact
    of new automated technology on the task of
    loading weapons on to aircraft in an aircraft

Case Study
Case Study
  • Tasks in Design 1 are manual or semiautomated,
    while, in Design 2, they are semi or fully
  • The second design saves manpower since it can be
    operated by one WA and is potentially more rapid
    to operate, but it is more expensive.

Case Study
  • When the operational performance times are
    compared, Design 2 is quicker for nearly all
    tasks, which is not surprising since it has more
    automated tasks.

Case Study
Case Study
  • The critical environmental variables for both
    designs, shows that incentives, motivation, duty
    time concurrency, and time constraints were all
    marked as vulnerable for Design 1.
  • Design 2 in contrast, fares better with only
    motivation, concurrency, and maintenance marked
    as vulnerable.

Case Study
Case Study
  • After identifying the most appropriate design,
    the problematic tasks, and the critical
    environmental variables, the analyst investigates
    the improvements required for the Autoload
    palette component, which was the weakest link in
    Design 2.

Validating the BN Models
  • Data mining techniques are used to test the
    assumptions in BN models.
  • All possible permutations are simulated and
    created a database of reliability and performance
    time predictions
  • Data mining Techniques
  • Relevance Analysis ranks input parameters of the
    model based on their relevance with output
  • Association Rules describe how often two or more
    facts cooccur in a dataset and were employed to
    check the causal associations in our model.
  • Classification partitions large quantities of
    data into sets with common characteristics and

Validating BN Models (Contd)
Validating the BN Models
  • Results
  • Sea state had only a minor influence on system
    error ? relevance analysis
  • IF (Duty-timehigh) ? survivedfail
  • IF (Workloadhigh) ? survivedfail (Association)
  • These rules indicate that causal influences of
    these variables are higher than assumed, alter
    NPT settings to overcome.
  • Crew motivation and agent ability problems ?

Discussion Conclusions
  • Automated testing of requirements specifications
    and designs for conformance to nonfunctional
    requirements using a set of scenarios and
    variations in the system environment have been

Discussion Conclusions
Discussion Conclusions
  • The SRA could be applied to any class of
    component-based problems where the selection of
    components needs to be optimized for
    nonfunctional requirement types of criteria.
  • The SRA tool was a development from previous BN
    requirements analyzer and has partially addressed
    the difficult problem of scenario-based testing

Discussion Conclusions
Discussion Conclusions
  • The SRA tool is aimed at requirements
    investigation in complex socio-technical systems
    and, hence, it complements model-checking tools
    which are more appropriate to later stages in
    development when specifications of agent behavior
    are available.