SAE 599 Modeling and Simulation for Systems Architecting and Engineering - PowerPoint PPT Presentation

1 / 40
About This Presentation
Title:

SAE 599 Modeling and Simulation for Systems Architecting and Engineering

Description:

Answers provide visibility into meeting the goals. Develop a set of metrics needed to answer the questions ... Homework assignments. Value-Based Model Background ... – PowerPoint PPT presentation

Number of Views:43
Avg rating:3.0/5.0
Slides: 41
Provided by: rayma5
Category:

less

Transcript and Presenter's Notes

Title: SAE 599 Modeling and Simulation for Systems Architecting and Engineering


1
SAE 599 - Modeling and Simulation for Systems
Architecting and Engineering
  • Dr. Raymond Madachy
  • October 17, 2007

2
Outline
  • Course project focus
  • Student presentations
  • Schedule and midterm options
  • Cargo plane model redux
  • Goal-Question-Metric
  • Software-intensive systems applications
  • Software business value model
  • SISOS hybrid team model for incremental
    development
  • Homework assignments

3
Course Projects
  • Strive for high impact projects
  • Work, research, publications
  • Consider model performance measures, and how your
    model will be validated and verified
  • Study previous project reports
  • See Class Projects at http//sunset.usc.edu/clas
    ses/cs599_99/ for example project reports using
    system dynamics
  • MBASE Architecting or COTS Glue Code are good
    examples
  • Also collecting the best of reports for
    discrete event projects with EZSIM and others
  • See proposal and final report for Food Service
    Design Comparison

4
Student Presentations
5
Schedule and Midterm Options
  • Dr. Hilary Bradbury-Huang will be our guest
    speaker next week (full lecture)
  • Sustainable cities and environmental applications
  • Soft modeling skills with a group exercise
  • More people in class, more fun
  • Dr. James Moore will talk on October 31 on urban
    planning and transportation applications (half
    period)
  • Midterm options
  • In class exam (half period on 10/31 available)
  • Take home exam with time span to-be-determined

6
Cargo Plane Programmed Solution Update
  • See http//rcf.usc.edu/madachy/php/cargoplanes7.p
    hp

7
Goal-Question-Metric
  • GQM is a framework for developing a metrics
    program 1 or simulation measures of performance
  • Steps
  • Generate a set of organizational goals
  • What do you want to improve?
  • Derive a set of questions relating to the goals
  • Answers provide visibility into meeting the goals
  • Develop a set of metrics needed to answer the
    questions

1 Victor Basili, Software Modeling and
Measurement The Goal/Question/Metric Paradigm,
CS-TR-2956, University of Maryland, 1992
8
GQM Goal Definition Template
  • Purpose To (characterize, evaluate, predict,
    motivate, etc.) the (process, product, model,
    metric, etc.) in order to (understand, assess,
    manage, engineer, learn, improve, etc.) it.
  • Perspective Examine the (cost, effectiveness,
    correctness, defects, changes, product metrics,
    reliability, etc.) from the point of view of the
    (developer, manager, customer, corporate
    perspective, etc.)
  • Environment The environment consists of the
    following process factors, people factors,
    problem factors, methods, tools, constraints,
    etc.

9
GQM Example for Inspections
Organizational Process Improvement
Inspection Process
Goal
Efficiently use inspections to improve process
Increase quality
Increase productivity
Is defect density being reduced over time?
Is productivity increasing over time?
Question
How well are inspections performing?
What is the return of inspections?
What are high-leverage opportunities for defect
prevention?
What are optimal inspection parameters?
Is the inspection process under control?
How effective have process changes been?
Metric
defect removal effectiveness
inspection efficiency
inspection ROI
inspection rate control chart
defect density trends
productivity trends
effectiveness vs. inspection rate
defect categorization
effectiveness vs. of inspectors
defect finding rate control chart
effectiveness vs. preparation/inspection time
10
Outline
  • Course project focus
  • Student presentations
  • Schedule and midterm options
  • Cargo plane model redux
  • Goal-Question-Metric
  • Software-intensive systems applications
  • Software business value model
  • SISOS hybrid team model for incremental
    development
  • Homework assignments

11
Value-Based Model Background
  • Purpose Support software business
    decision-making by experimenting with product
    strategies and development practices to assess
    real earned value
  • Description System dynamics model relates the
    interactions between product specifications and
    investments, software processes including quality
    practices, market share, license retention,
    pricing and revenue generation for a commercial
    software enterprise

12
Model Features
  • A Value-Based Software Engineering (VBSE) model
    covering the following VBSE elements
  • Stakeholders value proposition elicitation and
    reconciliation
  • Business case analysis
  • Value-based monitoring and control
  • Integrated modeling of business value, software
    products and processes to help make difficult
    tradeoffs between perspectives
  • Value-based production functions used to relate
    different attributes
  • Addresses the planning and control aspect of VBSE
    to manage the value delivered to stakeholders
  • Experiment with different strategies and track
    financial measures over time
  • Allows easy investigation of different strategy
    combinations
  • Can be used dynamically before or during a
    project
  • User inputs and model factors can vary over the
    project duration as opposed to a static model
  • Suitable for actual project usage or flight
    simulation training where simulations are
    interrupted to make midstream decisions

13
Model Sectors and Major Interfaces
  • Software process and product sector computes the
    staffing and quality over time
  • Market and sales sector accounts for market
    dynamics including effect of quality reputation
  • Finance sector computes financial measures from
    investments and revenues

14
Software Process and Product
effort and schedule calculation with dynamic
COCOMO variant
product defect flows
15
Finances, Market and Sales
investment and revenue flows
software license sales
market share dynamics including quality
reputation
16
Quality Assumptions
  • COCOMO cost driver Required Software Reliability
    is a proxy for all quality practices
  • Resulting quality will modulate the actual sales
    relative to the highest potential
  • Perception of quality in the market matters
  • Quality reputation quickly lost and takes much
    longer to regain (bad news travels fast)
  • Modeled as asymmetrical information smoothing via
    negative feedback loop

17
Market Share Production Function and Feature Sets
Cases from Example 1
18
Sales Production Function and Reliability
Cases from Example 1
19
Example 1 Dynamically Changing Scope and
Reliability
  • Shows how model can assess the effects of
    combined strategies by varying the scope and
    required reliability independently or
    simultaneously
  • Simulates midstream descoping, a frequent
    strategy to meet time constraints by shedding
    features
  • Three cases are demonstrated
  • Unperturbed reference case
  • Midstream descoping of the reference case after ½
    year
  • Simultaneous midstream descoping and lowered
    required reliability at ½ year

20
Control Panel and Simulation Results
Descope
Case 1
Unperturbed Reference Case
Descope Lower Reliability
Case 2
21
Case Summaries
22
Example 2 Determining the Reliability Sweet Spot
  • Analysis process
  • Vary reliability across runs
  • Use risk exposure framework to find process
    optimum
  • Assess risk consequences of opposing trends
    market delays and bad quality losses
  • Sum market losses and development costs
  • Calculate resulting net revenue
  • Simulation parameters
  • A new 80 KSLOC product release can potentially
    increase market share by 15-30 (varied in model
    runs)
  • 75 schedule acceleration
  • Initial total market size 64M annual revenue
  • Vendor has 15 of market
  • Overall market doubles in 5 years

23
Cost Components
3-year time horizon
24
Value-Based Model Conclusions
  • To achieve real earned value, business value
    attainment must be a key consideration when
    designing software products and processes
  • Software enterprise decision-making can improve
    with information from simulation models that
    integrate business and technical perspectives
  • Optimal policies operate within a multi-attribute
    decision space including various stakeholder
    value functions, opposing market factors and
    business constraints
  • Risk exposure is a convenient framework for
    software decision analysis
  • Commercial process sweet spots with respect to
    reliability are a balance between market delay
    losses and quality losses
  • Model demonstrates a stakeholder value chain
    whereby the value of software to end-users
    ultimately translates into value for the software
    development organization

25
Value-Based Model Future Work
  • Enhance product defect model with dynamic version
    of COQUALMO to enable more constructive insight
    into quality practices
  • Add maintenance and operational support
    activities in the workflows
  • Elaborate market and sales for other
    considerations including pricing scheme impacts,
    varying market assumptions and periodic upgrades
    of varying quality
  • Account for feedback loops to generate product
    specifications (closed-loop control)
  • External feedback from users to incorporate new
    features
  • Internal feedback on product initiatives from
    organizational planning and control entity to the
    software process
  • More empirical data on attribute relationships in
    the model will help identify areas of improvement
  • Assessment of overall dynamics includes more
    collection and analysis of field data on business
    value and quality measures from actual software
    product rollouts

26
Outline
  • Course project focus
  • Student presentations
  • Schedule and midterm options
  • Cargo plane model redux
  • Goal-Question-Metric
  • Software-intensive systems applications
  • Software business value model
  • SISOS hybrid team model for incremental
    development
  • Homework assignments

27
Spiral Hybrid Process Introduction
  • The spiral lifecycle is being extended to address
    new challenges for Software-Intensive Systems of
    Systems (SISOS), such as coping with rapid change
    while simultaneously assuring high dependability
  • A hybrid plan-driven and agile process has been
    outlined to address these conflicting challenges
    with the need to rapidly field incremental
    capabilities
  • A system-of-systems (SOS) integrates multiple
    independently-developed systems and is very
    large, dynamically evolving, unprecedented, with
    emergent requirements and behaviors
  • However, traditional static approaches cannot
    capture dynamic feedback loops and interacting
    phenomena that cause real-world complexity (e.g.
    hybrid processes, project volatility, increment
    overlap and resource contention, schedule
    pressure, slippages, communication overhead,
    slack, etc.)
  • A system dynamics model has being developed to
    assess the incremental hybrid process and support
    project decision-making
  • Both the hybrid process and simulation model are
    being evolved on a very large scale incremental
    project for a SISOS (U.S. Army Future Combat
    Systems)

28
Future Combat Systems (FCS) Network
29
Scalable Spiral Model Increment Activities
  • Organize development into plan-driven increments
    with stable specs
  • Agile team watches for and assesses changes, then
    negotiates changes so next increment hits the
    ground running
  • Try to prevent usage feedback from destabilizing
    current increment
  • Three team cycle plays out from one increment to
    the next

30
Spiral Hybrid Model Features
  • Estimates cost and schedule for multiple
    increments of a hybrid process that uses three
    specialized teams (agile re-baseliners,
    developers, VVers) per the scalable spiral
    model
  • Considers changes due to external volatility and
    feedback from user-driven change requests
  • Deferral policies and team sizes can be
    experimented with
  • Includes tradeoffs between cost and the timing of
    changes within and across increments, length of
    deferral delays, and others

31
Model Input Control Panel
32
Model Overview
  • Built around a cyclic flow chain for capabilities
  • Arrayed for multiple increments
  • Each team is represented with a level and
    corresponding staff allocation rate
  • Changes arrive a-periodically via the volatility
    trends time function and flow into the level for
    capability changes
  • Changes are processed by the agile team and
    allocated to increments per the deferral policies
  • Constant or variable staffing for the agile team
  • For each increment the required capabilities are
    developed into developed capabilities and then
    VVed into V Ved capabilities
  • Productivities and team sizes for development and
    VV calculated with a Dynamic COCOMO variant and
    continuously updated for scope changes
  • Flow rates between capability changes and V
    Ved capabilities are bi-directional for
    capability kickbacks sent back up the chain
  • User-driven changes from the field are identified
    as field issues that flow back into the
    capability changes

33
Volatility Cost Functions
  • The volatility effort multiplier for construction
    effort and schedule is an aggregate multiplier
    for volatility from different sources (e.g. COTS,
    mission, etc.) relative to the original baseline
    for increment
  • The lifecycle timing effort multiplier models
    increased development cost the later a change
    comes in midstream during an increment

34
Sample Response to Volatility
  • An unanticipated change occurs at month 8 shown
    as a volatility trend 1 pulse
  • It flows into capability changes 1 which
    declines to zero as the agile team processes the
    change
  • The change is non-deferrable for increment 1 so
    total capabilities 1 increases
  • Development team staff size dynamically responds
    to the increased scope

35
Sample Test Results
  • Test case for two increments of 15 baseline
    capabilities each
  • A non-deferrable change comes at month 8 (per
    previous slide)
  • The agile team size is varied from 2 to 10 people
  • Increment 1 business value also lost for agile
    team sizes of 2 and 4

36
Sample Test Results (cont.)
37
Spiral Hybrid Model Conclusions and Future Work
  • System dynamics is a convenient modeling
    framework to deal with the complexities of a
    SISOS
  • A hybrid process appears attractive to handle
    SISOS dynamic evolution, emergent requirements
    and behaviors
  • Initial results indicate that having an agile
    team can help decrease overall cost and schedule
  • Model can help find the optimum balance
  • Will obtain more empirical data to calibrate and
    parameterize model including volatility and
    change trends, change analysis effort, effort
    multipliers and field issue rates
  • Model improvements
  • Additional staffing options
  • Rayleigh curve staffing profiles
  • Constraints on development and VV staffing
    levels
  • More flexible change deferral options across
    increments
  • Increment volatility balancing policies
  • Provisions to account for (timed)
    business/mission value of capabilities
  • Additional model experimentation
  • Include capabilities flowing back from developers
    and VVers
  • Vary deferral policies and volatility patterns
    across increments
  • Compare different agile team staffing policies
  • Continue applying the model on a current SISOS
    and seek other potential pilots

38
Outline
  • Course project focus
  • Student presentations
  • Schedule and midterm options
  • Cargo plane model redux
  • Goal-Question-Metric
  • Software-intensive systems applications
  • Software business value model
  • SISOS hybrid team model for incremental
    development
  • Homework assignments

39
Homework
  • Readings (two weeks)
  • Law-Kelton Textbook Chapter 4
  • Understand probability concepts with less focus
    on mathematics of random variables
  • Law-Kelton Textbook Chapter 5
  • Software Process Dynamics Chapter 2
  • Software Process Dynamics Appendix A

40
Homework (cont.)
  • Problem (one week)
  • Use the existing Brookss Law model as a basis or
    create your own version for these enhancements
  • Part 1 evaluate the model with team sizes
    between 0 and 100 people
  • Debug, illustrate and explain the problem
  • Part 2 make the model scalable for team sizes up
    to 60 people
  • Make several runs to test the model and show your
    results
  • Part 3 add a feedback loop that controls
    personnel allocation rate by comparing actual
    production to planned production
  • The planned production assumes a constant
    development rate, with all 500 function points
    completed at 200 days. The existing model covers
    actual production
  • Add logic for a one-time only correction when the
    difference between actual and planned is 65
    function points
  • Run the model and show the results for adding 0,
    5, 10 and 20 people
  • Part 4 disaggregate the development flow chain
    and test the resulting model
  • Convert the single level for developed software
    into two levels for designed software and tested
    software. There will be a rate for software
    design and a rate for software construction.
  • Change the flowing units from function points to
    lines of code
  • Use the COCOMO II model to calibrate the nominal
    productivities for design and construction (e.g.
    use the effort distribution for elaboration and
    construction)
Write a Comment
User Comments (0)
About PowerShow.com