Title: SAE 599 Modeling and Simulation for Systems Architecting and Engineering
1SAE 599 - Modeling and Simulation for Systems
Architecting and Engineering
- Dr. Raymond Madachy
- October 17, 2007
2Outline
- Course project focus
- Student presentations
- Schedule and midterm options
- Cargo plane model redux
- Goal-Question-Metric
- Software-intensive systems applications
- Software business value model
- SISOS hybrid team model for incremental
development - Homework assignments
3Course Projects
- Strive for high impact projects
- Work, research, publications
- Consider model performance measures, and how your
model will be validated and verified - Study previous project reports
- See Class Projects at http//sunset.usc.edu/clas
ses/cs599_99/ for example project reports using
system dynamics - MBASE Architecting or COTS Glue Code are good
examples - Also collecting the best of reports for
discrete event projects with EZSIM and others - See proposal and final report for Food Service
Design Comparison
4Student Presentations
5Schedule and Midterm Options
- Dr. Hilary Bradbury-Huang will be our guest
speaker next week (full lecture) - Sustainable cities and environmental applications
- Soft modeling skills with a group exercise
- More people in class, more fun
- Dr. James Moore will talk on October 31 on urban
planning and transportation applications (half
period) - Midterm options
- In class exam (half period on 10/31 available)
- Take home exam with time span to-be-determined
6Cargo Plane Programmed Solution Update
- See http//rcf.usc.edu/madachy/php/cargoplanes7.p
hp
7Goal-Question-Metric
- GQM is a framework for developing a metrics
program 1 or simulation measures of performance - Steps
- Generate a set of organizational goals
- What do you want to improve?
- Derive a set of questions relating to the goals
- Answers provide visibility into meeting the goals
- Develop a set of metrics needed to answer the
questions
1 Victor Basili, Software Modeling and
Measurement The Goal/Question/Metric Paradigm,
CS-TR-2956, University of Maryland, 1992
8GQM Goal Definition Template
- Purpose To (characterize, evaluate, predict,
motivate, etc.) the (process, product, model,
metric, etc.) in order to (understand, assess,
manage, engineer, learn, improve, etc.) it. - Perspective Examine the (cost, effectiveness,
correctness, defects, changes, product metrics,
reliability, etc.) from the point of view of the
(developer, manager, customer, corporate
perspective, etc.) - Environment The environment consists of the
following process factors, people factors,
problem factors, methods, tools, constraints,
etc.
9GQM Example for Inspections
Organizational Process Improvement
Inspection Process
Goal
Efficiently use inspections to improve process
Increase quality
Increase productivity
Is defect density being reduced over time?
Is productivity increasing over time?
Question
How well are inspections performing?
What is the return of inspections?
What are high-leverage opportunities for defect
prevention?
What are optimal inspection parameters?
Is the inspection process under control?
How effective have process changes been?
Metric
defect removal effectiveness
inspection efficiency
inspection ROI
inspection rate control chart
defect density trends
productivity trends
effectiveness vs. inspection rate
defect categorization
effectiveness vs. of inspectors
defect finding rate control chart
effectiveness vs. preparation/inspection time
10Outline
- Course project focus
- Student presentations
- Schedule and midterm options
- Cargo plane model redux
- Goal-Question-Metric
- Software-intensive systems applications
- Software business value model
- SISOS hybrid team model for incremental
development - Homework assignments
11Value-Based Model Background
- Purpose Support software business
decision-making by experimenting with product
strategies and development practices to assess
real earned value - Description System dynamics model relates the
interactions between product specifications and
investments, software processes including quality
practices, market share, license retention,
pricing and revenue generation for a commercial
software enterprise
12Model Features
- A Value-Based Software Engineering (VBSE) model
covering the following VBSE elements - Stakeholders value proposition elicitation and
reconciliation - Business case analysis
- Value-based monitoring and control
- Integrated modeling of business value, software
products and processes to help make difficult
tradeoffs between perspectives - Value-based production functions used to relate
different attributes - Addresses the planning and control aspect of VBSE
to manage the value delivered to stakeholders - Experiment with different strategies and track
financial measures over time - Allows easy investigation of different strategy
combinations - Can be used dynamically before or during a
project - User inputs and model factors can vary over the
project duration as opposed to a static model - Suitable for actual project usage or flight
simulation training where simulations are
interrupted to make midstream decisions
13Model Sectors and Major Interfaces
- Software process and product sector computes the
staffing and quality over time - Market and sales sector accounts for market
dynamics including effect of quality reputation - Finance sector computes financial measures from
investments and revenues
14Software Process and Product
effort and schedule calculation with dynamic
COCOMO variant
product defect flows
15Finances, Market and Sales
investment and revenue flows
software license sales
market share dynamics including quality
reputation
16Quality Assumptions
- COCOMO cost driver Required Software Reliability
is a proxy for all quality practices - Resulting quality will modulate the actual sales
relative to the highest potential - Perception of quality in the market matters
- Quality reputation quickly lost and takes much
longer to regain (bad news travels fast) - Modeled as asymmetrical information smoothing via
negative feedback loop
17Market Share Production Function and Feature Sets
Cases from Example 1
18Sales Production Function and Reliability
Cases from Example 1
19Example 1 Dynamically Changing Scope and
Reliability
- Shows how model can assess the effects of
combined strategies by varying the scope and
required reliability independently or
simultaneously - Simulates midstream descoping, a frequent
strategy to meet time constraints by shedding
features - Three cases are demonstrated
- Unperturbed reference case
- Midstream descoping of the reference case after ½
year - Simultaneous midstream descoping and lowered
required reliability at ½ year
20Control Panel and Simulation Results
Descope
Case 1
Unperturbed Reference Case
Descope Lower Reliability
Case 2
21Case Summaries
22Example 2 Determining the Reliability Sweet Spot
- Analysis process
- Vary reliability across runs
- Use risk exposure framework to find process
optimum - Assess risk consequences of opposing trends
market delays and bad quality losses - Sum market losses and development costs
- Calculate resulting net revenue
- Simulation parameters
- A new 80 KSLOC product release can potentially
increase market share by 15-30 (varied in model
runs) - 75 schedule acceleration
- Initial total market size 64M annual revenue
- Vendor has 15 of market
- Overall market doubles in 5 years
23Cost Components
3-year time horizon
24Value-Based Model Conclusions
- To achieve real earned value, business value
attainment must be a key consideration when
designing software products and processes - Software enterprise decision-making can improve
with information from simulation models that
integrate business and technical perspectives - Optimal policies operate within a multi-attribute
decision space including various stakeholder
value functions, opposing market factors and
business constraints - Risk exposure is a convenient framework for
software decision analysis - Commercial process sweet spots with respect to
reliability are a balance between market delay
losses and quality losses - Model demonstrates a stakeholder value chain
whereby the value of software to end-users
ultimately translates into value for the software
development organization
25Value-Based Model Future Work
- Enhance product defect model with dynamic version
of COQUALMO to enable more constructive insight
into quality practices - Add maintenance and operational support
activities in the workflows - Elaborate market and sales for other
considerations including pricing scheme impacts,
varying market assumptions and periodic upgrades
of varying quality - Account for feedback loops to generate product
specifications (closed-loop control) - External feedback from users to incorporate new
features - Internal feedback on product initiatives from
organizational planning and control entity to the
software process - More empirical data on attribute relationships in
the model will help identify areas of improvement - Assessment of overall dynamics includes more
collection and analysis of field data on business
value and quality measures from actual software
product rollouts
26Outline
- Course project focus
- Student presentations
- Schedule and midterm options
- Cargo plane model redux
- Goal-Question-Metric
- Software-intensive systems applications
- Software business value model
- SISOS hybrid team model for incremental
development - Homework assignments
27Spiral Hybrid Process Introduction
- The spiral lifecycle is being extended to address
new challenges for Software-Intensive Systems of
Systems (SISOS), such as coping with rapid change
while simultaneously assuring high dependability - A hybrid plan-driven and agile process has been
outlined to address these conflicting challenges
with the need to rapidly field incremental
capabilities - A system-of-systems (SOS) integrates multiple
independently-developed systems and is very
large, dynamically evolving, unprecedented, with
emergent requirements and behaviors - However, traditional static approaches cannot
capture dynamic feedback loops and interacting
phenomena that cause real-world complexity (e.g.
hybrid processes, project volatility, increment
overlap and resource contention, schedule
pressure, slippages, communication overhead,
slack, etc.) - A system dynamics model has being developed to
assess the incremental hybrid process and support
project decision-making - Both the hybrid process and simulation model are
being evolved on a very large scale incremental
project for a SISOS (U.S. Army Future Combat
Systems)
28Future Combat Systems (FCS) Network
29Scalable Spiral Model Increment Activities
- Organize development into plan-driven increments
with stable specs - Agile team watches for and assesses changes, then
negotiates changes so next increment hits the
ground running - Try to prevent usage feedback from destabilizing
current increment - Three team cycle plays out from one increment to
the next
30Spiral Hybrid Model Features
- Estimates cost and schedule for multiple
increments of a hybrid process that uses three
specialized teams (agile re-baseliners,
developers, VVers) per the scalable spiral
model - Considers changes due to external volatility and
feedback from user-driven change requests - Deferral policies and team sizes can be
experimented with - Includes tradeoffs between cost and the timing of
changes within and across increments, length of
deferral delays, and others
31Model Input Control Panel
32Model Overview
- Built around a cyclic flow chain for capabilities
- Arrayed for multiple increments
- Each team is represented with a level and
corresponding staff allocation rate - Changes arrive a-periodically via the volatility
trends time function and flow into the level for
capability changes - Changes are processed by the agile team and
allocated to increments per the deferral policies - Constant or variable staffing for the agile team
- For each increment the required capabilities are
developed into developed capabilities and then
VVed into V Ved capabilities - Productivities and team sizes for development and
VV calculated with a Dynamic COCOMO variant and
continuously updated for scope changes - Flow rates between capability changes and V
Ved capabilities are bi-directional for
capability kickbacks sent back up the chain - User-driven changes from the field are identified
as field issues that flow back into the
capability changes
33Volatility Cost Functions
- The volatility effort multiplier for construction
effort and schedule is an aggregate multiplier
for volatility from different sources (e.g. COTS,
mission, etc.) relative to the original baseline
for increment - The lifecycle timing effort multiplier models
increased development cost the later a change
comes in midstream during an increment
34Sample Response to Volatility
- An unanticipated change occurs at month 8 shown
as a volatility trend 1 pulse - It flows into capability changes 1 which
declines to zero as the agile team processes the
change - The change is non-deferrable for increment 1 so
total capabilities 1 increases - Development team staff size dynamically responds
to the increased scope
35Sample Test Results
- Test case for two increments of 15 baseline
capabilities each - A non-deferrable change comes at month 8 (per
previous slide) - The agile team size is varied from 2 to 10 people
- Increment 1 business value also lost for agile
team sizes of 2 and 4
36Sample Test Results (cont.)
37Spiral Hybrid Model Conclusions and Future Work
- System dynamics is a convenient modeling
framework to deal with the complexities of a
SISOS - A hybrid process appears attractive to handle
SISOS dynamic evolution, emergent requirements
and behaviors - Initial results indicate that having an agile
team can help decrease overall cost and schedule - Model can help find the optimum balance
- Will obtain more empirical data to calibrate and
parameterize model including volatility and
change trends, change analysis effort, effort
multipliers and field issue rates - Model improvements
- Additional staffing options
- Rayleigh curve staffing profiles
- Constraints on development and VV staffing
levels - More flexible change deferral options across
increments - Increment volatility balancing policies
- Provisions to account for (timed)
business/mission value of capabilities - Additional model experimentation
- Include capabilities flowing back from developers
and VVers - Vary deferral policies and volatility patterns
across increments - Compare different agile team staffing policies
- Continue applying the model on a current SISOS
and seek other potential pilots
38Outline
- Course project focus
- Student presentations
- Schedule and midterm options
- Cargo plane model redux
- Goal-Question-Metric
- Software-intensive systems applications
- Software business value model
- SISOS hybrid team model for incremental
development - Homework assignments
39Homework
- Readings (two weeks)
- Law-Kelton Textbook Chapter 4
- Understand probability concepts with less focus
on mathematics of random variables - Law-Kelton Textbook Chapter 5
- Software Process Dynamics Chapter 2
- Software Process Dynamics Appendix A
40Homework (cont.)
- Problem (one week)
- Use the existing Brookss Law model as a basis or
create your own version for these enhancements - Part 1 evaluate the model with team sizes
between 0 and 100 people - Debug, illustrate and explain the problem
- Part 2 make the model scalable for team sizes up
to 60 people - Make several runs to test the model and show your
results - Part 3 add a feedback loop that controls
personnel allocation rate by comparing actual
production to planned production - The planned production assumes a constant
development rate, with all 500 function points
completed at 200 days. The existing model covers
actual production - Add logic for a one-time only correction when the
difference between actual and planned is 65
function points - Run the model and show the results for adding 0,
5, 10 and 20 people - Part 4 disaggregate the development flow chain
and test the resulting model - Convert the single level for developed software
into two levels for designed software and tested
software. There will be a rate for software
design and a rate for software construction. - Change the flowing units from function points to
lines of code - Use the COCOMO II model to calibrate the nominal
productivities for design and construction (e.g.
use the effort distribution for elaboration and
construction)