Title: Joint ADL CoLab BAA COCOMOSCORM: Content Complexity Measurement Tool
1Joint ADL Co-Lab BAACOCOMO-SCORM Content
Complexity Measurement Tool
Contract N61339-05-C-0115
- Interim Project Review
- 09 March 2006
2COCOMO-SCORM Project IPR AgendaMarch 9, 900
1130
- JADL Co-Lab (900-915)
- Purpose of Meeting
- Goals of this program
- Sparta/Anteon (915-1115)
- Project Management (Roger Smith)
- Research COCOMO Variables (Lacey Edwards)
- Research ISD Cost Items (Kelly Ward)
- Create COCOMO-SCORM Model (Lacey Edwards)
- Data Collection (Roger Smith)
- Model Calibration (Roger Smith)
- Questions and Discussion (1115-1130)
- Conclusion (Roger Smith)
3Project COCOMO-SCORM
BAA Research Interest Statement
Web-based and SCORM content is difficult to
measure in terms of complexity and development
time however, most people would agree that there
are certain qualities that make some content more
difficult to develop than other content (like
level of interactivity). What is needed is an
algorithm that takes into account all of the
decision points an instructional designer and
later the software developer makes when designing
and developing SCORM content. Being able to
quantify this in a comparative numerical measure
would make development and test cost estimates
more meaningful and accurate thereby, reducing
risk to a content development program.
Proposed Technical Concept
Every complex project requires project managers
and sponsors to calculate the expected level of
effort, duration, and cost of the project.
Developing web-based content to the Sharable
Content Object Reference Model (SCORM) is complex
enough that an algorithm is necessary to assist
both the government and developers in estimating
the size of the project. SPARTA and Anteon
propose to create an interactive courseware
estimation tool based on widely used
Instructional Systems Design (ISD) methodologies
and on the principles contained in the
well-established and widely accepted COCOMO II
model for software project estimation.
4Project Description
- Create an interactive project estimation tool for
ISD/SCORM content - Algorithmic foundation for the tool is the COCOMO
II algorithm developed for software projects by
the research team at the University of Southern
California and led by Barry Boehm - ISD methodology estimated will be the ADDIE model
developed at Florida State University and adopted
by the U.S. Armed Services - Resulting algorithm and tools will estimate the
person-months required to create a SCORM
conformant learning product - Dollar costs can be derived outside of the model
by applying industry or specific company labor
rates
5Problems to Solve
- Enumerate SCORM products and processes that
contribute to the staffing and duration of a
project - Identify mathematic and logical relationships
between these items - Quantify the level of contribution of each item
to project cost - Adjust the algorithm constants for SCORM
courseware - Define a bounded set of conditions under which
the algorithm can be relied upon - Validate the algorithm for various sets of data
within the bounded conditions
?
?
?
6Applicability Value to Community
- Consistent, objective, and reliable estimation
tool for SCORM content and projects - First step in formalizing an estimation method in
the ADL community - Create a tool that other projects can apply,
modify, and mature - COCOMO II has been evolving for 25 years.
- COCOMO-SCORM tool from this BAA will be the first
step in the long evolution and improvement of a
tool for this community
7Project Cost Estimation
Dollar Costs
Project Requirements
Quantitative Input
Qualitative Adjustments
Reuse
Person Months to Develop
Level of Instruction
Course Hours
Time to Develop
8Project Management
9Team Personnel
- SPARTA
- Roger Smith, PM
- Lacey Edwards
- Seth Lytle
- Anteon
- Kelly Ward
- Denise Stevens
- Tatjana Pitts
- Tim Richey
10Team Collaboration
- Continuous Research Collaboration
- Daily telephone and email exchange
- Bi-weekly Meetings
- Face-to-face exchange of ideas and progress
bi-weekly - Monthly Sub-contract Report
- Monthly written report of progress, status, and
future plans - In Process Review
- Pre-government meeting preparation and
collaboration - Training 2006 Presentation
- March 7, 2006
- I/ITSEC 2006 Paper Preparation
- Abstract Submitted February 2006
- Draft Paper Due June 2006
- Final Paper Due October 2006
11Schedule Phase 1
12Deliverables Phase 1
?
?
?
?
?
?
?
13Option Phase 2
14Research COCOMO Variables
15COCOMO II Research Approach
- Studied the definitive source on COCOMO II
Software Cost Estimation with COCOMO II by Barry
Boehm et.al. - Model form and variables
- Methodology used to create the COCOMO II model
- Researched how related models (COSYSMO, COPSEMO,
etc.) modified COCOMO II for their purposes - COSYSMO workshop
- Relationship with Dr. Ricardo Valerdi
- Surveyed software implementations of COCOMO II
16COCOMO Model Family
Other Independent Estimation Models
Software Cost Models
DBA COCOMO 2004
?
?
COCOMO 81 1981
COCOTS 2000
COSYSMO 2002
COCOMO II 2000
?
?
COINCOMO 2004
COSoSIMO 2004
Costing Secure System 2004
?
?
iDAVE 2003
COPLIMO 2003
COPSEMO 1998
Security Extension 2004
COQUALMO 1998
?
COPROMO 1998
CORADMO 1999
COSCOMO 2006
?
Software Extensions
Legend Model has been calibrated with historical
project data and expert (Delphi) data Model is
derived from COCOMO II Model has been calibrated
with expert (Delphi) data
Dates indicate the time that the first paper was
published for the model
17COCOMO II Tool Example
Users Perspective
18COCOMO II Drivers(29 Input Variables)
- Size
- Source Lines of Code (SLOC)
- Design Modification (DM)
- Code Modification (CM)
- Integration (IM)
- Assessment (AA)
- Understanding (SU)
- Unfamiliarity (UNFAM)
- Requirements Evolution (REVL)
- Product Effort Multipliers (EM)
- Required Reliability (RELY)
- Database Size (DATA)
- Product Complexity (CPLX)
- Required Reuse (RUSE)
- Documentation (DOCU)
- Platform EM
- Execution Time Constraints (TIME)
- Main Storage Constraints (STORE)
- Platform Volatility (PVOL)
- Personnel EM
- Analyst Capability (ACAP)
- Programmer Capability (PCAP)
- Personnel Continuity (PCON)
- Applications Experience (APEX)
- Platform Experience (PLEX)
- Language/Toolset Experience (LTEX)
- Project EM
- Use of Software Tools (TOOL)
- Multisite Development (SITE)
- Required Development Schedule (SCED)
- Scale Drivers
- Development Flexibility (FLEX)
- Process Maturity (PMAT)
- Precedentedness (PREC)
- Arch/Risk Resolution (RESL)
- Team Cohesion (TEAM)
19COCOMO Effort Multipliers
20COCOMO Project Variation
SF Flex 22.6 Higher 9 Lower
EM Flex From 115.6 times larger 17.6 times
smaller
21COCOMO II Model Form
17
PM A(Size)E P EMi
i1
5
E B .01S SFj
j1
Where PM effort in Person Months A
calibration constant derived from historical
project data Size Adjust Source Lines of Code
E represents diseconomy of scale (composed
of 5 scale factors) EM effort multiplier for
the ith cost driver. The geometric product
results in an overall effort adjustment factor to
the nominal effort. B calibration constant
derived from historical project data SF scale
factor for the jth cost driver. Provides
organization specific adjustments to the size of
the project.
22Reliability PRED(30)
- Reliability of COCOMO family of models is often
measured by the percentage of test cases that it
will estimate within 30 of the actual project
costs - e.g. If a project requires 300 person-months to
complete, then its PRED(30) range would be (210
to 390) - If the model estimates 70 of its test cases
within this range then the models PRED(30) 70 - COCOMO Family Model Levels
- COCOMO II (2000) PRED(30) 69
- COSYSMO PRED(30) 56
23PRED(30)
COCOMO II
COSYSMO
69
56
0
-30
30
0
-30
30
Variance of Estimated Value
24Research ISD Cost Items
25ADDIE Process
Job/Task Analysis Needs Analysis Learning Analys
is Situation Analysis Technical Analysis
Storyboards Programming Multimedia Testing
Design Plan Instructional Media Design
Report SCORM Plan
Reliability Validation Training Plan
Trials Pilot tests
26SCORM Cost Variables
- Historical Data
- Top Down
- Bottom Up
- Industry Standards
- Current Estimation Practices
27SCORM Cost Variables
- Types and quantity of media
A D D I E
- Entry point into ADDIE process
28COCOMO II Drivers(29 Input Variables)
- Size
- Source Lines of Code (SLOC)
- Design Modification (DM)
- Code Modification (CM)
- Integration (IM)
- Assessment (AA)
- Understanding (SU)
- Unfamiliarity (UNFAM)
- Requirements Evolution (REVL)
- Product Effort Multipliers (EM)
- Required Reliability (RELY)
- Database Size (DATA)
- Product Complexity (CPLX)
- Required Reuse (RUSE)
- Documentation (DOCU)
- Platform EM
- Execution Time Constraints (TIME)
- Main Storage Constraints (STORE)
- Platform Volatility (PVOL)
- Personnel EM
- Analyst Capability (ACAP)
- Programmer Capability (PCAP)
- Personnel Continuity (PCON)
- Applications Experience (APEX)
- Platform Experience (PLEX)
- Language/Toolset Experience (LTEX)
- Project EM
- Use of Software Tools (TOOL)
- Multisite Development (SITE)
- Required Development Schedule (SCED)
- Scale Drivers
- Development Flexibility (FLEX)
- Process Maturity (PMAT)
- Precedentedness (PREC)
- Arch/Risk Resolution (RESL)
- Team Cohesion (TEAM)
No
No
No
No
No
No
No
No
No
No
29COSCOMO Drivers(27 Input Variables)
- Size
- Source Lines of Code (SLOC)
- Design Modification (DM)
- Code Modification (CM)
- Integration (IM)
- Assessment (AA)
- Understanding (SU)
- Unfamiliarity (UNFAM)
- Scale Drivers
- Development Flexibility (FLEX)
- Process Maturity (PMAT)
- Precedentedness (PREC)
- Arch/Risk Resolution (RESL)
- Team Cohesion (TEAM)
- Product Effort Multipliers (EM)
- Required Reliability (RELY)
- Product Complexity (CPLX)
- Required Reuse (RUSE)
- Documentation (DOCU)
- Platform EM
- Platform Volatility (PVOL)
- Bandwidth (BAND)
- Personnel EM
- Senior Capability (SCAP)
- Developer Capability (DCAP)
- Personnel Continuity (PCON)
- Applications Experience (APEX)
- Platform Experience (PLEX)
- Development Tools Experience (DTEX)
- Project EM
- Lifecycle Tools (LIFE)
- Multisite Development (SITE)
- Required Development Schedule (SCED)
Change
New
Change
Change
Change
Change
30COCOMO-SCORM Drivers(24 Input Variables)
- Size of Product
- Hours of Courseware
- Level of Instruction
- New, Reused, Modified Courseware
- Requirements Evolution (Content Discarded)
- Product Effort Multipliers (EM)
- Required Reliability (RELY)
- Product Complexity (CPLX)
- Required Reuse (RUSE)
- Documentation (DOCU)
- Platform EM
- Bandwidth (BAND)
- Platform Volatility (PVOL)
- Personnel EM
- Senior Capability (SCAP)
- Developer Capability (DCAP)
- Personnel Continuity (PCON)
- Applications Experience (APEX)
- Platform Experience (PLEX)
- Development Tools Experience (DTEX)
- Project EM
- Lifecycle Tools (LIFE)
- Multisite Development (SITE)
- Required Development Schedule (SCED)
- Scale Factors
- Precedentedness (PREC)
- Development Flexibility (FLEX)
- Arch/Risk Resolution (RESL)
- Team Cohesion (TEAM)
- Process Maturity (PMAT)
Change
Change
Change
New
Change
Change
Change
New
31Create COCOMO-SCORM Model
32COCOMO-SCORM Drivers(24 Input Variables)
- Size of Product
- Hours of Courseware
- Level of Instruction
- New, Reused, Modified Courseware
- Requirements Evolution (Content Discarded)
- Product Effort Multipliers (EM)
- Required Reliability (RELY)
- Product Complexity (CPLX)
- Required Reuse (RUSE)
- Documentation (DOCU)
- Platform EM
- Bandwidth (BAND)
- Platform Volatility (PVOL)
- Personnel EM
- Senior Capability (SCAP)
- Developer Capability (DCAP)
- Personnel Continuity (PCON)
- Applications Experience (APEX)
- Platform Experience (PLEX)
- Development Tools Experience (DTEX)
- Project EM
- Lifecycle Tools (LIFE)
- Multisite Development (SITE)
- Required Development Schedule (SCED)
- Scale Factors
- Precedentedness (PREC)
- Development Flexibility (FLEX)
- Arch/Risk Resolution (RESL)
- Team Cohesion (TEAM)
- Process Maturity (PMAT)
33Quantitative Size Variables
34Qualitative Project Variables
Effort Multipliers
Scale Factors
Nominal/Standard Levels
35EM Effect on Project
15 Effort Multipliers for Nominal Project
Course Hours
Project Cost
Project Cost A Course
Hours (Person Months) (Constant) (Student Hours)
36Answers Effect Project Cost
15 Effort Multipliers for More Realistic Project
Project Cost
Course Hours
Nominal Cost
Project Cost A EMi
Course Hours (Person Months) (Constant)
(Modifiers) (Student Hours)
37Range of Cost Variation
Max X 115 ?
Course Hours
Nominal Cost
Flex From X times larger to Y times smaller
Min Y 0.05 ?
38SF Effect on Project
All Scale Factors Set to Nominal
SW Size
Project Cost
Very Low
L
Low
Nominal
High
Very High
Extra High
0.0 (always)
39SF Effect on Project
Scale Factors Other Than Nominal
Est. Cost
SW Size
Nom. Cost
Very Low
L
Low
Nominal
High
Very High
Extra High
0.0 (always)
40Range of Scale Effect
Max X 1.226
SW Size
Project Cost
Min Y 0.91
Flex 22.6 Higher 9 Lower
41COSCOMO Model Form
15
PM A(Size)E P EMi
i1
5
E B .01S SFj
j1
Where PM effort in Person Months A
calibration constant derived from historical
project data Size Adjust Source Lines of Code
E represents diseconomy of scale (composed
of 5 scale factors) EM effort multiplier for
the ith cost driver. The geometric product
results in an overall effort adjustment factor to
the nominal effort. B calibration constant
derived from historical project data SF scale
factor for the jth cost driver. Provides
organization specific adjustments to the size of
the project.
42Data Collection
43Projects Collected
- Defense Nuclear Weapons School, Dean Marvin
- Air Force Accounting Liaison Course, Gary Twogood
- Explosive Hazard Awareness Course, Larry Helms
- (more to follow )
- Leads
- Capt Scott Loller, Air Force Distance Learning
- Angela Lindsey, Joint Forces Staff College
- Maj Chris Edwards, UK Training Advisory Group
- Patricia Mulligan, Formerly AFAMS
44Data Collection Form
(see Excel file for entire form)
45COSCOMO Estimates
Survey was not completed. All remaining
variables were estimated.
46Defense Nuclear Weapons School
47AF Accounting Liaison Course
48Explosive Hazard Awareness
49Model Calibration
50Building PRED(30) for COSCOMO
Total Project Cost (PM)
DNWS -3.8
78
ALC 333
24
EHA -20
6
0
-30
30
Variance of Estimated Value
51Calibration of Variables
Effort Multipliers
Scale Factors
Nominal/Standard Levels
52Delphi Survey
- Using Delphi Method to gather independent
opinions about the level of effort associated
with each setting on each variable. - Collect individual responses
- Calculate group average
- Present averages to each person and allow them to
change their responses if they would like - Convene group work session to settle on
acceptable values - Repeat if necessary
- Surveys distributed to
- Dean Marvin, Kelly Ward, Denise Stevens, Tim
Richey, Tatjana Pitts
53Conclusion
54Completed Work
- Technical and Management Work Plan
(DI-MGMT-81117) - Researched COCOMO II
- COCOMO published literature
- COSYSMO seminar and relationship with Dr. Ricardo
Valerdi - Researched SCORM Cost Drivers
- ADDIE process steps
- SCORM project cost drivers
- Created COCOMO-SCORM Algorithm
- Transformed COCOMO variables into COSCOMO tool
- Created new variables for COSCOMO
55Current Work
- Creating COCOMO-SCORM Software Model
- Functional software tool
- Data Collection
- Collect historical project data
- Model Calibration Validation
- Calibrate model to fit historical data and expert
knowledge - Build Community
- Create a community of interest around the method
and tool
56Next Steps
- Complete COCOMO-SCORM Model
- Microsoft Excel implementation
- Accompanying Documentation
- Create Prototype GUI
- Final Project Review
- Phase 2 (optional)
57Right on Track