Joint ADL CoLab BAA COCOMOSCORM: Content Complexity Measurement Tool - PowerPoint PPT Presentation

1 / 57
About This Presentation
Title:

Joint ADL CoLab BAA COCOMOSCORM: Content Complexity Measurement Tool

Description:

'Web-based and SCORM content is difficult to measure in terms of complexity and ... Create an interactive project estimation tool for ISD/SCORM content ... – PowerPoint PPT presentation

Number of Views:100
Avg rating:3.0/5.0
Slides: 58
Provided by: roger228
Category:

less

Transcript and Presenter's Notes

Title: Joint ADL CoLab BAA COCOMOSCORM: Content Complexity Measurement Tool


1
Joint ADL Co-Lab BAACOCOMO-SCORM Content
Complexity Measurement Tool
Contract N61339-05-C-0115
  • Interim Project Review
  • 09 March 2006

2
COCOMO-SCORM Project IPR AgendaMarch 9, 900
1130
  • JADL Co-Lab (900-915)
  • Purpose of Meeting
  • Goals of this program
  • Sparta/Anteon (915-1115)
  • Project Management (Roger Smith)
  • Research COCOMO Variables (Lacey Edwards)
  • Research ISD Cost Items (Kelly Ward)
  • Create COCOMO-SCORM Model (Lacey Edwards)
  • Data Collection (Roger Smith)
  • Model Calibration (Roger Smith)
  • Questions and Discussion (1115-1130)
  • Conclusion (Roger Smith)

3
Project COCOMO-SCORM
BAA Research Interest Statement
Web-based and SCORM content is difficult to
measure in terms of complexity and development
time however, most people would agree that there
are certain qualities that make some content more
difficult to develop than other content (like
level of interactivity). What is needed is an
algorithm that takes into account all of the
decision points an instructional designer and
later the software developer makes when designing
and developing SCORM content. Being able to
quantify this in a comparative numerical measure
would make development and test cost estimates
more meaningful and accurate thereby, reducing
risk to a content development program.
Proposed Technical Concept
Every complex project requires project managers
and sponsors to calculate the expected level of
effort, duration, and cost of the project.
Developing web-based content to the Sharable
Content Object Reference Model (SCORM) is complex
enough that an algorithm is necessary to assist
both the government and developers in estimating
the size of the project. SPARTA and Anteon
propose to create an interactive courseware
estimation tool based on widely used
Instructional Systems Design (ISD) methodologies
and on the principles contained in the
well-established and widely accepted COCOMO II
model for software project estimation.
4
Project Description
  • Create an interactive project estimation tool for
    ISD/SCORM content
  • Algorithmic foundation for the tool is the COCOMO
    II algorithm developed for software projects by
    the research team at the University of Southern
    California and led by Barry Boehm
  • ISD methodology estimated will be the ADDIE model
    developed at Florida State University and adopted
    by the U.S. Armed Services
  • Resulting algorithm and tools will estimate the
    person-months required to create a SCORM
    conformant learning product
  • Dollar costs can be derived outside of the model
    by applying industry or specific company labor
    rates

5
Problems to Solve
  • Enumerate SCORM products and processes that
    contribute to the staffing and duration of a
    project
  • Identify mathematic and logical relationships
    between these items
  • Quantify the level of contribution of each item
    to project cost
  • Adjust the algorithm constants for SCORM
    courseware
  • Define a bounded set of conditions under which
    the algorithm can be relied upon
  • Validate the algorithm for various sets of data
    within the bounded conditions

?
?
?
6
Applicability Value to Community
  • Consistent, objective, and reliable estimation
    tool for SCORM content and projects
  • First step in formalizing an estimation method in
    the ADL community
  • Create a tool that other projects can apply,
    modify, and mature
  • COCOMO II has been evolving for 25 years.
  • COCOMO-SCORM tool from this BAA will be the first
    step in the long evolution and improvement of a
    tool for this community

7
Project Cost Estimation
Dollar Costs
Project Requirements
Quantitative Input
Qualitative Adjustments
Reuse
Person Months to Develop
Level of Instruction
Course Hours
Time to Develop
8
Project Management
9
Team Personnel
  • SPARTA
  • Roger Smith, PM
  • Lacey Edwards
  • Seth Lytle
  • Anteon
  • Kelly Ward
  • Denise Stevens
  • Tatjana Pitts
  • Tim Richey

10
Team Collaboration
  • Continuous Research Collaboration
  • Daily telephone and email exchange
  • Bi-weekly Meetings
  • Face-to-face exchange of ideas and progress
    bi-weekly
  • Monthly Sub-contract Report
  • Monthly written report of progress, status, and
    future plans
  • In Process Review
  • Pre-government meeting preparation and
    collaboration
  • Training 2006 Presentation
  • March 7, 2006
  • I/ITSEC 2006 Paper Preparation
  • Abstract Submitted February 2006
  • Draft Paper Due June 2006
  • Final Paper Due October 2006

11
Schedule Phase 1
12
Deliverables Phase 1
?
?
?
?
?
?
?
13
Option Phase 2
14
Research COCOMO Variables
15
COCOMO II Research Approach
  • Studied the definitive source on COCOMO II
    Software Cost Estimation with COCOMO II by Barry
    Boehm et.al.
  • Model form and variables
  • Methodology used to create the COCOMO II model
  • Researched how related models (COSYSMO, COPSEMO,
    etc.) modified COCOMO II for their purposes
  • COSYSMO workshop
  • Relationship with Dr. Ricardo Valerdi
  • Surveyed software implementations of COCOMO II

16
COCOMO Model Family
Other Independent Estimation Models
Software Cost Models
DBA COCOMO 2004
?
?
COCOMO 81 1981
COCOTS 2000
COSYSMO 2002
COCOMO II 2000
?
?
COINCOMO 2004
COSoSIMO 2004
Costing Secure System 2004
?
?
iDAVE 2003
COPLIMO 2003
COPSEMO 1998
Security Extension 2004
COQUALMO 1998
?
COPROMO 1998
CORADMO 1999
COSCOMO 2006
?
Software Extensions
Legend Model has been calibrated with historical
project data and expert (Delphi) data Model is
derived from COCOMO II Model has been calibrated
with expert (Delphi) data
Dates indicate the time that the first paper was
published for the model
17
COCOMO II Tool Example
Users Perspective
18
COCOMO II Drivers(29 Input Variables)
  • Size
  • Source Lines of Code (SLOC)
  • Design Modification (DM)
  • Code Modification (CM)
  • Integration (IM)
  • Assessment (AA)
  • Understanding (SU)
  • Unfamiliarity (UNFAM)
  • Requirements Evolution (REVL)
  • Product Effort Multipliers (EM)
  • Required Reliability (RELY)
  • Database Size (DATA)
  • Product Complexity (CPLX)
  • Required Reuse (RUSE)
  • Documentation (DOCU)
  • Platform EM
  • Execution Time Constraints (TIME)
  • Main Storage Constraints (STORE)
  • Platform Volatility (PVOL)
  • Personnel EM
  • Analyst Capability (ACAP)
  • Programmer Capability (PCAP)
  • Personnel Continuity (PCON)
  • Applications Experience (APEX)
  • Platform Experience (PLEX)
  • Language/Toolset Experience (LTEX)
  • Project EM
  • Use of Software Tools (TOOL)
  • Multisite Development (SITE)
  • Required Development Schedule (SCED)
  • Scale Drivers
  • Development Flexibility (FLEX)
  • Process Maturity (PMAT)
  • Precedentedness (PREC)
  • Arch/Risk Resolution (RESL)
  • Team Cohesion (TEAM)

19
COCOMO Effort Multipliers
20
COCOMO Project Variation
SF Flex 22.6 Higher 9 Lower
EM Flex From 115.6 times larger 17.6 times
smaller
21
COCOMO II Model Form
17
PM A(Size)E P EMi
i1
5
E B .01S SFj
j1
Where PM effort in Person Months A
calibration constant derived from historical
project data Size Adjust Source Lines of Code
E represents diseconomy of scale (composed
of 5 scale factors) EM effort multiplier for
the ith cost driver. The geometric product
results in an overall effort adjustment factor to
the nominal effort. B calibration constant
derived from historical project data SF scale
factor for the jth cost driver. Provides
organization specific adjustments to the size of
the project.
22
Reliability PRED(30)
  • Reliability of COCOMO family of models is often
    measured by the percentage of test cases that it
    will estimate within 30 of the actual project
    costs
  • e.g. If a project requires 300 person-months to
    complete, then its PRED(30) range would be (210
    to 390)
  • If the model estimates 70 of its test cases
    within this range then the models PRED(30) 70
  • COCOMO Family Model Levels
  • COCOMO II (2000) PRED(30) 69
  • COSYSMO PRED(30) 56

23
PRED(30)
COCOMO II
COSYSMO
69
56
0
-30
30
0
-30
30
Variance of Estimated Value
24
Research ISD Cost Items
25
ADDIE Process
Job/Task Analysis Needs Analysis Learning Analys
is Situation Analysis Technical Analysis
Storyboards Programming Multimedia Testing
Design Plan Instructional Media Design
Report SCORM Plan
Reliability Validation Training Plan
Trials Pilot tests
26
SCORM Cost Variables
  • Historical Data
  • Top Down
  • Bottom Up
  • Industry Standards
  • Current Estimation Practices

27
SCORM Cost Variables
  • Significant Factors
  • Types and quantity of media
  • Levels of interactivity

A D D I E
  • Entry point into ADDIE process

28
COCOMO II Drivers(29 Input Variables)
  • Size
  • Source Lines of Code (SLOC)
  • Design Modification (DM)
  • Code Modification (CM)
  • Integration (IM)
  • Assessment (AA)
  • Understanding (SU)
  • Unfamiliarity (UNFAM)
  • Requirements Evolution (REVL)
  • Product Effort Multipliers (EM)
  • Required Reliability (RELY)
  • Database Size (DATA)
  • Product Complexity (CPLX)
  • Required Reuse (RUSE)
  • Documentation (DOCU)
  • Platform EM
  • Execution Time Constraints (TIME)
  • Main Storage Constraints (STORE)
  • Platform Volatility (PVOL)
  • Personnel EM
  • Analyst Capability (ACAP)
  • Programmer Capability (PCAP)
  • Personnel Continuity (PCON)
  • Applications Experience (APEX)
  • Platform Experience (PLEX)
  • Language/Toolset Experience (LTEX)
  • Project EM
  • Use of Software Tools (TOOL)
  • Multisite Development (SITE)
  • Required Development Schedule (SCED)
  • Scale Drivers
  • Development Flexibility (FLEX)
  • Process Maturity (PMAT)
  • Precedentedness (PREC)
  • Arch/Risk Resolution (RESL)
  • Team Cohesion (TEAM)

No
No
No
No
No
No
No
No
No
No
29
COSCOMO Drivers(27 Input Variables)
  • Size
  • Source Lines of Code (SLOC)
  • Design Modification (DM)
  • Code Modification (CM)
  • Integration (IM)
  • Assessment (AA)
  • Understanding (SU)
  • Unfamiliarity (UNFAM)
  • Scale Drivers
  • Development Flexibility (FLEX)
  • Process Maturity (PMAT)
  • Precedentedness (PREC)
  • Arch/Risk Resolution (RESL)
  • Team Cohesion (TEAM)
  • Product Effort Multipliers (EM)
  • Required Reliability (RELY)
  • Product Complexity (CPLX)
  • Required Reuse (RUSE)
  • Documentation (DOCU)
  • Platform EM
  • Platform Volatility (PVOL)
  • Bandwidth (BAND)
  • Personnel EM
  • Senior Capability (SCAP)
  • Developer Capability (DCAP)
  • Personnel Continuity (PCON)
  • Applications Experience (APEX)
  • Platform Experience (PLEX)
  • Development Tools Experience (DTEX)
  • Project EM
  • Lifecycle Tools (LIFE)
  • Multisite Development (SITE)
  • Required Development Schedule (SCED)

Change
New
Change
Change
Change
Change
30
COCOMO-SCORM Drivers(24 Input Variables)
  • Size of Product
  • Hours of Courseware
  • Level of Instruction
  • New, Reused, Modified Courseware
  • Requirements Evolution (Content Discarded)
  • Product Effort Multipliers (EM)
  • Required Reliability (RELY)
  • Product Complexity (CPLX)
  • Required Reuse (RUSE)
  • Documentation (DOCU)
  • Platform EM
  • Bandwidth (BAND)
  • Platform Volatility (PVOL)
  • Personnel EM
  • Senior Capability (SCAP)
  • Developer Capability (DCAP)
  • Personnel Continuity (PCON)
  • Applications Experience (APEX)
  • Platform Experience (PLEX)
  • Development Tools Experience (DTEX)
  • Project EM
  • Lifecycle Tools (LIFE)
  • Multisite Development (SITE)
  • Required Development Schedule (SCED)
  • Scale Factors
  • Precedentedness (PREC)
  • Development Flexibility (FLEX)
  • Arch/Risk Resolution (RESL)
  • Team Cohesion (TEAM)
  • Process Maturity (PMAT)

Change
Change
Change
New
Change
Change
Change
New
31
Create COCOMO-SCORM Model
32
COCOMO-SCORM Drivers(24 Input Variables)
  • Size of Product
  • Hours of Courseware
  • Level of Instruction
  • New, Reused, Modified Courseware
  • Requirements Evolution (Content Discarded)
  • Product Effort Multipliers (EM)
  • Required Reliability (RELY)
  • Product Complexity (CPLX)
  • Required Reuse (RUSE)
  • Documentation (DOCU)
  • Platform EM
  • Bandwidth (BAND)
  • Platform Volatility (PVOL)
  • Personnel EM
  • Senior Capability (SCAP)
  • Developer Capability (DCAP)
  • Personnel Continuity (PCON)
  • Applications Experience (APEX)
  • Platform Experience (PLEX)
  • Development Tools Experience (DTEX)
  • Project EM
  • Lifecycle Tools (LIFE)
  • Multisite Development (SITE)
  • Required Development Schedule (SCED)
  • Scale Factors
  • Precedentedness (PREC)
  • Development Flexibility (FLEX)
  • Arch/Risk Resolution (RESL)
  • Team Cohesion (TEAM)
  • Process Maturity (PMAT)

33
Quantitative Size Variables
34
Qualitative Project Variables
Effort Multipliers
Scale Factors
Nominal/Standard Levels
35
EM Effect on Project
15 Effort Multipliers for Nominal Project
Course Hours
Project Cost
Project Cost A Course
Hours (Person Months) (Constant) (Student Hours)
36
Answers Effect Project Cost
15 Effort Multipliers for More Realistic Project
Project Cost
Course Hours
Nominal Cost
Project Cost A EMi
Course Hours (Person Months) (Constant)
(Modifiers) (Student Hours)
37
Range of Cost Variation
Max X 115 ?
Course Hours
Nominal Cost
Flex From X times larger to Y times smaller
Min Y 0.05 ?
38
SF Effect on Project
All Scale Factors Set to Nominal
SW Size
Project Cost
Very Low
L
Low
Nominal
High
Very High
Extra High
0.0 (always)
39
SF Effect on Project
Scale Factors Other Than Nominal
Est. Cost
SW Size
Nom. Cost
Very Low
L
Low
Nominal
High
Very High
Extra High
0.0 (always)
40
Range of Scale Effect
Max X 1.226
SW Size
Project Cost
Min Y 0.91
Flex 22.6 Higher 9 Lower
41
COSCOMO Model Form
15
PM A(Size)E P EMi
i1
5
E B .01S SFj
j1
Where PM effort in Person Months A
calibration constant derived from historical
project data Size Adjust Source Lines of Code
E represents diseconomy of scale (composed
of 5 scale factors) EM effort multiplier for
the ith cost driver. The geometric product
results in an overall effort adjustment factor to
the nominal effort. B calibration constant
derived from historical project data SF scale
factor for the jth cost driver. Provides
organization specific adjustments to the size of
the project.
42
Data Collection
43
Projects Collected
  • Defense Nuclear Weapons School, Dean Marvin
  • Air Force Accounting Liaison Course, Gary Twogood
  • Explosive Hazard Awareness Course, Larry Helms
  • (more to follow )
  • Leads
  • Capt Scott Loller, Air Force Distance Learning
  • Angela Lindsey, Joint Forces Staff College
  • Maj Chris Edwards, UK Training Advisory Group
  • Patricia Mulligan, Formerly AFAMS

44
Data Collection Form
(see Excel file for entire form)
45
COSCOMO Estimates
Survey was not completed. All remaining
variables were estimated.
46
Defense Nuclear Weapons School
47
AF Accounting Liaison Course
48
Explosive Hazard Awareness
49
Model Calibration
50
Building PRED(30) for COSCOMO
Total Project Cost (PM)
DNWS -3.8
78
ALC 333
24
EHA -20
6
0
-30
30
Variance of Estimated Value
51
Calibration of Variables
Effort Multipliers
Scale Factors
Nominal/Standard Levels
52
Delphi Survey
  • Using Delphi Method to gather independent
    opinions about the level of effort associated
    with each setting on each variable.
  • Collect individual responses
  • Calculate group average
  • Present averages to each person and allow them to
    change their responses if they would like
  • Convene group work session to settle on
    acceptable values
  • Repeat if necessary
  • Surveys distributed to
  • Dean Marvin, Kelly Ward, Denise Stevens, Tim
    Richey, Tatjana Pitts

53
Conclusion
54
Completed Work
  • Technical and Management Work Plan
    (DI-MGMT-81117)
  • Researched COCOMO II
  • COCOMO published literature
  • COSYSMO seminar and relationship with Dr. Ricardo
    Valerdi
  • Researched SCORM Cost Drivers
  • ADDIE process steps
  • SCORM project cost drivers
  • Created COCOMO-SCORM Algorithm
  • Transformed COCOMO variables into COSCOMO tool
  • Created new variables for COSCOMO

55
Current Work
  • Creating COCOMO-SCORM Software Model
  • Functional software tool
  • Data Collection
  • Collect historical project data
  • Model Calibration Validation
  • Calibrate model to fit historical data and expert
    knowledge
  • Build Community
  • Create a community of interest around the method
    and tool

56
Next Steps
  • Complete COCOMO-SCORM Model
  • Microsoft Excel implementation
  • Accompanying Documentation
  • Create Prototype GUI
  • Final Project Review
  • Phase 2 (optional)

57
Right on Track
Write a Comment
User Comments (0)
About PowerShow.com