Overview of the PADI Design System

1
Overview of the PADI Design System
Robert Mislevy, University of Maryland Geneva
Haertel, SRI International Robert Murphy, SRI
International
2
The Problem

• Creating good science inquiry tasks is hard.
• How to do it both soundly and efficiently?
• Whats the link with science standards?
• Whats the psychological perspective represented?
• How to design assessments for different content
  areas ages of students?
• How do you score it?
• How to sort out confounded evidence?
• How to leverage the use of technology?

3
The PADI Solution (1)

• Conceptual frameworks and representational
  (epistemic) forms
• Design patterns for assessing inquiry
• Task-design templates
• Assessment delivery system
• Object model for task design
• Software tools to aid the design process
• Optional scoring and calibration engine to sort
  out evidence

4
The PADI Solution (2)

• Libraries of exemplars
• Design Patterns for assessing inquiry
• Task templates and pointers to tasks
• Worked-through applications
• FOSS, BioKIDS GLOBE and others
• Evaluation studies
• FOSS, BioKIDS

5
Project Objectives

• To produce a conceptual framework and collection
  of design tools and resources for supporting the
  design of assessments of science inquiry.
• Formulate a design framework for science inquiry
  that is extensible, shareable, and practical
• Lay out the framework in a open-system object
  model
• Develop supporting software to create and work
  with design patterns and task templates
• Develop an optional scoring and calibration
  engines for the sorting out of evidence
• Provide an initial set of high quality exemplars
  to highlight the elements of a coherent
  assessment argument.
• Test out the utility of the framework, object
  model, and tools with applications (GLOBE, FOSS,
  BioKids etc.)

6
Scope of PADI project

• Not an authoring system. Conceptual framework and
  object model provide infrastructure around which
  tailored authoring systems can be built.
• Not a delivery system. Shared conceptions,
  representations, and object definitions enhance
  the efficiency of delivery system design by
  providing a common infrastructure (i.e. delivery
  system should be consistent with principles of
  ECD).
• Does not set out to define inquiry based on a
  single perspective. Structures for expressing
  assessment arguments (Design Patterns) and
  instantiating them in tasks (Task Templates)
  independent of the psychological perspective.

7
Main Components of PADI
PADI
Science Inquiry
Other
examples
Science
PADI
Cog Psych
Libraries
Standards
Literature
Applications
Exemplars
Design Patterns
Templates
Globe
BioKIDS
FOSS
Conceptual
Conceptual
models
models
Conceptual
Application
models
Application
Design
Design

Application
Design
Implementation
Implementation
Implementation
Evaluation study
8
Evidence Centered Design

• The assessment problem To draw inferences about
  student proficiencies based on limited
  observations.
• Evidence-Centered Design Defines elements of the
  assessment argument and relationships between
  them.
• What does it mean to know and do inquiry?
• What constitutes evidence of knowing?
• How can that evidence be elicited from students?
• What are the appropriate statistical techniques
  for making valid student inferences about what
  students know from what students do?
• PADI is implementing a special case of ECD to the
  study of science inquiry.

9
Evidence-Centered Design (1)
10
Assessment Design Models (2)
11
Assessment Design Models (3)
12
Assessment Design Models (3a)
13
Paradigms for Assessing Science Inquiry
14
Software Support Architecture for Designing
Assessment Objects
15
(No Transcript)
16
Evaluation Studies
17
Libraries of Assessment Objects
18

• PADI focus is on ...
• Design framework (Design patterns, templates,
  delivery system)
• Assessing inquiry in science
• PADI focus is not on ...
• Tasks, per se (although will be examples, through
  FOSS, BioKIDS, other exemplars)
• Authoring systems
• Delivery systems
• (exceptions scoring and calibration engine,
  GradeBook)