PADI Principled Assessment Design for Inquiry: PADI Design Patterns

1
PADIPrincipled Assessment Design for
InquiryPADI Design Patterns
Robert J. Mislevy University of Maryland
PADI is supported by the National Science
Foundation, through the IERI program. The prime
contractor is SRI, International subcontractors
are the University of Maryland, The University of
California at Berkeley, the Lawrence Hall of
Science, and the University of Michigan. The
Co-Principal Investigators are Robert Mislevy
(Maryland) and Geneva Haertel (SRI).
2
The Problem

• Creating good science inquiry tasks is hard.
• How to do it both soundly and efficiently?
• Whats the connection with science standards?
• Whats the connection with learning psychology?
• How to implement with given content, with given
  kinds ages of students?
• How do you score it?
• How to sort out confounded evidence?
• How to leverage technology?

3
The PADI Solution (1)

• Conceptual frameworks for
• Design patterns for assessing inquiry
• Task-design
• Assessment delivery
• Object model for task design
• ECD work with Steinberg Almond
• Software tools to aid the design process

4
The PADI Solution (2)

• Scoring engine to sort out evidence
• optional for any given application
• Libraries of
• Design patterns for assessing inquiry
• Rubrics
• Task models and tasks
• Worked-through applications
• Globe, FOSS, BioKids

5
Main Components of PADI
PADI
Science Inquiry
Other
examples
Cog Psych
Science
PADI
Libraries
Standards
Literature
Applications
Design Patterns
Exemplars
Rubrics
Globe
BioKids
Foss
Conceptual
Conceptual
models
models
Conceptual
Application
models
Application
Design
Design
Anders/Edys
Application
Example
Design
Implementation
Implementation
Implementation
Evaluation study
6
Evidence-Centered Design (1)
Design
Evidence-
Models
centered design
(Conceptual)
Delivery
System
Architecture
7
Assessment Design Models (2)
8
Design Patterns for Assessing Science Inquiry
Science Inquiry
Other
examples
Science
Cog Psych
PADI
Standards
Literature
Applications
9
Software Support Architecture for Designing
Assessment Objects
Design
Tools
(Specific
incarnation)
Design
Three-tiered
Object
Use Cases
Software
architecture
Model
10
PADI Applications
PADI
Applications
Globe
BioKids
Foss
Conceptual
Conceptual
models
models
Conceptual
Application
models
Application
Design
Design
Anders/Edys
Application
Example
Design
Implementation
Implementation
Implementation
11
Design Patterns for Assessing Science Inquiry
Evidence-
PADI
Science Inquiry
centered design
PADI
Libraries
Applications
Design
Tools
(Specific
incarnation)_
Design Patterns
Globe
BioKids
Foss
12
Motivation for Design Patterns

• In-between structure, connecting...
• Thinking about science learning inquiry
• Technical elements of assessment design
  delivery
• Narrative, not technical, contents
• Schema reflects assessment argument structure

13
Analogues

• Design Patterns in Architecture
• Design Patterns in Software Engineering
• Levi-Strausss Structure of Myths
• Poltis Thirty-Six Dramatic Situations

14
Desirable Features for Design Patterns

• Comprehensible to content specialists
• Content ties in with standards research
• Form ties in with ECD design structure, as
  realized in PADI object model
• Open system
• Lots of talked-through examples

15
The Content of Design Patterns

• Whats IN design patterns
• Structured according to assessment argument
• Conceptions of inquiry skills and how you can
  recognize them when you see them
• Key features of situations and variable aspects
• Whats NOT IN design patterns
• Particular science content (This does not imply
  that inquiry skills content are separable wrt
  knowledge, learning, or assessment!)
• Particulars of tasks (e.g., format, technology)
• Assessment purpose
• Technical specs for authoring, delivering tasks

16
Sources of Inspiration

• NSES Standards
• Cog Psych science learning research
• Assessment design research
• What we need for partners
• Existing assessments (e.g., GLOBE)

17
Attributes of a Design Pattern

• Title. A short name for referring to the design
  pattern
• Summary. Overview of the kinds of assessment
  situations students encounter and what one wants
  to know about.
• Rationale. Why the DP is an important aspect of
  inquiry
• Focal KSAs. Primary knowledge/skills/attributes
  of students that one wants to know about
• Additional KSAs. Other KSAs that may be required
  
• Potential observations. Some possible things one
  could see students doing that would give evidence
  about the KSAs
• Potential work products. Modes or formats in
  which students might produce the evidence

18
Attributes of a Design Pattern, cont.

• Potential rubrics. Links to scoring rubrics that
  might be useful.
• Characteristic features. Characteristics of
  situations that give students the opportunity to
  provide the desired evidence.
• Variable features. Features in the situation that
  can be varied in order to shift its difficulty or
  focus
• I am kind of Links to other DPs this one is a
  special case of.
• Kinds of me. Links to DPs that are special cases
  of this one.
• I am part of Links to DPs this one is a
  component or step of.
• Parts of me. DPs that are components or steps of
  this one.
• Educational standards. Links to related NSES
  Standards

19
Attributes of a Design Pattern, cont.

• Templates (aka task/evidence shells). Links to
  templates, more technical PADI objects, that use
  this DP
• Exemplar tasks. Links to sample tasks that
  instantiate this DP
• Online resources. Links to online materials that
  illustrate or give backing for this DP.
• References. Pointers to research and other
  literature that illustrate or give backing for
  this design pattern
• Miscellaneous associations. Other relevant
  information.

20
(No Transcript)
21
Same Design Pattern, Different Tasks

• EXAMPLE Flight of the Maplecopter
• (Baxter Glaser, 1998)
• Design patterns addressed
• Viewing real-world situations from a scientific
  perspective
• Scientific reasoning (including planning solution
  strategy, implementing strategy, self-monitoring,
  and generating explanations)
• Variable features
• Provides cues to approach from scientific
  perspective
• Process is open (task does not provide
  step-by-step instructions)
• Requires prior content knowledge about force and
  motion that is not provided in the task

22
Walking through a GLOBE design pattern

23
Conclusion

• Advanced assessment requires the coordination of
  expertise from very different fields.
• Assessment design is both science and art
• Capitalize on schemas for recurring structures,
  patterns, relationships
• Explicit validity arguments
• Accumulated wisdom sharable--not context bound
• Interoperability of elements processes (esp.
  technological settings)