Using the PADI Design System to Examine the Features of a NAEP Performance Assessment

1
Using the PADI Design System to Examine the
Features of a NAEP Performance Assessment

• Kathleen C. Haynie
• Kathleen Haynie Consulting
• Andrea A. Lash, Geneva D. Haertel,
• Edys S. Quellmalz, Angela Haydel DeBarger
• SRI International

2
Background

• Use of 21st century collaborative workplace tools
  is on the rise
• Technologically rich dispersed collaboration
  requires (Olson Olson, 2000)
• Common ground
• Coupling of work
• Technological readiness
• Collaborative readiness

3
PADI Project

• Seeks to improve assessment of inquiry in science
  learning
• Networked collaboration
• Development and use of Web-based tool

4
Central Process

• The central process for project strands working
  with the PADI Design System can be stated as
  follows
• The interaction between assessment materials
  (e.g., assessment tasks) and the
  Evidence-Centered Design framework (Mislevy,
  Steinberg, Almond, 2002) is mediated by various
  tools, representational forms, and
  representations utilized by collaborative
  workgroups.

5
PADI Assessment Design
6
NAEP Performance Assessment

• Goals
• To reverse engineer a performance task from a
  large-scale, national assessment
• To use the PADI System as an analytical tool for
  understanding task features
• In reverse engineering and analyzing a
  performance assessment task via PADI, what types
  of knowledge were created?

7
Reverse Engineering Analysis Process

• 1. Selection of a Set of Items (July August)
• 2. Exploration of the Floating Pencil Task
  (August December)
• 3. Development of a Task Specification (December
  March)

8
The Student Model

• We selected a set of items for analysis
• NSES inquiry standards, PADI Design Patterns
• Defined our Student Model
• Exploration of multiple student models
• Choice of NAEP framework
• Refined the Student Model

9
The Evidence Model

• We explored evaluation measurement issues via
  the PADI representational forms
• Evaluative Sub-Model based on NAEP rubric
• Multiple levels of conditional dependencies
• MRCML psychometric model
• Clarified our Measurement Model
• Based closely on NAEP practices
• Elements of PADI

10
The Task Model

• We explored attributes of the Floating Pencil
  task
• Content-lean, inquiry-constrained
• Use of lab materials High verbal demand
• Constructed Task Model
• Sought to define a Family of Tasks

11
So What?

• Process Floating Pencil team engaged in
  collaboration using technological workplace tools
• Results
• Analysis of the Floating Pencil task
• A trace on PADI Floating Pencil Task
  Specification
• The impact of our work on PADI Design System
• Next Steps
• Task family and TMVs
• Empirical data analysis
• Technical report on Floating Pencil work
• Consideration of alternative student models