The Thinking Behind PACT Performance Assessment for California Teachers

1
The Thinking BehindPACTPerformance
Assessment for California Teachers

• Raymond Pecheone
• Stanford University
• Nov. 9, 2007

1
2
The PACT Assessment System

• A performance assessment for teacher candidates
  created in response to SB 2042, with new subject
  matter standards, new program standards, and new
  assessment standards
• Alternate assessments permitted must meet
  California Quality Standards of
  reliability/validity (i.e., AERA/APA test
  standards).
• Aligned with the California Teaching Performance
  Expectations (standards) and California
  Curriculum Frameworks
• High stakes assessment designed to initially
  license beginning teachers

2
Nov 4, 1218 am 2.5 h Nov 4, 435 pm
2.25 h
4.45 h
3
California Teaching Performance Expectations

• TPE 1 Specific Pedagog-ical Skills for Subject
  Matter Instruction
• TPE 2 Monitoring Student Learning During
  Instruction
• TPE 3 Interpretation and Use of Assessments
• TPE 4 Making Content Accessible
• TPE 5 Student Engagement
• TPE 6 Developmentally Appropriate Teaching
  Practices
• TPE 7 Teaching English Learners
• TPE 8 Learning about Students
• TPE 9 Instructional Planning
• TPE 10 Instructional Time
• TPE 11 Social Environment
• TPE 12 Professional, Legal, and Ethical
  Obligations
• TPE 13 Professional Growth

3
4
PACT Institutions

• UC Berkeley
• UC Davis
• UC Irvine
• UCLA
• UC Riverside
• UC San Diego
• UC Santa Barbara
• UC Santa Cruz

• Cal Poly SLO
• CSU Channel Islands
• CSU Chico
• CSU Dominguez Hills
• CSU Monterey Bay
• CSU Northridge
• Humboldt State
• Sacramento State
• San Diego State
• San Francisco State
• San Jose State
• Sonoma State

• Stanford
• Holy Names University
• Mills College
• Notre Dame de Namur University
• Pepperdine University
• St. Marys College of California
• University of the Pacific
• University of San Diego
• Antioch University
• USC
• San Diego Intern

4
5
The PACT Assessment SystemElectronic Portfolio
Elements
The Capstone Teaching Event
Assessments Embedded in Local Programs examples

• Teaching Event
• Demonstrates
• Planning
• Instruction
• Assessing
• Reflecting
• Academic Language

Child Case Studies
Analyses of Student Learning
Curriculum /Teaching Analyses
Observation/Supervisory Evaluation Feedback
5
6
Teaching Event Records of Practice
6
7
Teaching Event Subject Areas

• Multiple Subjects
• Literacy
• Mathematics

• Single Subject
• Agriculture
• English language arts
• History social science
• Mathematics
• Science
• Art
• Music
• Physical Education
• World languages

7
8
PACT Teaching Event DNA

• Documents teaching of learning segment (3-5
  lessons or hours of instruction)
• Subject specific
• Standardized tasks core questions across
  programs
• Scored with common rubrics, passing standard
• During student teaching

8
9
What is Subject Specific about the Teaching Event?

• Focus of learning segment aligned to Ca.
  content standards
• Teaching/learning tasks on video clip(s)
• Additional prompts in some content areas (e.g.,
  misconceptions in science, dispositions in
  mathematics, description of text in
  English/language arts)
• Benchmarks

9
10
2-Day Subject Specific Scorer Training

• DAY 1
• Overview of PACT Teaching Event and scoring
  process
• Discussion on bias
• Note taking and Documentation
• Understanding Level 2
• DAY 2
• Understanding Level 1
• Understanding Level 3
• Independently score a Calibration Teaching Event
  Debrief

10
11
Scoring

• Trained and calibrated subject specific assessors
• Campus based with central audits regional
  scoring
• Rubric based scoring in real time (web based
  platforms)
• Organized around dimensions of teaching (PIARA)
  and guiding questions
• Sequentially Scored By Tasks

11
12
Guiding Questions and Analytic Rubrics

• PLANNING
• Establishing a Balanced Instructional Focus
• Making Content Accessible
• Designing Assessments
• INSTRUCTION
• Engaging Students in Learning
• Monitoring Student Learning During Instruction

• ASSESSMENT
• Analyzing Student Work From an Assessment
• Using Assessment to Inform Teaching
• REFLECTION
• Monitoring Student Progress
• Reflecting on Teaching
• ACADEMIC LANGUAGE
• Understanding Language Demands
• Supporting Academic Language Development

12
13
PACT Rubrics (one example)
13
14
Data Charts 2003-04Campus Scores
14
15
Data Charts 2003-04Content Area Scores
15
16
PACT ScoresAssessment of Student Learning (2003
- 2005)
16
17
What We Learn from the PACT Analyses

• How our candidates do
• On different aspects of teaching
• In different subject areas
• In comparison to other institutions
• Over time
• With different kinds of supports

17
18
Actions
Scoring theTE
Program Meetings
Professional Development
PACT Advisor
Analysis of Candidate Work
Collaborativeplanning acrossUniversity K-12
schools
18
19
Faculty Learning Program Improvement

• Increased articulation across courses, structures
  and roles
• Changes in content of some courses
• Structural changes in Teacher Education Program

19
20
PACT ScoresInter-rater Reliability
Sample Size 2,580 Spearman-Brown Reliability
Estimate 0.88
20
21
PACT Validity Studies

• Content validity
• Development teams, Program directors, Program
  faculty, Leadership team
• TPE alignment study
• Concurrent validity
• Evaluation of score validity
• Decision Consistency Holistic vs. Analytic
  ratings
• Bias and fairness review

• Construct validity
• Factor Analysis(2002-03 Pilot Year)
• Reflection Assessment
• Instruction
• Planning
• Predictive Validity (Carnegie/CT Study)

21
22
The Research Base for Teacher Licensing Tests

• Weak relationship between traditional licensing
  tests and teacher effectiveness (NRC, 2001)
• Strauss Sawyer (1986)
• Ferguson (1991, 1998)
• Ferguson Ladd (1996)
• Clotfelter, Ladd, Vigdor (forthcoming)
• Goldhaber (2005, 2006)
• Effect sizes quite small in recent value added
  research (.01 .06)

22
23
For More Information...

• See Teaching Event Handbooks and Rubrics at
  www.pacttpa.org.

23
24
Educative Assessment

• Teachers Matter
• Subject Matter Matters
• Building Knowledge Matters
• Authenticity Matters
• Integration of Practice Matters

24