Westerns Performance Accountability System: Using Electronic Portfolios and Teacher Work Samples to

1
Westerns Performance Accountability System
Using Electronic Portfolios and Teacher Work
Samples to Provide Evidence of Teacher Graduates
Ability to Impact P-12 Student Learning

• Roger Pankratz
• Western Kentucky University

2
Westerns Actions to Become Accountable for
Impact on P-12 Learning

• Shift in focus from teaching to learning 1998
• Collaboration with ten Renaissance Group
  institutions 1998
• Application and acquisition of a Title II
  Partnership for Teacher Quality Grant - 1999

3
Eleven Project Sites

• CSF California State, Fresno
• EMU Eastern Michigan
• ESU Emporia State
• ISU Idaho State
• KSU Kentucky State
• LCV Longwood, Virginia
• MTS Middle Tennessee State
• MUP Millersville, Pennsylvania
• SEM Southeast Missouri State
• UNI University of Northern Iowa
• WKU Western Kentucky

4
Eleven Partnerships
5
Partner Schools

• CSF Central Unified School District
• EMU Ypsilanti Public School
• ESU Olathe School District
• ISU League of Schools / Magic Valley School
  Partnership
• KSU Franklin County Public School
• LVC Regional 8 Superintendents Network
• MTS Metro Nashville-Davidson / Rutherford
  County
• MUP School District of Lancaster
• SEM Charles R-1 School District
• UNI Waterloo Community Schools
• WKU Bowling Green City / Warren County Schools

6
Project Goals

• Become accountable for the impact of teacher
  candidates on P-12 student learning
• Improve teacher performance in key areas and show
  an increase in teachers ability to facilitate
  learning of all students

7
Project Objectives

• Accountability Systems
• Teacher Work Samples
• Team Mentoring
• Program Redesign
• Business Partnerships
• Networking
• Research and Dissemination

8
Mandate for Accountability

• NCATE/Kentucky Standards
• CPE Goals
• Bush Administration Agenda

9
NCATE 2000 StandardTarget Performance

• Teacher candidates accurately assess and analyze
  student learning, make appropriate adjustments to
  instruction, monitor student learning and have a
  positive effect on learning for all students.
• NCATE Standard p. 16

10
NCATE 2000 StandardTarget Performance

• The unit is implementing its assessment system
  and providing regular and comprehensive data on
  program quality, unit operations and candidate
  performance at each stage of a program, including
  first years of practice.
• 
  
  NCATE Standard p.22

11
CPE Goals for Improving Teacher Quality

• Goal 1 Make the improvement of P-12 teacher
  quality (both at the preparation level and at the
  practicing teacher level) a top priority in all
  state supported universities.

12
CPE Goals for Improving Teacher Quality

• Goal 2 Develop accountability systems at each
  institution to provide credible evidence that
  candidates can facilitate the learning of all
  students and that graduates at all levels are
  making an impact on student learning.

13
CPE Goals for Improving Teacher Quality

• Goal 3 Develop valid and reliable performance
  assessments for Kentuckys New Teacher Standards
  and Kentuckys Experienced Teacher Standards that
  can be used for exit of teacher preparation
  programs, Rank II and Rank I programs as credible
  evidence that the benchmarks of performance
  adopted by the Education Professional Standards
  Board are met.

14
Bush Administration Agenda for Education

• Reed Lyons Presentation to Renaissance Group
  03/25/02
• Accountability for performance of graduates
• Flexibility in preparation of practitioners
• More choices for parents in P-12 schooling
• Improvement in quality of education research

15
Westerns Components for an Accountability System

• Comprehensive data management system
• Ongoing performance assessment system of teacher
  candidates
• Mentoring teams (teacher educators, arts and
  science faculty and school practitioners)
• A research program that links teacher performance
  to student learning

16
Westerns Data Management System

• Aligned with teacher standards
• Multiple measures of performance
• Multiple data points (admissions induction)
• Provides analysis of student, course, program and
  faculty
• Performance/Progress

17
Data Collection Sources

• Admissions data
• Electronic portfolio data
• Field experience data
• Exit data
• Post-graduation follow-up
• data

18
Admissions Data Include(see the next few pages
for a sample)

• Demographic Information
• Major/Major Number
• Teacher Orientation Date
• Recommendations
• Writing Sample Score
• Hearing/Speech Screening
• Certification and Degree Program Dates

• GPA at Admission
• ACT, SAT, PPST, or GRE scores
• Certification Codes
• Committee Approval Date
• Semester Hours Completed at Admission
• Estimated Graduation Date
• Criminal and Health Check Dates

19
(No Transcript)
20
(No Transcript)
21
(No Transcript)
22
Electronic Portfolio Data

• Record of completion of Critical Performances
  aligned with Kentuckys New Teacher Standards
  that includes
• Performance scores
• Performance exhibits
• Time of completion
• Number of attempts toward completion (in the
  process of being added)

23
(No Transcript)
24
Field Experience Data

• Documentation of location/hours
• Documentation of diversity experiences
• Documentation of special needs student experiences

25
Exit Data(In the process of being added to
Electronic System)

• Documentation of student teaching experiences
• Teacher Work Sample evaluation
• Student teaching observation evaluation
• Exiting GPA
• Praxis II scores

26
Post-Graduation Follow-Up Data

• Using the shared Renaissance instrument, new
  teachers are surveyed regarding their preparation
  at WKU.
• Paper versions were mailed and an on-line
  electronic version has been made available.
• So far, 92 alumni have responded.
• Data has been entered but not analyzed.

27
System Requirements

• Dual CPU server with 180 gigabytes set up as a
  Raid 5 system with redundant power supplies and
  processors
• Auto loading tape system for backup
• Data systems coordinator at ½ FTE release time
• Countless hours of technical support over two
  years of development

28
Electronic Portfolio Systems Server
29
Westerns Performance Assessment System

• Aligned with Kentucky Teacher Standards
• Based on Four Levels of Critical Performance
• Uses Electronic Portfolios to record, store and
  manage performance data
• Uses Teacher Work Samples in student teaching as
  evidence of candidates ability to facilitate
  learning

30
Kentucky New Teacher Standards 1993

• Design instruction
• Create learning climate
• Implement instruction
• Assess learning progress
• Reflect on and evaluate teaching
• Collaborate with parents and professionals
• Engage in professional development
• Demonstrate content knowledge
• Use technology for instruction

31
Critical Performance

• Specific descriptions about what teacher
  candidates must know and be able to do at
  different levels of growth and development toward
  one or more teaching standards.

32
(No Transcript)
33
Critical Performances Related to Nine New Teacher
Standards
34
Critical Performances for New Teacher Standard IV
Assessment of Student Learning
35
Critical Performances Profile for Jane Doe
36
Teacher Work Samples in Student Teaching

• A major source of data for Level IV critical
  performances that provide evidence of candidates
  ability to facilitate learning of all students.

37
Teacher Work Samples

• A process that enables teacher candidates to
  demonstrate teaching performances directly
  related to planning, implementing, assessing
  student learning and evaluating teaching and
  learning a four-week standards-based
  instructional unit.

38
Teacher Work Samples Focus on Seven Teacher
Processes

• Use of context
• Learning goals
• Assessment plan
• Design for instruction
• Instructional decision making
• Analysis of student learning
• Reflection and self-evaluation

39
Research on Teacher Preparation 1996-1999

• Performance Areas Where New Teachers Said They
  Were Least Prepared
• Ability to address student discipline problems
• Ability to use a variety of assessments including
  portfolios, open-ended questions and performance
  events
• Ability to design instruction for students of
  diverse ethnic and cultural backgrounds
• Ability to identify professional development
  options and plan a program for your professional
  growth
• Ability to design instruction to achieve
  Kentuckys learning goals and academic
  expectations

40
Responses from 800 New Teachers in Kansas
41
Kentucky Scholastic Audit

• What Distinguishes Successful Schools from
  Unsuccessful Schools
• Intentional verbal communication about curriculum
• Test scores used to identify curriculum gaps
• Samples of student work used to inform
  instruction
• Alignment of instruction and assessment with
  state learning goals

42
Teacher Work Samples Components

• Teaching process standards
• Teaching performance prompt
• Scoring rubric
• Twenty page narrative plus attachments teaching
  exhibit
• lthttp//fp.uni.edu/itqgt

43
(No Transcript)
44
(No Transcript)
45
(No Transcript)
46
TWS Development at WKU

• Fall 2000
• 7 elementary ed candidates
• Spring 2001
• 17 elementary ed candidates
• Fall 2001
• 42 elementary ed candidates
• Spring 2002
• 115 elementary, 22 middle, 48 secondary ed, 30
  other candidates

47
Performing Scores on 134 TWS Exhibits at 8
Institutions
48
Performance Level of Teacher Candidates Based on
High Standards

• Spring 2001 Fall 2002
• n110 n134
• Beginning (1) 16 18
• Developing (2) 42 48
• Proficient (3) 35 29
• Expert (4) 07 05

49
Establishing Credibility Evidence for Teacher
Work Samples

• Training for Scoring
• Benchmarking
• Reliability
• Validity
• Impacts on PK-12 Student Learning

50
Training for Scoring

• Review Teaching Processes and Standards
• Study TWS Prompt and Scoring Rubrics
• Agree on Road Map of Evidence
• Develop Indicator Definitions
• Conduct Anti-Bias Training
• Practice Scoring

51
Benchmarked Performances

• Proto-typical performances at each level of the
  developmental scoring rubric.
• 1 Beginning
• 2 Developing
• 3 Proficient
• 4 Expert
• Benchmarks are used for training of scorers and
  mentoring of candidates.

52
TWS Holistic Score Distribution January 2002

• Beginning 18
• Developing 48
• Proficient 29
• Expert 05
• n 135

53
Inter-Rater Reliability Total Analytic
Scores

• Extent to which scoring decisions are consistent
• across expert judges using the analytic rubric.
• Dependability Coefficients
• (Shavelson Webb, 1991)
• 5 raters .88
• 3 raters .82
• 1 rater .60

54
Inter-Rater Reliability Analytic
Sub-Scales
Sub-Scale
5 Raters
3 Raters
1 Rater
Contextual Factors
.86
.79
.55
Learning Goals
.85
.77
.53
Assessment Plan
.86
.78
.54
Design for Instruction
.86
.79
.55
Instructional Decision-Making
.78
.78
.42
Analysis of Student Learning
.82
.74
.48
Reflection and Self-Evaluation
.86
.79
.56
55
Inter-Rater Reliability
Trained raters are able to score with sufficient
inter-rater agreement to make high-stakes
decisions about performance levels. Moreover,
scores are sufficiently dependable that reliable
decisions can be made for each teaching process
separately.
56
Validity
Expert judgments of . . .

• Content Representativeness (Crocker,
  1997)
• - Realism
  - Criticality
  - Necessity
  - Frequency
• Alignment with INTASC Principles
  

57
Validity Realism
98.1 of the raters strongly agreed with the
statement The performance tasks required by
the TWS represent actual lessons delivered to
students in PK-12 school classrooms.
58
Validity Criticality
97.5 of the expert judges rated the tasks
required by the TWS as critical or important
to teaching.
59
Validity Frequency
All of the tasks required by the TWS were rated
by the expert judges as high frequency activities
for teachers 90 of more of the raters said
weekly or daily for all of the tasks.
60
Validity Necessity
Overall, does the TWS assess knowledge and skills
that are necessary for beginning
teachers? Absolutely 70 Yes 30 No 0
61
Alignment with INTASC Principles
INTASC Principle
Directly
Implicitly
Not at All
5
45
50
Subject matter
50
5
45
Human development and learning
2
8
95
Adapting instruction
20
2
78
Instructional strategies
7
50
43
Classroom motivation and management
30
58
12
Communication skills
2
3
95
Instructional planning skills
0
7
93
Assessment of student learning
7
40
53
Professional commitment/responsibility
Partnerships
55
38
7
62
Alignment with INTASC Principles
INTASC Principle
Directly
Implicitly
Not at All
5
45
50
Subject matter
50
5
45
Human development and learning
2
8
95
Adapting instruction
20
2
78
Instructional strategies
7
50
43
Classroom motivation and management
30
58
12
Communication skills
2
3
95
Instructional planning skills
0
7
93
Assessment of student learning
7
40
53
Professional commitment/responsibility
Partnerships
55
38
7
63
Benefits of TWS

• Focus on P-12 student learning
• Focus on development of assessment tools
• Focus on key standards
• Alignment of standards, curriculum and assessments

64
Westerns Progress on Performance Assessments

• Teacher Work Sample implemented
• Most critical performances identified
• Electronic portfolio operational
• Course/experience alignment in process

65
Westerns Progress on Its Accountability System

• Electronic data management system developed
• Data sources and entry points identified
• Baseline data entered
• Research data entered
• Research program initiated

66
WKUS VISIONNext Steps
67
Enhancing the Electronic System Student Progress
Profiles

• Informing students and faculty about progress on
  performances and other program requirements

68
Enhancing the Electronic System Course Profiles

• Recording student completion and success on
  critical performances

69
Enhancing Critical Performances

• Faculty development, discussion, and adoption of
  Alignment Maps
• Filling gaps on Maps
• Re-evaluating alignment and purpose of existing
  critical performances
• Obtaining faculty feedback and student data to
  begin establishing reliability and validity

Alignment Map (click above to open)
70
Enhancing the Capacity of the Data Management
System

• Using data to answer key questions
• How well prepared is candidate X with respect to
  KY New Teacher Standards?
• How well prepared is candidate X to facilitate
  learning of all students?
• What are the candidates strengths and
  weaknesses?
• What factors contribute to these strengths and
  weaknesses?
• What are the programs strengths and weaknesses?
• What aspects of the program contribute to these
  strengths and weaknesses?

71
(No Transcript)
72
(No Transcript)
73
(No Transcript)
74
(No Transcript)
75
(No Transcript)
76
(No Transcript)
77
(No Transcript)
78
Challenges to Developing An Accountability System

• Obtaining faculty buy-in
• Finding resources for ongoing system development
  and maintenance
• Maintaining high standards while fostering
  creativity
• Expanding use of valid and reliable assessment
  tools
• Using context to design instruction
• Making credible links between candidate
  performance and impact on P-12 learning

79
What Western and Renaissance Partnership
Institutions Have Learned About Essential
Elements for Developing Performance
Accountability Systems

• Commitment to accountability and data management
• Designated accountability system coordinator with
  released time
• Development resources

80
What Western and Renaissance Partnership
Institutions Have Learned About Essential
Elements for Developing Performance
Accountability Systems

• Development process involving a majority of
  faculty
• Data sources
• Data gathering points
• Standards of performance
• Indicators of performance
• Scoring rubrics
• Performance tasks (prompts)
• Benchmarking of performances
• Validation of systems
• Scorer reliability

81
What Western and Renaissance Partnership
Institutions Have Learned About Essential
Elements for Developing Performance
Accountability Systems

• Resources for scale-up and maintenance of system
• Faculty
• Training materials
• Data management

82
Performance Assessment in KTIP

• Standards of Performance Yes
• Indicators of Performance Yes
• Scoring Rubrics No
• Performance Tasks (Prompts) Some
• Benchmarked Performances No
• Scorer Reliability - No

83
Persons to Contact About Westerns Accountability
System

• Sam Evans
• sam.evans_at_wku.edu
• Tony Norman
• antony.norman_at_wku.edu
• Tabitha Toby Daniel
• tabitha.daniel_at_wku.edu

84
(No Transcript)
85
(No Transcript)
86
(No Transcript)
87
Hands-on Teacher Work Samples - I

• Form groups of three or four
• Quick read TWS Exemplar
• Use the rubric and scoring guide to rate process
  2 (goals), process 6 (analysis of student
  learning) and process 7 (reflection and
  evaluation) as
• 1beginning
• 2developing
• 3proficient
• 4expert
• Share your ratings

88
Hands-on Teacher Work Samples - II

• Identify three strengths of Teacher Work Samples
• Identify three concerns or challenges you see in
  implementing Teacher Work Samples