Westerns Performance Accountability System: Using Electronic Portfolios and Teacher Work Samples to - PowerPoint PPT Presentation

Loading...

PPT – Westerns Performance Accountability System: Using Electronic Portfolios and Teacher Work Samples to PowerPoint presentation | free to view - id: 3ab33-MmU4N



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

Westerns Performance Accountability System: Using Electronic Portfolios and Teacher Work Samples to

Description:

Westerns Performance Accountability System: Using Electronic Portfolios and Teacher Work Samples to – PowerPoint PPT presentation

Number of Views:93
Avg rating:3.0/5.0
Slides: 89
Provided by: rogerpa
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Westerns Performance Accountability System: Using Electronic Portfolios and Teacher Work Samples to


1
Westerns Performance Accountability System
Using Electronic Portfolios and Teacher Work
Samples to Provide Evidence of Teacher Graduates
Ability to Impact P-12 Student Learning
  • Roger Pankratz
  • Western Kentucky University

2
Westerns Actions to Become Accountable for
Impact on P-12 Learning
  • Shift in focus from teaching to learning 1998
  • Collaboration with ten Renaissance Group
    institutions 1998
  • Application and acquisition of a Title II
    Partnership for Teacher Quality Grant - 1999

3
Eleven Project Sites
  • CSF California State, Fresno
  • EMU Eastern Michigan
  • ESU Emporia State
  • ISU Idaho State
  • KSU Kentucky State
  • LCV Longwood, Virginia
  • MTS Middle Tennessee State
  • MUP Millersville, Pennsylvania
  • SEM Southeast Missouri State
  • UNI University of Northern Iowa
  • WKU Western Kentucky

4
Eleven Partnerships
5
Partner Schools
  • CSF Central Unified School District
  • EMU Ypsilanti Public School
  • ESU Olathe School District
  • ISU League of Schools / Magic Valley School
    Partnership
  • KSU Franklin County Public School
  • LVC Regional 8 Superintendents Network
  • MTS Metro Nashville-Davidson / Rutherford
    County
  • MUP School District of Lancaster
  • SEM Charles R-1 School District
  • UNI Waterloo Community Schools
  • WKU Bowling Green City / Warren County Schools

6
Project Goals
  • Become accountable for the impact of teacher
    candidates on P-12 student learning
  • Improve teacher performance in key areas and show
    an increase in teachers ability to facilitate
    learning of all students


7
Project Objectives
  • Accountability Systems
  • Teacher Work Samples
  • Team Mentoring
  • Program Redesign
  • Business Partnerships
  • Networking
  • Research and Dissemination

8
Mandate for Accountability
  • NCATE/Kentucky Standards
  • CPE Goals
  • Bush Administration Agenda

9
NCATE 2000 Standard Target Performance
  • Teacher candidates accurately assess and analyze
    student learning, make appropriate adjustments to
    instruction, monitor student learning and have a
    positive effect on learning for all students.
  • NCATE Standard p. 16

10
NCATE 2000 Standard Target Performance
  • The unit is implementing its assessment system
    and providing regular and comprehensive data on
    program quality, unit operations and candidate
    performance at each stage of a program, including
    first years of practice.


  • NCATE Standard p.22

11
CPE Goals for Improving Teacher Quality
  • Goal 1 Make the improvement of P-12 teacher
    quality (both at the preparation level and at the
    practicing teacher level) a top priority in all
    state supported universities.

12
CPE Goals for Improving Teacher Quality
  • Goal 2 Develop accountability systems at each
    institution to provide credible evidence that
    candidates can facilitate the learning of all
    students and that graduates at all levels are
    making an impact on student learning.

13
CPE Goals for Improving Teacher Quality
  • Goal 3 Develop valid and reliable performance
    assessments for Kentuckys New Teacher Standards
    and Kentuckys Experienced Teacher Standards that
    can be used for exit of teacher preparation
    programs, Rank II and Rank I programs as credible
    evidence that the benchmarks of performance
    adopted by the Education Professional Standards
    Board are met.

14
Bush Administration Agenda for Education
  • Reed Lyons Presentation to Renaissance Group
    03/25/02
  • Accountability for performance of graduates
  • Flexibility in preparation of practitioners
  • More choices for parents in P-12 schooling
  • Improvement in quality of education research

15
Westerns Components for an Accountability System
  • Comprehensive data management system
  • Ongoing performance assessment system of teacher
    candidates
  • Mentoring teams (teacher educators, arts and
    science faculty and school practitioners)
  • A research program that links teacher performance
    to student learning

16
Westerns Data Management System
  • Aligned with teacher standards
  • Multiple measures of performance
  • Multiple data points (admissions induction)
  • Provides analysis of student, course, program and
    faculty
  • Performance/Progress

17
Data Collection Sources
  • Admissions data
  • Electronic portfolio data
  • Field experience data
  • Exit data
  • Post-graduation follow-up
  • data

18
Admissions Data Include… (see the next few pages
for a sample)
  • Demographic Information
  • Major/Major Number
  • Teacher Orientation Date
  • Recommendations
  • Writing Sample Score
  • Hearing/Speech Screening
  • Certification and Degree Program Dates
  • GPA at Admission
  • ACT, SAT, PPST, or GRE scores
  • Certification Codes
  • Committee Approval Date
  • Semester Hours Completed at Admission
  • Estimated Graduation Date
  • Criminal and Health Check Dates

19
(No Transcript)
20
(No Transcript)
21
(No Transcript)
22
Electronic Portfolio Data
  • Record of completion of Critical Performances
    aligned with Kentuckys New Teacher Standards
    that includes
  • Performance scores
  • Performance exhibits
  • Time of completion
  • Number of attempts toward completion (in the
    process of being added)

23
(No Transcript)
24
Field Experience Data
  • Documentation of location/hours
  • Documentation of diversity experiences
  • Documentation of special needs student experiences

25
Exit Data (In the process of being added to
Electronic System)
  • Documentation of student teaching experiences
  • Teacher Work Sample evaluation
  • Student teaching observation evaluation
  • Exiting GPA
  • Praxis II scores

26
Post-Graduation Follow-Up Data
  • Using the shared Renaissance instrument, new
    teachers are surveyed regarding their preparation
    at WKU.
  • Paper versions were mailed and an on-line
    electronic version has been made available.
  • So far, 92 alumni have responded.
  • Data has been entered but not analyzed.

27
System Requirements
  • Dual CPU server with 180 gigabytes set up as a
    Raid 5 system with redundant power supplies and
    processors
  • Auto loading tape system for backup
  • Data systems coordinator at ½ FTE release time
  • Countless hours of technical support over two
    years of development

28
Electronic Portfolio Systems Server
29
Westerns Performance Assessment System
  • Aligned with Kentucky Teacher Standards
  • Based on Four Levels of Critical Performance
  • Uses Electronic Portfolios to record, store and
    manage performance data
  • Uses Teacher Work Samples in student teaching as
    evidence of candidates ability to facilitate
    learning

30
Kentucky New Teacher Standards 1993
  • Design instruction
  • Create learning climate
  • Implement instruction
  • Assess learning progress
  • Reflect on and evaluate teaching
  • Collaborate with parents and professionals
  • Engage in professional development
  • Demonstrate content knowledge
  • Use technology for instruction

31
Critical Performance
  • Specific descriptions about what teacher
    candidates must know and be able to do at
    different levels of growth and development toward
    one or more teaching standards.

32
(No Transcript)
33
Critical Performances Related to Nine New Teacher
Standards
34
Critical Performances for New Teacher Standard IV
Assessment of Student Learning
35
Critical Performances Profile for Jane Doe
36
Teacher Work Samples in Student Teaching
  • A major source of data for Level IV critical
    performances that provide evidence of candidates
    ability to facilitate learning of all students.

37
Teacher Work Samples
  • A process that enables teacher candidates to
    demonstrate teaching performances directly
    related to planning, implementing, assessing
    student learning and evaluating teaching and
    learning a four-week standards-based
    instructional unit.

38
Teacher Work Samples Focus on Seven Teacher
Processes
  • Use of context
  • Learning goals
  • Assessment plan
  • Design for instruction
  • Instructional decision making
  • Analysis of student learning
  • Reflection and self-evaluation

39
Research on Teacher Preparation 1996-1999
  • Performance Areas Where New Teachers Said They
    Were Least Prepared
  • Ability to address student discipline problems
  • Ability to use a variety of assessments including
    portfolios, open-ended questions and performance
    events
  • Ability to design instruction for students of
    diverse ethnic and cultural backgrounds
  • Ability to identify professional development
    options and plan a program for your professional
    growth
  • Ability to design instruction to achieve
    Kentuckys learning goals and academic
    expectations

40
Responses from 800 New Teachers in Kansas
41
Kentucky Scholastic Audit
  • What Distinguishes Successful Schools from
    Unsuccessful Schools
  • Intentional verbal communication about curriculum
  • Test scores used to identify curriculum gaps
  • Samples of student work used to inform
    instruction
  • Alignment of instruction and assessment with
    state learning goals

42
Teacher Work Samples Components
  • Teaching process standards
  • Teaching performance prompt
  • Scoring rubric
  • Twenty page narrative plus attachments teaching
    exhibit
  • lthttp//fp.uni.edu/itqgt

43
(No Transcript)
44
(No Transcript)
45
(No Transcript)
46
TWS Development at WKU
  • Fall 2000
  • 7 elementary ed candidates
  • Spring 2001
  • 17 elementary ed candidates
  • Fall 2001
  • 42 elementary ed candidates
  • Spring 2002
  • 115 elementary, 22 middle, 48 secondary ed, 30
    other candidates

47
Performing Scores on 134 TWS Exhibits at 8
Institutions
48
Performance Level of Teacher Candidates Based on
High Standards
  • Spring 2001 Fall 2002
  • n110 n134
  • Beginning (1) 16 18
  • Developing (2) 42 48
  • Proficient (3) 35 29
  • Expert (4) 07 05

49
Establishing Credibility Evidence for Teacher
Work Samples
  • Training for Scoring
  • Benchmarking
  • Reliability
  • Validity
  • Impacts on PK-12 Student Learning

50
Training for Scoring
  • Review Teaching Processes and Standards
  • Study TWS Prompt and Scoring Rubrics
  • Agree on Road Map of Evidence
  • Develop Indicator Definitions
  • Conduct Anti-Bias Training
  • Practice Scoring

51
Benchmarked Performances
  • Proto-typical performances at each level of the
    developmental scoring rubric.
  • 1 Beginning
  • 2 Developing
  • 3 Proficient
  • 4 Expert
  • Benchmarks are used for training of scorers and
    mentoring of candidates.

52
TWS Holistic Score Distribution January 2002
  • Beginning 18
  • Developing 48
  • Proficient 29
  • Expert 05
  • n 135

53
Inter-Rater Reliability Total Analytic
Scores
  • Extent to which scoring decisions are consistent
  • across expert judges using the analytic rubric.
  • Dependability Coefficients
  • (Shavelson Webb, 1991)
  • 5 raters .88
  • 3 raters .82
  • 1 rater .60

54
Inter-Rater Reliability Analytic
Sub-Scales
Sub-Scale
5 Raters
3 Raters
1 Rater
Contextual Factors
.86
.79
.55
Learning Goals
.85
.77
.53
Assessment Plan
.86
.78
.54
Design for Instruction
.86
.79
.55
Instructional Decision-Making
.78
.78
.42
Analysis of Student Learning
.82
.74
.48
Reflection and Self-Evaluation
.86
.79
.56
55
Inter-Rater Reliability
Trained raters are able to score with sufficient
inter-rater agreement to make high-stakes
decisions about performance levels. Moreover,
scores are sufficiently dependable that reliable
decisions can be made for each teaching process
separately.
56
Validity
Expert judgments of . . .
  • Content Representativeness (Crocker,
    1997)
  • - Realism
    - Criticality
    - Necessity
    - Frequency
  • Alignment with INTASC Principles

57
Validity Realism
98.1 of the raters strongly agreed with the
statement The performance tasks required by
the TWS represent actual lessons delivered to
students in PK-12 school classrooms.
58
Validity Criticality
97.5 of the expert judges rated the tasks
required by the TWS as critical or important
to teaching.
59
Validity Frequency
All of the tasks required by the TWS were rated
by the expert judges as high frequency activities
for teachers 90 of more of the raters said
weekly or daily for all of the tasks.
60
Validity Necessity
Overall, does the TWS assess knowledge and skills
that are necessary for beginning
teachers? Absolutely 70 Yes 30 No 0
61
Alignment with INTASC Principles
INTASC Principle
Directly
Implicitly
Not at All
5
45
50
Subject matter
50
5
45
Human development and learning
2
8
95
Adapting instruction
20
2
78
Instructional strategies
7
50
43
Classroom motivation and management
30
58
12
Communication skills
2
3
95
Instructional planning skills
0
7
93
Assessment of student learning
7
40
53
Professional commitment/responsibility
Partnerships
55
38
7
62
Alignment with INTASC Principles
INTASC Principle
Directly
Implicitly
Not at All
5
45
50
Subject matter
50
5
45
Human development and learning
2
8
95
Adapting instruction
20
2
78
Instructional strategies
7
50
43
Classroom motivation and management
30
58
12
Communication skills
2
3
95
Instructional planning skills
0
7
93
Assessment of student learning
7
40
53
Professional commitment/responsibility
Partnerships
55
38
7
63
Benefits of TWS
  • Focus on P-12 student learning
  • Focus on development of assessment tools
  • Focus on key standards
  • Alignment of standards, curriculum and assessments

64
Westerns Progress on Performance Assessments
  • Teacher Work Sample implemented
  • Most critical performances identified
  • Electronic portfolio operational
  • Course/experience alignment in process

65
Westerns Progress on Its Accountability System
  • Electronic data management system developed
  • Data sources and entry points identified
  • Baseline data entered
  • Research data entered
  • Research program initiated

66
WKUS VISION Next Steps
67
Enhancing the Electronic System Student Progress
Profiles
  • Informing students and faculty about progress on
    performances and other program requirements

68
Enhancing the Electronic System Course Profiles
  • Recording student completion and success on
    critical performances

69
Enhancing Critical Performances
  • Faculty development, discussion, and adoption of
    Alignment Maps
  • Filling gaps on Maps
  • Re-evaluating alignment and purpose of existing
    critical performances
  • Obtaining faculty feedback and student data to
    begin establishing reliability and validity

Alignment Map (click above to open)
70
Enhancing the Capacity of the Data Management
System
  • Using data to answer key questions
  • How well prepared is candidate X with respect to
    KY New Teacher Standards?
  • How well prepared is candidate X to facilitate
    learning of all students?
  • What are the candidates strengths and
    weaknesses?
  • What factors contribute to these strengths and
    weaknesses?
  • What are the programs strengths and weaknesses?
  • What aspects of the program contribute to these
    strengths and weaknesses?

71
(No Transcript)
72
(No Transcript)
73
(No Transcript)
74
(No Transcript)
75
(No Transcript)
76
(No Transcript)
77
(No Transcript)
78
Challenges to Developing An Accountability System
  • Obtaining faculty buy-in
  • Finding resources for ongoing system development
    and maintenance
  • Maintaining high standards while fostering
    creativity
  • Expanding use of valid and reliable assessment
    tools
  • Using context to design instruction
  • Making credible links between candidate
    performance and impact on P-12 learning

79
What Western and Renaissance Partnership
Institutions Have Learned About Essential
Elements for Developing Performance
Accountability Systems
  • Commitment to accountability and data management
  • Designated accountability system coordinator with
    released time
  • Development resources

80
What Western and Renaissance Partnership
Institutions Have Learned About Essential
Elements for Developing Performance
Accountability Systems
  • Development process involving a majority of
    faculty
  • Data sources
  • Data gathering points
  • Standards of performance
  • Indicators of performance
  • Scoring rubrics
  • Performance tasks (prompts)
  • Benchmarking of performances
  • Validation of systems
  • Scorer reliability

81
What Western and Renaissance Partnership
Institutions Have Learned About Essential
Elements for Developing Performance
Accountability Systems
  • Resources for scale-up and maintenance of system
  • Faculty
  • Training materials
  • Data management

82
Performance Assessment in KTIP
  • Standards of Performance Yes
  • Indicators of Performance Yes
  • Scoring Rubrics No
  • Performance Tasks (Prompts) Some
  • Benchmarked Performances No
  • Scorer Reliability - No

83
Persons to Contact About Westerns Accountability
System
  • Sam Evans
  • sam.evans_at_wku.edu
  • Tony Norman
  • antony.norman_at_wku.edu
  • Tabitha Toby Daniel
  • tabitha.daniel_at_wku.edu

84
(No Transcript)
85
(No Transcript)
86
(No Transcript)
87
Hands-on Teacher Work Samples - I
  • Form groups of three or four
  • Quick read TWS Exemplar
  • Use the rubric and scoring guide to rate process
    2 (goals), process 6 (analysis of student
    learning) and process 7 (reflection and
    evaluation) as
  • 1…beginning
  • 2…developing
  • 3…proficient
  • 4…expert
  • Share your ratings

88
Hands-on Teacher Work Samples - II
  • Identify three strengths of Teacher Work Samples
  • Identify three concerns or challenges you see in
    implementing Teacher Work Samples
About PowerShow.com