Update on the Program Review Process: What Worked and What Didnt

1
Update on the Program Review Process What
Worked and What Didnt

• NCATE Clinic
• Margaret Crutchfield
• May 2006

2
Principles for the Reform of the Program Review
Process

• Continue to rely on expertise of SPAs
• Reduce costs and burdens for SPAs and
  institutions
• Streamline process
• Consistency across SPAs
• Closer alignment to NCATE Standard 1

3
Continue to rely on expertise of SPAs?

• SPAs develop standards
• Select and co-train reviewers
• Maintain audit teams
• Collaborate to develop consistent policies,
  forms, etc

4
Have we reduced costs and burdens for SPAs and
institutions?

• SPAs
• Reduced staff time commitments but not yet to the
  expected level
• Will be reducing financial costs
• Have reduced some of the logistical burden of
  dealing with reports, etc
• SPAs still provide significant staff and
  volunteer time to make this process work
• Institutions
• Consistency in the process has been a great
  improvement
• But making the required changes in assessments
  has taken time

5
Have we streamlined the process?

• Yes, for institutions
• Submit on-line
• Identical forms etc have reduced confusion
• No, for SPA reviewers and audit teams
• Technology has not been our friend
• The audit review has been added to the process

6
Is the process more consistent across SPAs?

• Consistent assessments
• 5 required
• State licensure
• Content
• Planning
• Clinical Practice
• Student Learning
• Consistent program templates
• Some SPA-specific requirements

7

• Consistent policies for entire program review
  system
• Consistent decision criteria
• Still some inconsistency among reviewer
  decision-making

8
Provide closer alignment to NCATE Standard 1?

• Candidates preparing to work in schools as
  teachers or other school personnel know the
  content of their fields, demonstrate professional
  and pedagogical knowledge, skills, and
  dispositions and apply them so that students
  learn. Assessments indicate that candidates meet
  professional, state, and institutional standards.

9
Required Assessments

• Content Knowledge
• State Test
• Another Content Based Assessment
• Professional and Pedagogical Knowledge, Skills,
  and Dispositions
• Planning
• Clinical Practice
• Candidate Impact on Student Learning
• Student Learning

10
Principles for the Reform of the Program Review
Process
11
Issues Addressed This Past Year
12
What are the numbers?
13
Challenges

• Maintaining deadlines
• Having sufficient reviewers
• Maintaining consistent reviewer decision-making
• Approval rates

14
Approval Rates

• Pilot rates very low (31 and 22). In response,
• SPAs expanded criteria for nationally recognized
  with conditions
• NCATE and SPAs provided more resources and
  guidance for programs especially concerning
  alignment issues
• NCATE provided specific guidance on how to use
  unit-wide assessments to demonstrate candidate
  mastery of SPA standards

15
What was the result for initial reports?
16
Approval rates for revised reports?

• 62 revised reports were received through Fall 05
• 60 received a positive decision

17
Approval Rates by SPA
18
Consistency in training reviewers and reviewer
decision-making

• Great variety in content of SPA reviewer training
  and in the amount of time SPAs devote to reviewer
  training
• A larger issue is ongoing training and updating
  of reviewers

19
What NCATE is doing

• Asking SPAs to help develop criteria for training
• Developing on-line training modules for reviewers
• Improving NCATE Guidelines for Reviewers
  including specific examples, models, and
  problems
• Implementing a newsletter for reviewers to keep
  them up-to-date and connected
• First issue sent out in 3/06

20
What are the SPAs doing?

• Several have developed training materials
• Several have increased resources on their web
  site
• Many have helped NCATE develop materials for
  NCATEs web site

21
Ongoing challenges

• Continue to increase uniformity of system where
  appropriate
• Continue to support and train reviewers to
  improve consistency of reviewer decision-making
• Continue to support and provide guidance to
  institutions

22

• Deal with technical issues until new database is
  ready
• Making substantial progress in developing new
  reviewer interface, pilot in Spring 07, implement
  fully in Fall 07

23
One More Important Lesson

• It is possible for programs to demonstrate that
  their candidates have mastered the SPA program
  standards using a limited number of comprehensive
  assessments.