Program Review Process

1
Program Review Process

• Steve Fiechtner
• SD Department of Education
• Steve.fiechtner_at_state.sd.us

2
SD Department of Education

• Accreditation process has two primary components
• Unit review
• Program Review

3
A brief review. . .

• Unit School, College, or Department of
  Education or other entities that prepare
  personnel to work in schools
• Program Specific Discipline Area

4
(No Transcript)
5

• SPA Specialized Professional Association

6
Purpose of the Program Review

• Determine whether or not the program has in place
  a limited number (6-8) of comprehensive
  assessments that demonstrate candidate mastery of
  the SPA/State standards.
• Candidate performance on these assessments is
  appropriate to demonstrate mastery.
• Provide information for unit to use to respond to
  Unit Standard 1 and SD Administrative Rule
  2453.

7

• Candidates preparing to work in schools as
  teachers or other school personnel know the
  content of their fields, demonstrate professional
  and pedagogical knowledge, skills, and
  dispositions and apply them so that students
  learn. Assessments indicate that candidates meet
  professional, state, and institutional standards.

8
6-8 Assessments The rules

• Institution must submit a minimum of six
  assessments, unless the SPA/State specifies more
  than six required assessments
• Institution may submit additional assessments
  when SPA/State does not specify all eight
  assessments
• Five specific types of assessments are required.

9
Required Assessments

• State licensure exam for program area (if
  availableotherwise another content based
  assessment)
• Content Assessment
• Assessment of Planning (e.g., unit plan)
• Student teaching/internship assessment
• Assessment of candidate impact on student
  learning or providing a supporting learning
  environment

10
Components of the new program report

• Cover sheet (online)
• Section I Contextual information
• Section II Assessment Chart
• Section III Standards Chart
• Section IV Evidence of Meeting Standards
  (Assessments Findings)
• Section V Use of Assessment Results to Improve
  Performance

11
Section I Context

• Provide descriptions of
• State or institutional policies that may
  influence application of standards
• Clinical experiences
• Criteria for admission, retention, and completion
  of the program
• Relationship of program to units conceptual
  framework
• Relationship of program assessments to the units
  assessment system

12
Section I Context (continued)

• Attach
• Program of study
• Chart of candidates enrolled and completing the
  program over past three years (Attachment A)
• Chart on faculty expertise and experience
  (Attachment B)

13
Section I

• Do not include more than is asked for
• Do not include hyperlinks
• Include what is USEFUL to the reviewers-reference
  in body of report
• Make sure contextual info is consistent with the
  rest of the report

14
Section II Assessment Chart

• List the 6-8 assessments that will document
  evidence of meeting standards.
• For each assessment, indicate name, type, and
  administration point in the program

15
Section II Chart

• Give specific names to assessments (Jr. Year
  Content Portfolio Review)
• Make sure assessments and administration points
  correlate to Program of Study in Section I
• For the five required assessments, DO NOT change
  order or make substitutions

16
The five required assessments - 1

• Assessment 1 - Licensure data
• Although the assessment instrument is not
  required, providing alignment information is
  useful
• In findings, describe state requirements for
  licensure tests, as well as the s in your data
  sets

17
The five required assessments - 2

• Assessment 2 - Another content-based
  assessment
• Grades
• Comprehensive exam
• Content-based portfolio assessment
• Case studies action research

18
The five required assessments - 3

• Assessment 3
• Assessment of ability to plan instruction
• (classroom-based programs)
• e.g., unit or lesson plan assignment
• Assessment of ability to fulfill specified
• professional roles (advanced programs)
• e.g., needs assessment project

19
The five required assessments - 4

• Assessment 4
• Assessment of effective clinical practice
• Student teacher/internship/practicum evaluation

20
The five required assessments - 5

• Assessment 5
• Assessment of candidate impact on student
  learning
• Teacher work sample or other classroom-based
  project
• Induction year evaluation or portfolio
• Employer or other external surveys

21
Section III Standards Assessment Chart

• Show (in final column) which of your 6-8
  assessments provide evidence of meeting each
  standard.
• In most cases, more than one assessment will
  provide evidence for meeting a single standard.
• Limit your evidence to a maximum of 3
  assessments.
• All standards should be covered by AT LEAST one
  assessment.

22
Using Section II

• Section II
• What do we have in place that will satisfy the
  five required assessments?
• Can we use what we have, or do we need something
  new?
• What other assessments do we use that show
  candidates meet standards?

23
Using Section III

• Section III
• Do the assessments youve listed in Section II
  cover all standards? Is the alignment specific
  enough to provide solid evidence?
• Is evidence for some standards weaker than it is
  for others?

24
Section IV Assessments and Findings

• This is the ?of your program report
• Section IV will now include, for each of your
• 6-8 assessments
• Assessment instrument
• Scoring guide
• Data table(s)
• Findings (2 page narrative)

25
What is an assessment?

• For the purposes of the program report, an
  assessment is an evaluated activity or
  requirement by which a program determines that
  specific outcomes or standards have been mastered
  by a candidate.
• The assessment package consists of the
  assessment instrument, scoring guide, and
  candidate data derived from the assessment.

26
What isnt an assessment

• A syllabus (although it might be used as an
  assessment instrument for a course grade)
• A list of program objectives or expected
  candidate outcomes
• State or professional standards by which a
  program is designed/assessed

27
What might the assessment instrument look like?

• Examples
• A sample of a comprehensive test
• An evaluation form (student teaching, survey)
• A course outline or outlines (if grades are used
  as the assessment)
• A description of the assignment (unit plan,
  teacher work sample)

28
What about the scoring guide?

• The scoring guide should
• Include explicit statements of proficiencies
  candidates are expected to demonstrate in their
  responses.
• Be constructed so that different levels of
  candidate proficiency are clearly distinguished.

29
Scoring Guides

• The assessment/data are only as good as the
  scoring guide.
• If possible, the scoring guide should be specific
  to the discipline and aligned to the standards.
• Scoring guide should be correlated to and
  appropriate for the assessment.

30
Presenting Your Data

• Data tables should be self-contained documents
  readable and user-friendly
• Shown the parts of the sum, not (just) the sum of
  the parts!
• Data should be presented by categories used in
  the scoring guides

31
Characteristics of Well-Presented Data
Specify academic year or cohort group, e.g.
2007-2009 candidates admitted spring 2007
Note Provide legends for abbreviations,
acronyms, etc.
32
Disaggregate Data for

• Different levels of a program (e.g. baccalaureate
  and Initial Masters)
• Different sites at which a complete program is
  offered
• Different cohorts or grading periods

33
How Much Data?

• Three years of data are optimal not necessary
  to provide more
• Remember Data are only as good as the
  assessment/scoring guide

34
Assessment Findings

• For each of the 6-8 assessments, provide a
  narrative including
• A brief description of the assessment and its use
  in the program
• The alignment of the assessment with specific
  SPA/State standards it addresses
• A brief summary of the data findings and
• An interpretation of how findings provide
  evidence of meeting standards.

35
Section V - Use of Assessment Results to Improve
Program

• Describe findings based on aggregated data, and
  discuss how findings have been or will be
  applied to program improvement.
• Organize discussion around content knowledge
  professional and pedagogical knowledge and
  skills, and impact on student learning.
•  

36
Section V

• Focus your discussion on the assessments and
  their findings
• Describe a systematic approach to data review
  toward program improvement
• Discuss plans or action taken to address downward
  trends or poor candidate performance

37
All Parts of Assessments Must Interrelate!
STANDARDS
38
What does that mean?

• Parts of the assessment are aligned with the
  SPA/State standards
• These standards can be seen in the elements of
  the scoring guide
• The data is broken down by the elements in the
  scoring guide

39
An example

• NSTA Standard 2teachers of science must
  demonstrate that they (a) understand the
  historical and cultural development of science
  and the evolution of knowledge in their
  discipline (b) understand the philosophical
  tenets, assumptions, goals, and values that
  distinguish science from technology and from
  other ways of knowing the world .

40
Part of the assessment A Science Portfolio Module

• By the time you have completed this course you
  will have written a series of essays on science
  and how your instructional strategies incorporate
  your understanding of science and its role in
  society. These essays constitute an important
  component of your teaching portfolio. You will
  be assessed on your understanding of scientific
  concepts, the nature of science, and the
  applications of science in society.
• The specific essay assignments include essays on
  The nature of science, the skills of science,
  science and knowledge, values in science,
  applications of science, science and language,
  science and inquiry, collaboration in science,
  science and technology, science in the community.
  Each essay accompanies a teaching assignment, in
  most cases a lesson plan.

41
One element from the rubric
Findings
42
Data Chartn 25 candidates, Fall 2007 semester
43
What Criteria Are Used to Evaluate Assessments?

• Assessments are aligned to standards
• Assessments reflect the rigor of standards
• Assessments are free from bias
• Scoring guides clearly identify levels of
  proficiency
• Data provide evidence of candidate mastery of
  standards

44
Roles of Reviewers

• Judge alignment of assessment and candidate data
  with SPA standards
• Clearly communicate strengths and weaknesses in
  relation to the standards
• Make a judgment with a clear and open mind
• Make a judgment based on accepted criteria rather
  than personal bias

45

• The job of the reviewer is not to pass or fail
  programs, but to make as objective an assessment
  as possible about the degree to which a given
  program meets the SPA/State standards.

46
Reviewers have ethical obligations to be

• Objective
• Reflective
• Conscientious
• Discrete

47
Reviewers should avoid

• Discussing program review results with those
  outside of the review system.
• Revealing deliberations or personal doubts about
  review results.
• Suggesting specific changes to institutional
  programs.
• Using subjective or opinionated language.
• Writing comments that are jocular, humorous,
  flippant, comparative or otherwise abusive.

48
Program Review Process?

• Will be managed by South Dakota DOE
• South Dakota DOE will set timelines, deadlines,
  etc

49
Program Review Process

• Institutions submit reports electronically 6 to
  12 months prior to site visit. (This may be
  different for NCATE institutions.)
• SD DOE notifies reviewers of their assignment to
  review the report
• Each reviewer completes a review of the program
• Team members read each others reports and
  discuss differences

50

• Lead reviewer compiles final report and submits
  the report to DOE
• Final reviewer report by SD DOE consultant
• SD DOE posts report to institution
• BOE team accesses report for on-site visit

51
Resources

• Program Approval Handbook
• Recognition reports
• Sample reports

52
Making Decisions

• Holistic decision made for each standard
• Final decision made in collaboration with SD DOE
  consultant

53
Program review decisions

• SPA/State reviewers will make one of the
  following decisions based on the program report
• The program is nationally recognized. (If you
  are an NCATE institution submitting reports
  nationally.)
• A program will be SD DOE recognized or not
  recognized.
• Institutions have an opportunity to submit a
  rejoinder for programs not recognized.

54
Rejoined Reports

• Will be sent to original team if possible
• Process is similar to original review but is
  limited to those items not met in original
  review
• Reviewers will have access to original program
  report and original recognition report

55
Who to Contact at SD DOE

• For questions related to overall process,
  submission requirements, etc.
• Steve Fiechtner
• SD Department of Education
• Steve.fiechtner_at_state.sd.us
• 605-773-4774

56
Who to Contact at SPAs

• For questions related to the standards and
  what are appropriate assessments for a particular
  discipline
• SPA Program Review Coordinators
  http//www.ncate.org/programreview/spacontact.asp?
  ch88