MSP Program Evaluation

1
MSP Program Evaluation

• Carol L. Fletcher, Ph.D.
• TRC Project Director Meetings
• 1/27/09 and 2/5/09

2
TRC Criteria

• How has your Regional Collaborative impacted
• Teacher Content Knowledge
• Classroom Practice
• Student Achievement

3
Guskey Model

• Based on mediators of relationship between
  professional development and improvements in
  student learning
• i.e. administrator knowledge and practice, school
  culture, teacher knowledge and practice, parent
  knowledge and practice

4
Five Critical Levels of PD Evaluation

• Inform planning, formative and summative
  evaluations
• Helps to identify the what and the why
  evaluation
• Hierarchically arranged from simple to complex
• Success at one level is necessary for success at
  following levels

5
Level 1 Participants Reactions
What questions are addressed?
How will information be gathered?

• Did they like it?
• Will it be useful
• Was the leader knowledgeable and helpful?
• Where the facilities comfortable?

• Questionnaires
• Interviews
• Participant journals

6
Level 2 Participants Learning
What questions are addressed?
How will information be gathered?

• Did participants acquire the targeted knowledge
  and skills?

• Questionnaires
• Interviews
• Participant journals/portfolios
• Demonstrations

7
Level 3 Organization Support and Change
What questions are addressed?
How will information be gathered?

• Did it affect organizational climate, procedures,
  or policies
• Were sufficient resources made available?
• Were successes recognized shared?

• Questionnaires
• Structured Interviews
• Participant journals/portfolios
• Focus groups
• District/school records

8
Level 4 Participants use of new knowledge and
skills
What questions are addressed?
How will information be gathered?

• Did participants effectively apply the new
  knowledge and skills?

• Questionnaires
• Structured Interviews
• Participant journals/portfolios/ reflections
• Direct observation
• Videotapes

9
Level 5 Student learning outcomes
What questions are addressed?
How will information be gathered?

• Did it affect student performance or achievement?
• Are students more engaged learners?
• Do students select more rigorous coursework?
• Did it affect student attitudes toward science or
  math courses/careers?

• Questionnaires
• School/student records
• Structured Interviews
• Participant journals/portfolios/ reflections
• Direct observation
• Videotapes

10
NSF Developed Tools

• Horizon Research
• Assessing Teacher Learning About Science Teaching
  (ATLAST)
• www.horizon-research.com/atlast/
• Instruments to measure student and teacher
  science content knowledge
• Force and Motion
• Flow of Matter and Energy in Living Systems
• Plate Tectonics

11
ATLAST continued

• Student tests - paper copies only
• Teacher tests - paper or online versions
• The student and teachers assessments measure
  knowledge of the science content, while the
  teacher assessment also measures a teacher's
  ability to use science content knowledge to
  analyze student thinking, and to make
  instructional decisions. All of the items on the
  teacher assessment are set in instructional
  contexts.

12
ATLAST continued

• HRI provides the following analyses
• Percent of respondents choosing each answer
  choice
• Percent of respondents answering each item
  correctly
• Group mean score and
• Significance testing and computation of effect
  sizes for pre- and post-test group mean scores.

13
(No Transcript)
14
MOSART
www.cfa.harvard.edu/smgphp/mosart

• Misconception Oriented Standards-based Assessment
  Resource for Teachers (MOSART)
• Harvard PI - Phillip Sadler
• Content instrument for K-12 physical science and
  earth science, based on research about student
  misconceptions
• Completion of online tutorial (30 min) required
  to download instruments

15
(No Transcript)
16
MOSART continued

• K-4 5-8 Physical Science, Earth Science,
  Astronomy/Space Science
• 9-12 Physics, Chemistry, Earth Science,
  Astronomy/Space Science
• Multiple versions of the test are available for
  pre/post administration

17
MSPnet Toolbox

• http//hub.mspnet.org/index.cfm/msp_tools
• Clearinghouse for tools used and shared by MSP
  projects
• Tools may include assessment instruments,
  evaluation protocols, form letters, etc.

18
(No Transcript)
19
University of Louisville
http//louisville.edu/education/research/centers/c
rmstd/diag_sci_assess_middle_teachers.html

• Diagnostic Assessments for Middle School Teachers
• Each assessment is composed of 25 items20
  multiple-choice and 5 open-response.
• Six versions of each assessment are available in
  paper-and-pencil format

20
University of Louisville (cont.)

• Assessments available free of charge
• Optional scoring service for 10/teacher
• Detailed summary of teachers' performance that
  includes scores on individual items, on each
  science subdomain in the content area, and on
  four different knowledge types (memorized,
  conceptual understanding, higher-order thinking,
  pedagogical content knowledge)

21
University of Louisville (cont.)
Physical Science Life Science Earth/Space Science
Matter Structure/Function Atmosphere/Hydrosphere
Motion Forces Internal Regulation Lithosphere
Energy Heredity/Diversity Space
Interdependence
22
MSP Knowledge Management and Dissemination
http//www.mspkmd.net/

• Knowledge reviews good for citing research base
• Searchable Instrument database for science and
  math

23
(No Transcript)
24
Online Evaluation Resource Library (OERL)
http//oerl.sri.com/

• Teacher and student instruments
• Content assessments, interviews,
  surveys,classroom observations

25
(No Transcript)
26
Thinking About Mathematics Instruction (TMI)

• http//www2.edc.org/tmi/tmi_survey.html
• Survey to investigate elementary and middle
  school principals leadership content knowledge
  (LCK) for mathematics

27
(No Transcript)
28
Distributed Leadership

• http//www.sesp.northwestern.edu/dls/instruments/
• Instructional Leadership Daily Practices Log
• Principal Experience Sampling Method (ESM) Log
• School Staff Network Survey

29
(No Transcript)