IES Evaluations and Data Collection Instruments

1
IES Evaluations and Data Collection Instruments

• Lauren Angelo
• National Center for Education Evaluation
• and
• Sally Atkins-Burnett
• Mathematica Policy Research (MPR)

Lauren.angelo_at_ed.gov
SAtkins-Burnett_at_mathematica-mpr.com
2
Todays Presentation

• NCEE Studies that Involve Mathematics
• Developing Criteria for Choosing an Instrument
• Assessing Teacher Knowledge in the NCEE Math PD
  Study
• Assessing Student Achievement in the NCEE Math PD
  Study
• Assessing Classroom Practice in the NCEE Math
  Curricula Study

3
NCEE Studies that Include Mathematics
Impact Evaluation of a Professional Development
Model in Math Impact Evaluation of Math
Curricula Impact Evaluation of Education
Technology Impact Evaluation of Academic
Instruction for After-School Programs Program
Selection and Design for an Evaluation of the
Impact of Academically Focused Career and
Technical Education (CTE) Evaluation of Promising
Boost Up Math Programs
http//ies.ed.gov/ncee/
4
The Impact of a Professional Development Model in
Math
Research Question What is the impact on teacher
knowledge, teacher practice, and student
achievement of providing 7th grade teachers with
intensive professional development that focuses
on common student misconceptions in the areas of
fractions, decimals, ratio, and
proportion? Intervention A professional
development intervention consisting of a
three-day summer institute, five one-day seminars
held during the school year, and 10 days of
intensive in-school coaching. Targeted approach
focused on foundational math concepts and
strategies for improving instruction. Design
School-level randomized trial, 16 districts (8
schools per district), approximately 200 7th
grade teachers Schedule August 2005-2010
Currently preparing for implementation and data
collection
5
The Impact of a Professional Development Model in
Math
Content Knowledge Pedagogical Content Knowledge
Student Achievement
Math PD
Instruction
6
NCEE Math PD Teacher Knowledge Test
Purpose Assess teacher knowledge of math content
and pedagogy

• NCEE Math PD Teacher Knowledge Test Criteria
• Adequate and even rational number coverage
• Math Common Content Knowledge (CCK)
• Pedagogical Content Knowledge (PCK)
• Sensitive to growth for our population of
  teachers
• 30- to 40-minute administration
• 3 Parallel forms

• General Instrument Criteria
• Content Coverage
• Adequate Psychometrics
• Reliability and Validity
• Test Properties
• Administration Considerations

7
NCEE Math PD Teacher Knowledge Test

• Item Development and Review
• Fit the framework
• Accurate content
• Appropriate for population
• Standards of good items (one correct answer,
  appropriate and plausible distracters)
• Piloting
• Item difficulty
• Biserial correlations
• Distracter analysis

8
NCEE Math PD Teacher Knowledge Test Framework

Pedagogical Content Knowledge (PCK)
Planning Instruction
Delivering Instruction
Assessing Understanding
Current Status Piloting Data Collection To be
Completed Spring 2009
9
NCEE Math PD Student Achievement Test
Purpose Assess student math achievement in
rational number content that could assess growth

• NCEE MATH PD Student Achievement Test Criteria
• Ample coverage of rational number content across
  various difficulty levels for students in study
• Sensitive to growth
• Multiple forms, machine scoring, 40 minutes or
  less

• General Instrument Criteria
• Content Coverage
• Adequate Psychometrics
• Reliability and Validity
• Test Properties
• Administration Considerations

10
NCEE Math PD Study Student Achievement Test

• Chose to Use Northwest Evaluation Association
  (NWEA) Measures of Academic Progress in
  Mathematics
• Item bank
• Rational number content
• Adaptive test
• Computer administered
• Customization/Development
• Customization to specific parameters
• Items supplemented to ensure coverage across
  difficulty levels for each content area

Current Status Inclusion of new items Data
Collection To begin this fall
11
Evaluation of Math Curricula
Research Question What is the relative
effectiveness of different math curricula on
student achievement in early elementary school?
Under what condition is each curriculum most
effective? What is the relationship between
teacher knowledge of math content and pedagogy
and the effectiveness of math curricula? Intervent
ion 4 math curricula for elementary school
students (grade 1 and 2) Design Randomized
trial, schools are randomly assigned to one of
the four curricula (Investigations in Number,
Data, and Space (PSF) Math Expressions (HM)
Saxon Math (HA) Scott Foresman-Addison Welsey
(PSF)) Schedule September 2005-2009 with an
optional year Curricula currently being
implemented (2006-2007) in first-grade
classrooms next year curricula will be
implemented in both first and second grades.