Don Boyd, Pam Grossman, Karen Hammerness, Hamp Lankford, Susanna Loeb, Matt Ronfeldt

1
Recruiting Effective Math Teachers, How Do Math
Immersion Teachers Compare? Evidence from New
York City

• Don Boyd, Pam Grossman, Karen Hammerness, Hamp
  Lankford, Susanna Loeb, Matt Ronfeldt Jim
  Wyckoff
• www.teacherpolicyresearch.org
• This work is supported by IES Grant R305E6025.
  The views expressed may not reflect those of the
  funder.

2
New Math Certified Teachers Hired in New York
City, by Pathway, 2002-2008
3
Research Questions

• How does the preparation of Math Immersion
  teachers compare to math teachers entering
  through other pathways?
• How do the achievement gains of the students
  taught by Math Immersion teachers compare to
  those of students taught by math teachers
  entering through other pathways?
• How does the retention of Math Immersion
  candidates compare to math teachers entering
  through other pathways?

4
Prior Research

• Achievement effects of alternate route teachers
  comparable to traditional preparation programs on
  average (Decker et al., 2004 (RCT) Boyd et al.,
  2006 Kane et al., 2007 Harris and Sass, 2008
  Constantine et al., 2009 (RCT))
• TFA in NC high schools exceeds other paths (Xu et
  al., 2007)
• More limited work on aspects of preparation that
  may make a difference (Constantine et al., 2009
  Boyd et al., 2009 and Harris and Sass, 2007)

5
Data Collection

• Program analysis
• State documents, program documents, accreditation
  reports, interviews, surveys, course syllabi
• 5 Math Immersion programs,18 institutions, and
  TFA that prepare most traditional route teachers
  for NYC schools
• Surveys
• 603 new NYC middle and high school math teachers
  (2005)
• Questions about their preparation in math e.g,
  opportunities to learn math content, math
  methods, etc.
• Administrative data
• All NYC teachers 2004-2008 rich measures of
  teacher qualifications, including certification
  exams and areas, teacher retention.
• Student achievement 2004-2008 value-added scores
  in math and ELA, grades 6-8 linked to teachers.
• Data on schools and students

6
Effect of Preparation Pathways

• General specification
• Aigcstb0b1Aig-1cst-1Xigcstb2Cgcstb3Tgcstb4
  Pgcstb5 ws eigcst
• Achievement as a function of
• prior achievement,
• student characteristics
• classroom characteristics
• teacher characteristics (sometimes)
• Preparation pathway (e.g., math immersion)
• Student or school fixed-effects
• random error

7
Attributes of Students Taught by First Year Grade
8 Math Teachers by Pathway, 2006
Student Attributes CR NYCTF NYCTF-MI TFA other
Lagged Math Achievement 0.238 -0.125 -0.051 -0.139 -0.061

Proportion Black 0.292 0.277 0.322 0.442 0.403
Proportion Hispanic 0.358 0.496 0.493 0.527 0.372
Proportion Free Lunch 0.547 0.664 0.635 0.619 0.66
Classsize 27.6 27.8 26.9 26.3 26.1
Lagged Student Absences 12.3 13.4 13.1 14.8 13.5

Lagged Suspensions 0.037 0.064 0.062 0.023 0.042
8
Attributes of Entering Math Certified New York
City Teachers by Pathway, 2004-2008
  CR CR NYCTF-MI NYCTF-MI TFA TFA
Teacher Attributes High School Middle School High School Middle School High School Middle School
Female 0.648 0.732 0.479 0.546 0.492 0.551
Black 0.073 0.105 0.142 0.200 0.082 0.141
Hispanic 0.065 0.046 0.085 0.074 0.066 0.043
Age 29.7 28.9 31.1 30 23.6 23.5

Last Score 255 251 274 271 279 279
CST Math Score 262 251 257 251 268 269
SAT Math 600 556 616 589 710 648
SAT Verbal 506 483 577 564 627 623
             
Number of Teachers 478 157 1098 542 61 98
9
Effect of Preparation Paths Relative to NYCTF-MI
  1 3 5 7 9
Pathways Level Level Level Level Level
College Recommend 0.016 0.005 0.017 0.004 0.006
  1.86 0.47 2.60 0.40 0.55

NYC Teaching Fellows 0.021 0.022 0.023 0.015 0.012
  1.87 1.68 2.74 1.38 0.85

Teacher for America 0.055 0.018 0.068 0.032 0.046
  3.71 0.86 5.74 1.88 2.77

Other -0.011 -0.003 -0.004 0.002 -0.02
  1.27 0.28 0.66 0.27 -1.74
NYCTF-MI Below   -0.044
    -1.52
NYCTF-MI NA   -0.014
    -1.04
Teacher controls   ü ü  
School fixed effects ü ü ü
Student fixed effects     ü ü  
10
Distribution of Teacher Value Added by Pathway,
with Empirical Bayes Shrinkage, 2004-2008
Teachers
11
Effect of Pathways and Experience Relative to
Math Immersion of Same Experience, Grades 6-8,
2004-08
  No Teacher Controls No Teacher Controls No Teacher Controls No Teacher Controls
  Experience Experience Experience Experience
Pathway 1 2 3 4
College Recommending 0.018 0.024 0.010 0.028
  1.60 1.90 0.65 1.53

NYCTF 0.011 0.010 0.005 0.065
  0.74 0.58 0.24 2.76

TFA 0.054 0.056 0.041 0.048
  3.13 2.64 1.09 1.29

Other -0.028 -0.032 -0.018 0.009
  2.22 2.61 1.29 0.50
Same variables as model 1 above.
12
Effect of Pathways and Math Immersion Programs
2004-08, Relative to NYCTF-MI Program Z
Pathway and Program Level Level
College Recommend 0.057 0.033
  3.94 1.89
NYC Teaching Fellows 0.062 0.047
  3.81 2.54
Teacher for America 0.096 0.031
  4.96 1.21
Other 0.030 0.027
  2.12 1.55
Institution A 0.034 0.018
  1.50 0.71
Institution B 0.051 0.029
  2.66 1.28
Institution C 0.048 0.035
  1.71 1.16
Institution D 0.055 0.037
  2.99 1.72
Teacher controls ü
School fixed effects ü ü
Same variables as earlier model specification.
13
Teacher Retention by Pathway, Math Certified
Teachers, 2004-2008
  NYCTF-MI NYCTF-MI   CR CR
Experience Transfer Leave   Transfer Leave
1 12.2 12.4 9.6 13.4
2 18.7 26.5 12.3 19.1
3 23.6 36.4 16.0 27.7
4 26.5 42.1   18.0 31.4

  NYCTF NYCTF   TFA TFA
Experience Transfer Leave   Transfer Leave
1 8.9 15.7 5.0 8.2
2 16.2 29.6 9.9 58.8
3 19.2 42.3 12.1 75.6
4 24.4 47.5   13.2 78.7
14
Simulation of Average Value Added by Pathway and
Experience Accounting for Attrition
Simulation Average Value Added Average Value Added Average Value Added Average Value Added
Year NYCTF-MI CR NYCTF TFA
1 0.000 0.018 0.011 0.054
2 0.045 0.068 0.053 0.103
3 0.066 0.086 0.072 0.086
4 0.052 0.081 0.088 0.088
         
  Value Added by Pathway and Experience Value Added by Pathway and Experience Value Added by Pathway and Experience Value Added by Pathway and Experience
Experience NYCTF-MI CR NYCTF TFA
1st year 0.000 0.018 0.011 0.054
2nd year 0.051 0.075 0.061 0.107
3rd year 0.085 0.095 0.090 0.126
4th year 0.063 0.091 0.128 0.111
15
Conclusions

• MI teachers have about the same value-added as
  College Recommended teachers
• Driven largely by selection, TFA performs much
  better than either College Rec or Math Immersion
• Some evidence that both selection and preparation
  make a difference
• Hypothesis selective post BA program with
  tailored coursework that includes content and
  high quality field experience can meaningfully
  improve student achievement

16

• For papers and surveys
• www.teacherpolicyresearch.org

17
Pathways to Teaching in NYC, New Teachers, 2002-08
18
Outline

• Research questions
• Data and methods
• Math preparation in Math Immersion and College
  Recommending programs
• Achievement gains by pathway
• Retention by pathway
• Summing up

19
The Teacher Workforce and Student Outcomes
20
Attributes of Entering Math Certified NYCTF-MI
Teachers by Preparing Campus, 2004-2008
  Campus D Campus D Campus Z Campus Z
Teacher Attributes High School Middle School High School Middle School
Female 0.468 0.534 0.509 0.538
Black 0.108 0.165 0.125 0.127
Hispanic 0.088 0.046 0.069 0.093
Age 31.7 31.9 29.9 29.7

Last Score 272 268 277 276
CST Math Score 255 249 257 257
SAT Math 618 586 625 616
SAT Verbal 573 542 589 586
     
Number of Teachers 154 116 322 119
21
Required Credit Hours for Key Courses by Pathway,
2004-2008
College Recommending Math Courses Math Methods Classroom Manage. Learning
Graduate programs        
Mean 4.93 5.79 0.86 3.75
Standard deviation 5.34 3.29 1.83 2.16
Undergrad prog.  
Mean 11.00 4.71 1.75 4.50
Standard deviation 11.29 1.38 2.26 2.70
   
Math Immersion Math Courses Math Methods Classroom Manage. Learning
Mean 12.60 8.40 0.60 2.40
Standard deviation 5.77 2.51 1.34 1.34
22
Survey of 1st year NYC TeachersMiddle and High
School Math

• In your preparation to become a teacher, prior
  to September 2004, how much opportunity did you
  have to the following
• learn different ways that students solve
  particular problems
• learn theoretical concepts underlying
  mathematical applications
• explore how to apply mathematical materials to
  real world problems
• learn specific techniques for teaching Algebra
  (Geometry, Number Theory, Probability and
  Statistics, Calculus)
• learn about typical difficulties students have
  with Algebra (Geometry, Calculus)
• study or analyze student math work
• study examples o secondary mathematics teaching
  in the form of videotapes, written cases, etc.
• Practice what you learned about teaching math in
  your field experience
• etc.

23
Teachers' Perceptions of Preparation by Pathways
Relative to NYCTF-MI, (2005 Survey of 1st Year
Teachers)
Pathway Preparation in Specific Strategies General Opps to Learn Teaching Math Subject Matter Prep in Math
College Recommending 0.331 0.386 0.038
  2.99 3.54 0.33
Teaching Fellows 0.274 -0.350 -0.462
  2.50 -3.32 -4.12
Teach For America 0.604 -0.007 -0.561
  2.74 -0.03 -2.48
Other Path 0.004 0.371 0.320
  0.04 3.31 2.74
 
N 558 543 541
24
Estimated Value Added Model
Student Measures   Class Average Measures Class Average Measures Experience  
Lag score 0.593 Hispanic -0.161 2nd year 0.050
  269.33   6.81 8.92
Lag score sqrd -0.005 Black -0.152 3rd year 0.082
  3.70   6.11 12.70
Female 0.010 Asian 0.099 4th year 0.091
  6.58   3.71 12.22
Asian 0.126 Class size 0.000 5th year 0.100
  35.45   0.85 12.64
Hispanic -0.059 English home -0.026 6th year 0.096
  19.07   1.48 11.01
Black -0.060 Free lunch 0.014  
  18.21   1.57 Pathways  
Change school -0.078 Lagged absent -0.007 Coll. Recomm. 0.016
  16.22   13.30 1.86
English home -0.060 Lag suspended -0.002 NYCTF 0.021
  31.51   0.15 1.87
Free Lunch -0.017 Lag ELA score 0.194 TFA 0.055
  10.46   24.73 3.71
Lagged absent -0.005 Lag Math score 0.076 Other -0.011
  64.92   9.16   1.27
Lag suspended -0.024 Std Dev ELA score 0.043  
  12.20   4.78 N 651191
Also includes student and class ELL status, std
dev class math score, indicators for experience
through 21 years, year and grade effects
25
Challenges of this type of analysis

• Conceptualizing relationships
• Research designs
• Collecting appropriate data
• Achievement tests, tested grades, subjects
• Strong controls from administrative data
• Other data about teachers
• Legal/political
• Privacy
• Concerns about misuse
• Technical/modeling
• Models that isolate contribution of teacher
  attributes

26
Effect of Preparation Paths Relative to NYCTF-MI
  1 2 3 4
Pathways Level Level Level Level
College Recommend 0.016 0.017 0.005 0.004
  1.86 2.60 0.47 0.40
NYC Teaching Fellows 0.021 0.023 0.022 0.015
  1.87 2.74 1.68 1.38
Teacher for America 0.055 0.068 0.018 0.032
  3.71 5.74 0.86 1.88
Other -0.011 -0.004 -0.003 0.002
  1.27 0.66 0.28 0.27
     
Teacher controls   ü ü
School fixed effects ü ü  
Student fixed effects   ü   ü
Same variables as earlier model specification.