What Does Research Tell Us About Identifying Effective Teachers? - PowerPoint PPT Presentation

1 / 14
About This Presentation

What Does Research Tell Us About Identifying Effective Teachers?


What Does Research Tell Us About Identifying Effective Teachers? Jonah Rockoff Columbia Business School Nonprofit Leadership Forum, May 2010 – PowerPoint PPT presentation

Number of Views:112
Avg rating:3.0/5.0
Slides: 15
Provided by: Jona375


Transcript and Presenter's Notes

Title: What Does Research Tell Us About Identifying Effective Teachers?

What Does Research Tell Us About Identifying
Effective Teachers?
  • Jonah Rockoff
  • Columbia Business School

Nonprofit Leadership Forum, May 2010
First, Lets Define Effective
  • Can be an inputs based concept
  • Observable actions or characteristics
  • Can be outcomes based concept
  • Measured by student success
  • Recent work of economists focuses on outcomes
  • Use a value-added approach
  • Outcomes measured are typically standardized
    exams in math and reading, usually
    elementary/middle school
  • Movement to bring rigorous analysis to teacher
    evaluations based on in-class observation

Basics of Value Added Analysis
  • VA is all about comparing actual student outcomes
    to a counterfactual expectation
  • Suppose we knew the right counterfactual
    expectation for each child, call it A
  • Expected achievement w/ some basic level of
    educational quality (e.g., the average teacher)
  • Subtract expectation (A) from actual student
    achievement (A) call this G
  • To get VA for a teacher, take the average G
    across all of the students she taught

Setting Expectations
  • How to set up the counterfactual expectation is
    the big question in value-added work
  • Typically, we estimate expectations with data
  • Example set expectation (A) as the average
    achievement of students w/ the same prior test
  • Quality of estimates contingent on quality of the
    data and the process that generates it
  • Expectations set too low make teachers look good
    expectations set too high make them look bad

Potential Statistical Problems
  • 1 Systematic sorting of students
  • Concern here is bias
  • Unfair treatment of teachers that is systematic
  • Example Ps friends get easier kids
  • 2 Instability of VA estimates
  • Concern here is imprecision
  • If estimates are very noisy, using them for
    rewards/consequences means lots of mistakes
  • Also means it may be a poor motivational tool

Basic Findings from VA Research
  • Substantial variation in VA across teachers
  • 1 s.d. in VA ? 0.1 to 0.2 s.d. in achievement
  • A bit more variation in math than reading/ELA
  • Much of the variation is within schools
  • VA estimates appear to contain real power to
    predict teacher effectiveness as measured by
    student achievement
  • Stability across years is enough to appear useful
    in teacher evaluation
  • Bias is not a big deal overall, though it could
    matter for individual teachers

Results on Stability from KS, KRS
(1) Group Teachers, Years 1/2 (2) Compare in
Years 3/4 (3) Large Persistent Differences
Why Get Excited About Value Added?
  • Why not just hire good teachers?
  • Wise selection is the best means of improving the
    school system, and the greatest lack of economy
    exists wherever teachers have been poorly chosen.
  • Frank Pierrepont Graves, NYS Commissioner, 1932
  • Because it is, unfortunately, easier said than
  • Decades of work on type of certification,
    graduate education, exam scores, GPA, college
  • (Very) small, positive effects on student
  • Rockoff et al. (2008) non-traditional predictors
  • Personality, content knowledge, cognitive
    ability, self-efficacy, commercial teacher
    selection test score
  • Result no silver bullets, but moderate power to
    distinguish when pool measures into an index

What You Get is What You See?
  • Why not identify individuals likely to be
    effective teachers through direct observation of
  • There is consistent evidence that subjective
    evaluations of existing teachers are strongly
    related to gains in student achievement
  • Research extends back nearly a century
  • Hill (21), Gotham (45), Brookover (45),
    Anderson (54)
  • More recent analysis focuses on rubric-based
    teaching evaluations and principal opinions
  • Schacter Thum, Milanowski, Tyler et al.,
    Rockoff Speroni, Jacob Lefgren, Harris
    Sass, Rockoff et al.

Less Math, but No Less Difficult
  • One nice aspect of subjective evaluation is it
    does not rely on complicated formulae
  • However, the details of how evaluation is done
    present issues similar to VA analysis
  • Context (Does one size fit all?)
  • Focus (What goes on the evaluation form?)
  • Bias (Are evaluators fair and impartial?)
  • Imprecision (A few lessons ? a whole year?)

A (Modest?) Proposal
  • Provide VA estimates to principals
  • Help them with the problem of estimating A
  • Let them combine VA with other information (e.g.,
    observation) to evaluate teachers
  • NYC has done this Teacher Data Reports
  • Piloted in using a randomized control trial
  • Treatment principals received reports and
  • Rockoff et al. (2009) study this pilot using
    baseline and follow-up surveys of principals

Principals Evaluations and VA
  • Substantial variation in baseline evaluations
  • Strong relationship with VA estimates

New and Useful Information?
  • Were treatment principals evaluations affected
    by the VA reports?
  • Are the effects greater for more precise VA?

In Conclusion
  • Identifying highly effective teachers is near
    impossible if all you have to go on is a CV
  • Value-added and in-class observation offer
    potential insight into this problem
  • Both, of course, are imperfect
  • Innovative evaluation policies that begin to
    harness this information can raise teacher
    quality and improve student outcomes
Write a Comment
User Comments (0)
About PowerShow.com