Discussion of Identifying Effective Teachers Using Performance on the Job Gordon, Kane

1
Discussion ofIdentifying Effective Teachers
Using Performance on the Job(Gordon, Kane
Steiger)

• Henry Braun
• Boston College
• Teacher Quality Conference
• Northwestern University
• May 1 2008

2
GKS A Modest Proposal

• A plan to restructure the labor market and
  working conditions for teachers Making
  performance a more significant factor in
  teacher advancement and compensation with
  value-added measures given special prominence
• Comprehensive
• Timely
• Scalable
• Costed
• Defended

3
What really matters?

• the success of U.S. public education depends
  upon the skills of the 3.1 million teachers
  managing classrooms in the elementary and
  secondary schools around the country. Everything
  else educational standards, testing, class
  size, greater accountability is background
  Without the right people standing in front of the
  classroom, school reform is a futile exercise.
  (p.5)

4
An Ecological Perspective on Reform

• What are the highest priority education goals?
• e.g. Greater high school completion rates with
  students well-prepared for life beyond high
  school
• What are the most critical requirements?
• e.g. A teaching force that is (i) demonstrably
  effective, (ii) adequately large, (iii)
  appropriately distributed
• e.g. Schools that are (i) well led and
  collaborative, (ii) properly resourced, (iii)
  learning organizations
• What is the context in which the proposed reforms
  will be implemented?
• e.g. (i) a technically flawed accountability
  system that generates perverse incentives, (ii)
  problematic union-management relationships, (iii)
  concerns about over-reliance on test scores
• What are the warrants for systems validity?
• e.g. (i) alignment of ends and means, (ii)
  coherence of system components, (iii)
  feasibility, (iv) designs to mitigate negative
  consequences

5
Teaching The Current Scene

• Initial entry based largely on credentials
• Tenure an early and semi-automatic decision
• Uniform salary schedules based on credentials and
  years of experience
• Seniority rights
• High mobility rates (localized)
• High levels of out-of-field teaching (localized)

6
The Argument

• Significant variation among teachers in
  effectiveness (as measured through value-added
  analysis)
• Existing barriers to entry are dysfunctional
• Tenure decisions are now generally
  information-poor
• Value-added measures of effectiveness are a
  credible basis for decision-making
• Current distribution of teacher quality is
  sub-optimal

7
Recommendation 1 Reduce barriers to entry

• A well-documented need to increase access to the
  teaching profession
• Current licensing standards are often
  artificially lowered
• How should we assess (pedagogical) content
  knowledge? What standards?
• How would this work in the elementary grades?
• What about special education?
• What would be the function of the M. Ed.?

8
Reflections on Teaching

• I came to understand that teaching is hard and
  that being smart and well-educated doesnt
  necessarily mean one will be good at it.
• (Checker Finn, on his first year of teaching
• social studies in Newton HS, Newton, MA)
• The evidence indicates that neither an extreme
  central-ized bureaucratization nor a complete
  deregulation of teacher requirements is a wise
  approach for improving teacher quality.
• (Jennifer King Rice, 2008)

9
Recommendation 2 Modify tenure process

• A more rigorous evaluation for tenure is needed
• Outcome-based indicators should be considered
• Better training for evaluators (e.g. Principals)
• More equitable provision of support and mentoring
• Lengthier probationary period
• Salary consequences?
• The question we dare not ask Why tenure?

10
Recommendation 3 Introduce differential pay

• Inequitable distribution of experienced and more
  effective teachers
• Use substantial bonuses to shift incentives
• Need a stable index for ranking hard-to-staff
  schools that incorporates more than student
  disadvantage
• Avoid hard cut-offs
• Bonus teachers should bring school resource
  dollars to jumpstart improved working conditions
• Must retain bonus-eligible teachers already in
  such schools
• Countervailing pressure of NCLB sanctions

11
Recommendation 4 Change teacher compensation

• Seven challenges are well worth considering!
• Improve credibility of all forms of evaluation
• Pay attention to a number of desired outcomes
  (i.e., aim for balanced accountability)
• Dont place excessive burden on test scores
• Idiosyncratic school-level conditions are
  problematic
• School leadership
• Out-of-field teaching
• Grade-shifting

12
Recommendation 5 Develop data systems

• Yes, yes, yes
• Molly Bloom (1922), Hillary Clinton (2008)
• Federal government stepping up to the plate here!
• Need substantial investment (and commitment) to
  build (and to maintain) data warehouses
• Many state education departments are
  under-staffed in technology and data management

13
Using (Test-based) Performance Indicators (I)

• Intuitively attractive and commonsensical
• Certain measures of performance contain useful
  information
• If indicators are seen as (relatively) fair, then
  attaching stakes to performance focuses attention
  and can be constructive
• Provides a quantitative and comparable basis for
  decision-making

14
Using (Test-based) Performance Indicators (II)

• Cognitive outcomes are a subset of education
  goals
• Current (state) tests are proxies of variable
  quality for the target cognitive outcomes
• Mis-alignment between proxies and targets can
  lead to sub-optimal resource allocation (Murnane
  Cohen, George Baker, and others)
• Disagreement on effectiveness of extrinsic
  motivation (Ed Deci, and others)
• Mixed findings on success of pay-for-performance
  schemes (Podgursky, Harris, and others)
• Implications of Campbells Law

15
Value-added Modeling (I)

• VAMs attempt to isolate the contributions of
  teachers to student learning as measured by test
  scores
• VAMs adjust current student performance for
  differences in prior performance and certain
  background characteristics (creating a level
  playing field)
• Superior to status models and growth models as a
  basis for performance evaluation
• Many models are in use and have been studied
  extensively with regard to their operating
  characteristics and sensitivity to departures
  from assumptions
• Policy use of VAM results implicitly relies on a
  causal inference

16
Value-added Models (II)

• Fundamental technical problem in using VAMs
• In making causal attributions from
  observational data, no statistical model, however
  complex, and no method of analysis, however
  sophisticated, can fully compensate for the lack
  of randomization.
• (Braun, 2005)
• Practical Questions
• (i) How close can we come?
• (ii) How do we know when we are close enough?
• (iii) What are the limitations on use?

17
Value-added Models (III)

• Other technical considerations in the use of VAMs
• Test quality (substantive)
• Test characteristics (psychometric)
• Balancing model complexity against test
  information density
• Volatility
• Bias
• How do we go beyond math and language arts?

18
Value-added Models (IV)

• Operational and political considerations
• Administrative complexity
• VAM infrastructure
• Multiple review systems
• Training and auditing
• Fairness
• Transparency
• Level playing field
• NCLB/AYP
• Core opposition to use of student results
• Unions (e.g. New York state)
• Concern with over-reliance on tests

19
Twas Ever Thus

• For more than a hundred years much complaint has
  been made of the unmethodical way in which
  schools are conducted, but it is only within the
  last thirty that any serious attempt has been
  made to find a remedy for this state of things.
  And with what result? Schools remain exactly as
  they were!
• John Amos Comenius, 1632
• (quoted by C. E. Silberman in
• Crisis in the Classroom, 1970)

20
The Status Quo Is Not Viable

• Demographic trends and static literacy
  distributions portend a downward shift in per
  capita human capital
• (Americas Perfect Storm)
• Staggering opportunity costs in not increasing
  high school graduation rates
• (The Price We Pay)
• More than 50 of non-capital school spending goes
  to teachers
• (Understanding the Effectiveness of Teacher
  Attributes)
• Too many ships in the American convoy are taking
  on water
• (Any newspaper)

21
CONCLUSIONS

• This is a thoughtful proposal to address a
  serious national need, grounded in the belief
  that data-based decision-making should play a
  greater role in shaping the teaching profession.
• I agree -- in principle!
• As always -- the devil is in the details!
• More attention is needed to
• Full design of the new system
• Infrastructure
• School realities
• Consequential validity

22
Parting Thoughts

• Teacher compensation is the elephant in the
  room.
• We need a constructive debate around various
  proposals to assist us in moving forward
• and rigorous evaluation to tell us what works
  and why!
• Question yet to be answered
• Is the uniform salary schedule to teacher
  compensation as democracy is to forms of
  government --- the worst possible system, except
  for all the others?