Using Impact Evaluation for Results Based Policy Making

1
Using Impact Evaluation for Results Based Policy
Making

• Arianna Legovini
• Impact Evaluation Cluster, AFTRL

Slides by Paul J. Gertler Sebastian Martinez
2
Answer Three Questions

• Why is evaluation valuable?
• What makes a good impact evaluation?
• How to implement evaluation?

3
IE Answers How do we turn this teacher
4
into this teacher?
5
Why Evaluate?

• Need evidence on what works
• Limited budget forces choices
• Bad policies could hurt
• Improve program/policy implementation
• Design eligibility, benefits
• Operations efficiency targeting
• Information key to sustainability
• Budget negotiations
• Informing beliefs and managing press

6
Allocate limited resources?

• Benefit-Cost analysis
• Comparison of choices
• Highest return investment
• Benefit
• Change in outcome indicators
• Measured through impact evaluation
• Cost
• Additional cost of providing benefit
• Not accounting cost

7
Traditional M E

• Monitoring
• Outcome trends over time
• e.g. poverty, school enrollment, mortality
• Process Evaluation
• Implementation
• Efficiency
• Targeting
• Administrative Data
• Management Information Systems

8
Impact Evaluation Answers

• What is effect of program on outcomes?
• How much better off are beneficiaries
  because of the intervention?
• How would outcomes change under alternative
  program designs?
• Does the program impact people differently (e.g.
  females, poor, minorities)
• Is the program cost-effective?
• Traditional ME cannot answer these

9
For Example IE Answers

• What is the effect of Job Training on employment
  and earnings?
• How much do cash transfers lower poverty?
• Do scholarships increase on school attendance for
  girls more than boys?
• Does contracting out primary health care to
  private sector lead to an increase in access?
• Does replacing dirt floors with cement reduce
  parasites improve child health?
• Do improved roads increase access to labor
  markets raise income for the poor?

10
Types of Impact Evaluation

• Efficacy
• Proof of Concept
• Pilot under ideal conditions
• Effectiveness
• Normal circumstances capabilities
• Impact will be lower
• Impact at higher scale will be different
• Costs will be different as there are
  economies of scale from fixed costs

11
So, Use impact evaluation to.

• Scale up pilot-interventions/programs
• Kill programs
• Adjust program benefits
• Inform (i.e. Finance Press)
• e.g. PROGRESA/OPORTUNIDADES (Mexico)
• Transition across presidential terms
• Expansion to 5 million households
• Change in benefits
• Battle with the press

12
Next question please

• Why is evaluation valuable?
• What makes a good impact evaluation?
• How to implement evaluation?

13
Assessing impact

• examples.
• How much does an anti-poverty program lower
  poverty?
• What is beneficiarys income with program
  compared to without program?
• Compare same individual with without programs
  at same point in time
• Never observe same individual with and without
  program at same point in time

14
Solving the evaluation problem

• Counterfactual what would have
  happened without the program
• Need to estimate counterfactual
• i.e. find a control or comparison group
• Counterfactual Criteria
• Treated counterfactual groups have identical
  characteristics on average,
• Only reason for the difference in
  outcomes is due to the intervention

15
2 Counterfeit Counterfactuals

• Before and after
• Same Individual before the treatment
• Non-Participants
• Those who choose not to enroll in program
• Those who were not offered the program

16
Before and After Examples

• Agricultural assistance program
• Financial assistance to purchase inputs
• Compare rice yields before and after
• Find fall in rice yield
• Did the program fail?
• Before is normal rainfall, but after is drought
• Could not separate (identify) effect of financial
  assistance program from effect of rainfall
• School scholarship program on enrollment

17
Before and After

• Compare Y before and after intervention
• A-B Estimated Impact
• B counterfactual Estimate
• Does not control for time varying factors
• C True counterfactual
• A-C True impact
• A-B is under-estimate

Y
After
Before
A
B
C
t-1
t
Time
18
Non-Participants.

• Compare non-participants to participants
• Counterfactual non-participant outcomes
• Problem why did they not participate?

19
Job training program example

• Eligible group offered job training
• Compare employment earning of those who sign up
  to those who did not
• Who signs up?
• Those who are most likely to benefit,
  i.e. those with more ability
• Would have higher earnings than non-participants
  without job training
• Poor estimate of counterfactual

20
Health Insurance Example

• Health insurance offered
• Compare health care utilization of those who got
  insurance to those who did not
• Who buys health insurance?
• Expect large medical expenditures
• Less healthy
• Who does not buy? The healthy!
• Cannot separately identify impact of insurance
  from health on utilization

21
What's wrong?

• Selection bias People choose to participate for
  specific reasons
• Many times reasons are directly related
  to the outcome of interest
• Job Training ability and earning
• Health Insurance health status
  and medical expenditures
• Cannot separately identify impact of the program
  from these other factors/reasons

22
Program placement example

• Govt offers family planning program to villages
  with high fertility
• Compare fertility in villages offered program to
  fertility in villages not offered
• Program targeted based on fertility, so
• Treatments have high fertility
• Counterfactuals have low fertility
• Cannot separately identify program impact from
  geographic targeting criteria

23
Need to know

• Why some get program and others not
• How beneficiaries get into treatment versus
  control group
• If reasons correlated w/ outcome
• cannot identify/separate program impact from
• other explanations of differences in outcomes
• The process by which data is generated

24
Possible Solutions

• Guarantee comparability of treatment
  and control groups
• ONLY remaining difference is intervention
• In this seminar we will consider
• Experimental design/randomization
• Quasi-experiments
• Regression Discontinuity
• Double differences
• Instrumental Variables

25
These solutions all involve

• knowing how the data are generated
• Randomization
• Give all equal chance of being in
  control or treatment groups
• Guarantees that all factors/characteristics will
  be on average equal btw groups
• Only difference is the intervention
• If not, need transparent observable criteria
  for who is offered program

26
The Last Question

• Why is evaluation valuable?
• What makes a good impact evaluation?
• How to implement evaluation?

27
Implementation Issues

• Policy relevance
• Political Economy
• Finding a good control group.
• Retrospective versus prospective designs
• Making the design compatible with operations
• Ethical Issues
• Relationship to results monitoring

28
The Policy Context

• IE needs answers policy questions
• What policy questions need to be answered?
• What outcomes answer those questions?
• What indicators measures outcomes?
• How much of a change in the outcomes
  would determine success?
• Example teacher performance-based pay
• Scale up pilot?
• Criteria Need at least a 10 increase in test
  scores with no change in unit costs

29
Political Economy

• Is IE needed for some policy purpose?
• Ex ante build into institutions of government
  decision-making
• Stakeholders Collaboration btw country,
  stakeholders evaluation team
• How will negative results affect program
  managers, policy makers stakeholders?
• Job performance vs knowledge generation
• Reward for using IE to change/close
  weak programs

30
Two paths to Control Groups

• Retrospective (very hard)
• Try to evaluate after program implemented
• Statistically model how governments individuals
  made allocation choices
• Cannot alter treatment or control group
• Prospective
• Can introduce some reasons for participation that
  are uncorrelated with outcomes
• Complement operational objectives
• Easier and more robust

31
Easier in prospective designs

• Generate good control groups
• Most interventions cannot immediately deliver
  benefits to all those legible
• Budgetary limitations
• Logistical limitations
• Typically phased in
• Those who go first are potential treatments
• Those who go later are potential controls
• Use Rollout to find control groups

32
Who goes first among equals?

• Cost considerations
• What is most efficient scale to deliver program
• Operations, social and political costs
• Individual/household or community?
• e.g. welfare program, roads, health insurance
• Eligibility criteria
• Are benefits targeted?
• How are they targeted?
• Can we rank eligible's priority?
• Are measures good enough for fine rankings?

33
Ethical Considerations

• Do not delay benefits Rollout based on
  budget/administrative constraints
• Equity equally deserving beneficiaries deserve
  an equal chance of going first
• Transparent accountable method
• Give everyone eligible an equal chance
  (e.g. Colombia School Vouchers, Mexico Tu
  Casa)
• If rank based on some criteria, then criteria
  should me quantitative and public

34
Retrospective Designs

• Hard to find good control groups
• Must live with arbitrary allocation rules
• Many times rules not transparent
• Administrative data must
• be good enough to make sure program
  was implemented as described
• identify beneficiaries, otherwise
  surveys will be costly
• Unless originally randomized, need
  pre-intervention baseline survey
• both controls and treatments

35
Retrospective evaluation.

• Need to control for differences between control
  treatment groups
• Unless have baseline difficult to use
  quasi-experimental methods
• Sometimes can do it with baseline if
• Know why beneficiaries are beneficiaries
• Observable criteria for program rollout

36
IE and Monitoring Systems

• Projects/programs regularly collect
  data for management purposes
• Typical content
• Lists of beneficiaries
• Distribution of benefits
• Expenditures
• Outcomes
• Ongoing process evaluation
• Key for impact evaluation

37
Monitoring systems key

• Verify who is beneficiary
• When started
• What benefits were actually delivered
• Compliance with any conditionalities
• Necessary condition for program
  to have an impact
• benefits need to get to targeted beneficiaries
• program implemented as designed

38
Use Monitoring data for IE

• Program monitoring data usually only collected in
  areas where active
• If start in control areas at same time as in
  treatment areas have baseline for both
• Add a couple of outcome indicators
• Very cost-effective as little need
  for additional special surveys
• Most IEs use only monitoring

39
Overall Messages

• Impact evaluation useful for
• Validation program design
• Adjusting program structure
• Communicating to finance ministry
  civil society
• A good evaluation design requires estimating the
  counterfactual
• What would have happen to beneficiaries
  if had not received the program
• Need to know all reasons why beneficaries got
  program others did not

40
Design Messages

• Address policy questions
• Institutional use of results
• Stakeholder buy-in
• Easiest to use prospective designs
• Take advantage of phase rollout
• Transparency accountability use quantitative
  and public criteria
• Equity give eligibles equal chance of going 1st
• Good monitoring systems administrative data can
  improve IE and lower costs