An introduction to Impact Evaluation (IE) for HIV/AIDS Programs

1
An introduction to Impact Evaluation (IE) for
HIV/AIDS Programs
Léandre Bassolé ACTafrica, The World Bank

• March 12, 2009
• Cape Town

2
Background

• Initial response to AIDS epidemic responding to
  a crisis.
• Build political commitment
• Establish institutions
• Scale up coverage of HIV prevention, treatment
  and mitigation services
• Now we need to know what really works?

3
Traditional ME and Impact Evaluation

• ME monitoring process evaluation

• Is program being implemented efficiently?
• Is targeted population being reached?
• Are outcomes moving in the right direction?

Descriptive analysis

• Impact Evaluation

• What was the effect of the program on outcomes?
• How would outcomes change under alternative
  program designs?
• Is the program cost-effective?

Causal analysis
4
Why does impact evaluation matter?

• To know if the program had an impact and the
  average size of that impact
• Assess if policies work
• Assess the net benefits/costs of the program
• Assess the distribution of gains and losses

5
The IE problem

• What do we mean by impact evaluation ?
• Impact the difference between the relevant
  outcome indicator with the program and that
  without it.
• However, we can never observe someone in two
  different states of nature at the same time.
• While a post-intervention indicator is
  observed, its value in the absence of the program
  is not, i.e., it is a counter-factual.
• So all IE deals with overcoming the issue of
  missing data. Requires counterfactual analysis.

6
What we need
Given the problem of missing data (individuals
have only 1 existence) we can compare 2 groups
we need A counterfactual a control/comparison
group that will allow us to attribute any change
in the participant group to the intervention
(causality) what would have happened
without the program
7
Common IE practices

• 1. Before and after 2. Participants-non-partici
  pants
• BUT, its difficult to assess the
• TRUE AVERAGE CAUSAL EFFECT
• How to solve the FUNDAMENTAL PROBLEM OF
  EVALUATION?

8
Comparison group issues

• Two central problems
• Programs are targeted
• Program areas will differ in observable and
  unobservable ways precisely because the program
  intended this
• Individual participation is (usually) voluntary
• Participants will differ from non-participants in
  observables and unobservable ways.

9
Tools to identify a Counterfactual

• Randomized Designs
• Quasi-experimental Designs
• Matching
• Instrumental variables
• Regression discontinuity

10
Some general principles to consider when planning
an IE

• Government ownershipwhat matters is
  institutional buy-in
• Relevance and applicabilityasking the right
  questions
• Flexibility and adaptability
• Horizon matters

11
Summing upMethods/Practicalities

• Randomization is the gold standard
• Be flexible, be creative use the context
• IE requires good monitoring and monitoring will
  help you understand the effect size

12
Summing upMethods/Practicalities

• Making IE works for you may require a change in
  the culture of project design and
  implementation..that is to maximize the
  evidence-based upon which policy decisions can be
  made to improve the chances for success
• Impact evaluation is more than a tool it is an
  analytical framework for policy development

13
THANK YOU