Impact evaluation of new aid instruments

1
Impact evaluation of new aid instruments
NONIE Sub-group 3 NONIE meeting The Hague /
Scheveningen May 2007
2
How do we evaluate new aid modalities?

• Focus on the process funding modalities,
  coordination, implementation, harmonisation,
  power relations, incentives
• Sector Wide Approach (SWAp)
• General Budget Support (GBS)
• Paris Declaration
• Focus on results and impact

3
Evaluation of new aid modalities
Policy of donors and recipient country
Implementation Plan
Money
4
Evaluation of new aid modalities
Policy of donors and recipient country
Implementation Plan
Impact on beneficiaries
Money
5
Why?

• We need to show results in order to maintain the
  support for development cooperation
• We need to know how we can improve interventions
  in order to realise our common objectives
• A common policy formulation and a common
  financing mean a common interest in and
  responsibility for the evaluation of the sector
  policy

6
But you cant discern the contribution of a
specific donor.?

• So what!
• Yes you can!
• See for instance the child health study of the
  IEG in Bangladesh
• It was concluded that immunization saved the life
  of 2 million children at a cost of 100 - 300
  per life saved.

7
Vanishing benefits
8
What can we learn?

• A result of the Uganda and Zambia (and Ghana)
  studies is that investments in books, classrooms
  and teachers are much more effective when
  combined with good management.
• The Ghana study found that schools in poorer
  communities are poorer resourced. We see this in
  the Zambia study as well.

9
Impact evaluations are an important instrument in
the improvement of development policy
Policy of donors and recipient country
Implementation Plan
Impact on beneficiaries
Money
10
How to do an impact evaluation of sector support

• You cant do it without the owner (recipient
  country)
• Identify stakeholders and potential partners
  (demand generation)
• Develop the program theory together
• The learning function (capacity building) is an
  intrinsic part of the evaluation
• Include local consultants
• Establish a solid factual (see sub-group 2)
• Analysis be there

11
Methodological issues

• There is no clear cut start.
• The intervention is not discrete and not
  necessarily targeted at a specific group,
• There is no control group.
• The heterogeneity of interventions makes it more
  difficult to isolate specific interventions.

12
Western Province Program in Zambia (WEPEP)
Management
Learning achievements
Class size
Teachers
Books
Desks
13
Advantages

• At the sector level we may have much more units
  of analysis (households, communities, schools,
  etc) and therefore we may have larger samples.
• The heterogeneity of the objects of analysis
  creates the possibility of a multivariate
  analysis.
• This will probably lead to a regression based
  approach.

14
Data collection

• At the sectoral level more secondary
  (administrative) data (used for monitoring
  purposes) may be available.
• National surveys (DHS, LCMS) may be more useful,
  because the interventions are not restricted to a
  specific region.
• Linking data gives an opportunity to check the
  reliability.
• On the basis of missing data you may to conduct
  an additional survey.
• And once again test, test, test.

15
Challenges

• Ascertain ownership and the contribution of local
  stakeholders
• Include the main development agencies, but
• the evaluation must be manageable
• keep focused
• Whenever possible, do a great deal of the
  analysis in the recipient country
• You cant do an evaluation behind your desk
  (top). Visit the field.

16
Conclusions

• NONIE is especially functional in the case of
  country wide studies and new aid modalities.
• The results will help the development partners
  and development agencies to improve their
  interventions.

17
Thank you