Developing Consensus Principles and Standards for Evaluating Global and Regional Partnership Programs (GRPPs)

1
Developing Consensus Principles and Standards for
Evaluating Global and Regional Partnership
Programs (GRPPs)

• Progress Report to the Fifth Meeting of the DAC
  Evaluation Network, November 16, 2006
• Chris Gerrard
• GRPP Coordinator, IEG

2
Outline of the Presentation

• Background
• Stakeholder consultative workshop held
  onSeptember 28-29, 2006
• Presentation of the revised Draft Principles and
  Standards document
• Demand for a second, companion document
• IEGs continuing role moving forward

3
Background to the Presentation

• IEGs presentation to the Fourth Meeting onMarch
  31, 2006
• Broad support expressed for collectively
  developing consensus principles and standard for
  evaluating GRPPs
• Request that IEG play a leading role in
  developing them

4
What are GRPPs?

• Programmatic partnerships in which
• The partners contribute and pool resources
  (financial, technical, staff, reputational, etc.)
  toward achieving agreed-upon objectives over time
• The activities of the program are global,
  regional, or multi-country (not single country)
  in scope
• The partners establish a new organization with a
  governance structure and management unit to
  deliver these activities.
• The majority of GRPPs are housed in existing
  (host) organizations

5
Developing Consensus Evaluation Principles

• Why?
• To improve the independence and quality of
  program-level evaluations of GRPPs in order to
  enhance their relevance and effectiveness.
• For Whom?
• Governing bodies and management units of GRPPs
• Evaluators engaged in evaluating these programs
• High-level policy makers in aid agencies and
  developing countries

6
Workshop ParticipantsSeptember 28-29, 2006

• 17 Bilateral agencies and OECD secretariat
• 6 UN organizations
• 6 Multilateral development banks
• 3 Foundations
• 2 Developing country chairs of governing bodies
• 5 NGOs/private sector
• 6 GRPP management
• 3 GRPP evaluators
• 3 Evaluation associations

7
Objectives of the Workshop

• To review an initial set of draft principles and
  standards for GRPPs which IEG had prepared, based
  upon
• Extensive review of existing sets of related
  principles
• DAC Principles and Standards
• UNEG Norms and Standards
• ECG, evaluation associations, etc.
• Evaluative work that IEG done on GRPPs over the
  past few years www.worldbank.org/ieg/grpp

8
A Self-Standing and Comprehensive Document

• Evaluation Governance and Process Issues
• Enabling conditions, including ME framework
• Independence and impartiality
• Participation and inclusion
• Transparency and disclosure
• Planning for scope and methodology
• Evaluation Content and Criteria Issues
• Relevance
• Effectiveness (or efficacy)
• Efficiency or cost-effectiveness
• Governance and management
• Sustainability, risk, and strategy for devolution
  or exit

9
Major Improvements Between First and Second
Drafts

• Enabling conditions, including ME framework
• Independence and impartiality
• Participation and inclusion
• Planning for scope and methodology
• Relevance
• Effectiveness (or efficacy)
• Efficiency
• Governance and Management
• Sustainability

10
Demand for a Second Companion Document

• Workshop participants who represented the
  governing bodies and management of GRPPs liked
  the structure and content of the present
  document.
• Evaluators working in bilateral agencies, UN
  organizations and MDBs as well as GRPP evaluators
  desired something more detailed and specific to
  GRPPs, including guidance notes and good practice
  examples on specific issues
• A number of participants also offered to help
  contribute to the second document

11
Particular Requests for Guidance Notes and Good
Practice Examples

• Establishing good results-based ME systems
• Developing generic indicators for generic types
  of GRPP activities
• Ensuring independence of evaluation and
  mitigating conflicts of interest in small
  programs
• Preparing terms of references, including scope
  and methodology
• Assessing relevance, effectiveness, and
  efficiency
• Assessing legitimacy and effectiveness of
  governance and management
• Assessing alternative strategies for devolution
  or exit

12
ME Framework Often Lacking Among 36 Programs
IEG Has Reviewed
13
Information on Evaluation Process Also Lacking
in Evaluation Reports
14
Information on Evaluation Process inEvaluation
Reports (cont.)
15
Difficult to Assess Degree of Independence from
Evaluation Reports
16
Most Evaluations Assessed Relevance Few Assessed
Efficiency
17
Most Evaluations Used Interviews Document
Reviews Half Used Surveys
18
Suggested Options for Moving Forward

• Disseminate the draft Principles and Standards
  widely, without endorsing them at this time
• Encourage GRPP governing bodies management to
  be guided by these, and to provide feedback based
  on experience
• Revise the document in 1-2 years in the light of
  experience and feedback received
• Start preparing the second companion document,
  based on
• Reviews of existing GRPP evaluation policies, ME
  frameworks, and completed evaluations
• Consultations with partner representatives on
  governing bodies, GRPP management, GRPP
  evaluators, evaluation associations, and other
  evaluation experts