Bangladesh Joint Country Assistance Evaluation: Assessing Total ODA at the Country Level Presentation to OECD DAC November 2006

1
Bangladesh Joint Country Assistance Evaluation
Assessing Total ODA at the Country
LevelPresentation to OECD DAC November 2006

• Bruce Murray
• Director General
• Operations Evaluation Department
• Asian Development Bank

2
Very Early Stages

• Everything said is subject to change.

3
Benefits of Total ODA Evaluation Richard
Mannings Challenge

• Consistent with Paris Declaration (i) mutual
  accountability (ii) partnering (iii)
  harmonization (iv) alignment.
• Reduced transaction costs for Government
• Objectivity and legitimacy
• Broader scope
• Ability to address issues that are beyond those
  addressed in evaluations undertaken by one donor
• October 2005 Paper The effectiveness of ODA an
  evaluation proposal focuses on the impact of
  total ODA on policy choice and implementation in
  a country.

4
Why Bangladesh as Pilot for Total ODA Evaluation?

• Many donors active in country
• 4 major donors have a joint country assistance
  program (WB ADB JBIC DfID)
• Some major donors require country evaluations by
  2009
• Use of SWAps.

5
Phasing of the Evaluation

• Phase 1 joint country evaluation by the World
  Bank, ADB, JBIC/JICA and DFID (80 of ODA).
• Phase 2 integrate the 28 donors delivering the
  other 20 of ODA.
• Daunting coordination problems. Need for strong
  information sharing between donors involved in
  Phases 1 and 2. Mechanism not yet developed.

6
Issues Requiring Resolution Involving the
Government

• Involving Government as partner in the evaluation
  work Paris Declaration on mutual
  accountability.
• Proposal to Government must be an open one,
  including on approach and methodology (listen to
  Governments views)
• Identify issues that the Government wants
  addressed (e.g., donor coordination
  harmonization use of country systems)
• Work with Government evaluation units and local
  research organizations.
• Involvement of NGOs that deliver much of the
  bilateral aid.

7
Issues Requiring Resolution Approach and
Methodology

• Identify issues that cannot be addressed in the
  country assistance evaluations of individual
  donors.
• What is the counterfactual?
• Need for common approach and methodology for the
  separate building blocks of the evaluation.
• Alignment with joint Country Strategy and PRSP.
  Note Joint Strategy assigns lead donors by
  sectors and themes.
• Lead donors evaluate all interventions in the
  sector/thematic area, regardless of funding (data
  base being assembled to map available evaluation
  reports and ongoing operations and strategy).

8
Issues Requiring Resolution Study Management

• Need a strong evaluation management mechanism
• Composition and role of steering committee
  Government, 4 major donors and 1 OECD DAC
  representative.
• Coordination among task managers and teams,
  mission planning, etc.
• How to balance internal needs of donors for
  evaluation of their programs and joint evaluation
  (attribution to total ODA, not individual donors)
• Reporting one report on total ODA or do donors
  also need separate reports on their operations
  for accountability? Design methodology to be
  flexible enough to produce single or multiple
  reports.
• Logistics, timing and cost sharing of the
  evaluation

9
Issues Needing Resolution Involving other Donors

• How to involve the donors delivering the other
  20 of ODA?
• 1 representative from the OECD DAC Total ODA task
  force on the steering group
• Involvement other donors up front in identifying
  the evaluation questions.
• Sharing of information during the sector/thematic
  evaluations
• Providing information during for the sector/
  thematic evaluations
• Role of country offices and Headquarters
  evaluation units.

10
Proposed Schedule
Present to end of 2006 Data collection and data sharing among main donors (on going) decision on first sector/thematic evaluations among WB/ADB/JBIC-JICA/DFID. Nomination of steering committee for joint evaluation and lead agencies for sectors/themes.
January-April 2007 Develop management structure for evaluation. Preparation of approach paper for CAE, one sector/thematic evaluation and for Jamuna Bridge/Access Road /Railway Link joint evaluation
April/May 2007 Proposal for joint country evaluation submitted to Government joint mission to Bangladesh to discuss proposal.
June 2007 Detailed decisions on joint country evaluation, costing and timing and mechanism for coordination with the other donors.
July 2007 Final decision on allocation of other sector/thematic evaluations to various evaluation partners
August-October 2007 Preparation of detailed evaluation approach papers for initial sector/thematic evaluations, preparation of TORs, consultant selection, detailed planning for joint missions, etc.
November 2007-June 2008 Field work
July December 2008 Discussion and drafting of individual sector/theme reports
January April 2009 Drafting and presentation of synthesis report Dissemination activities.
11
Challenges of Joint Evaluations (DAC Guidelines)

• Donor need for accountability and attribution
  often takes precedence over joint efforts to
  assess interrelated impacts of multiple agencies
• Finding subjects suitable for joint evaluations
  (program support, multilateral, multidonor funded
  projects) is difficult
• Processes for coordinating joint evaluations can
  be complex and increase the cost and duration of
  the evaluation

12
DAC Lessons Joint Evaluations

• Joint evaluations only reduce transaction costs
  if they replace, not add to, individual
  evaluations
• Identify key partners in evaluation involvement
  of government agencies as evaluators is good
  practice
• Agree on the management structure early (broad
  membership steering committee and smaller
  management group mixed approaches, some areas of
  joint evaluation, others to be delivered
  separately)

13
DAC Lessons Joint Evaluations

• Key areas of agreement
• Plan for dissemination and follow up