Developing an Evaluation Strategy

1
Developing an Evaluation Strategy experience in
DFIDNick YorkDirector Country, Corporate
and Global Evaluations, World Bank IEGFormer
Chief Professional Officer and Head of
Evaluation, DFIDPresentation to IEG Evaluation
Week 19 March 2013
2
Policy context

• First UK government department to have a formal
  evaluation policy (2009)
• Coalition government (May 2010) announced their
  commitment to rigorous, independent evaluation of
  a rapidly expanding aid programme
• Decided to honour the international commitments
  on aid (0.7 of GNI) but
• Strong focus on value for money and impact
• Decisions on the basis of evidence not
  guesswork
• Decision to set up Independent Commission for Aid
  Impact

3
Evaluation strategy - part of a wider strategy on
value for money, evidence and results

• Programme design and appraisal significant
  changes in DFID to move to new rigorous business
  case process including ex ante appraisal of
  evidence for new programs
• Evidence in decision making major changes to
  DFIDs research strategy and use of evidence in
  policy and programme design
• Strengthening performance monitoring and data
  systems in DFID and also in developing
  countries
• Developing skills and a culture where use of
  evidence and evaluation is the norm

4
Key strategic themes in evaluation policy and
approach in DFID

• Independence do you tackle this through
  structures and reporting, or professionalism and
  integrity of staff
• Relevant and useful recognise the endemic
  weakness in many evaluation systems on follow up
  and feedback into decision making
• High quality including rigour on methods and
  use of multidisciplinary, cutting edge techniques
• Partnership we depend on data systems and
  engagement from other development partners
• Capacity and systems in DFID and in developing
  countries

5
Independent Commission on Aid Impact

• Visibly independent and well resourced
• reports to Parliament, through international
  development select committee, not to DFID
  Ministers.
• Senior commissioners with strong leadership
  skills
• Staffing from National Audit Office secondees and
  private sector
• 5m per annum programme
• Covers all Overseas Development Aid, including in
  DFID but also other government departments.
• Forward work programme was based on public
  consultation but also kept flexibility to respond
  to topical issues from Parliament and media
• Strong follow up arrangements through
  reevaluation after 1 year

6
Embedding evaluation in DFID

• External scrutiny is only part of the picture
  need to build evaluation into the programme cycle
  and make it a routine part of how we do business
• Ambitious expansion in decentralised evaluations
  is planned number and quality
• Evaluations commissioned by operational teams
  with professional analytical input and support
  from centre
• Part of this is a much greater emphasis on
  rigorous methods and impact evaluation with clear
  counterfactual
• Requires a change in culture and an expansion in
  evaluation skills within operational/policy teams

7
Leadership, Incentives and Support

• Leadership is vital - Ministers and Board clear
  that high quality evaluation (including at
  operational level) is a priority within DFIDs
  approach to value for money
• Shift away from central evaluation department to
  encouraging staff to take ownership of agenda
• New evaluation cadre of specialists with peer
  recognition from accreditation, training and
  guidance
• Use professional groups such as economists,
  statisticians and health, education, governance,
  social development specialists etc.
• Financial incentives come from career
  opportunities, making additional resources
  available for new studies, and pump priming and
  offering funds for training and support

8
Strategic choices

• Types of evaluation process, summative, impact
  evaluation, post project evaluation
• Delivery modes from within DFID or through
  international systems and partners
• Role of centre versus operational and embedded
  analysts, how much to manage/deliver internally
  or contract out
• Timing - Design into programmes from the start
  to allow impact to be measured rigorously.
• Focus on external scrutiny and upward/outward
  accountability or providing evidence for
  decision making and internal lesson learning

9
Building a culture?

• Clear policy framework, leadership and
  incentives
• Take seriously the need for better data and
  indicators, including baselines that take account
  of how we will want to assess impact later in the
  programme
• Major shift in timing and approach from mainly ex
  post approach to prospective, formative and ex
  post. Build evaluations into programmes from the
  start and consider and plan for ME before
  programs are designed and resources are released.
  
• Build evaluation skills in DFID and in partner
  organisations, including systems for independent
  evaluation in country

10
Lessons and challenges (1)

• Shift to operational ownership of evaluations has
  major benefits on relevance, design, scope to
  measure impact and potential for lesson learning
  by decision makers
• More decentralised approach places big demands on
  skills and requires thought on quality standards
  and continuous professional development
• Creates risks around strategic focus and requires
  central mechanisms (led by policy teams) for
  lesson learning across the whole organisation (as
  opposed to within programs)

11
Lessons and challenges (2)

• Major expansion in impact evaluations (using a
  range of methods) and use of systematic reviews.
  This creates more global knowledge output from
  evaluation.
• Decentralisation of ME expertise allows much
  closer working with client countries, including
  for evaluation capacity development and joint,
  country-led evaluation
• Independence of evaluation and accountability -
  now mainly driven by ICAI (super independent) and
  within DFID the focus is on quality, credibility
  and integrity of evaluations with more
  potential for lesson learning