Developing evaluation capacity for national poverty reduction strategies Zenda Ofir

1
Developing evaluation capacity for national
poverty reduction strategies Zenda Ofir

• Workshop on Current Challenges for Development
  Evaluation
• University of Sussex, 2-3 October 2006

2
ECD in the South a real challenge

• The history and image of (development) evaluation
  
• The management / implementation gap (esp. in
  Africa, less so in Latin America and Asia)
• Positive developments, including towards creating
  a demand for evaluation information

3
ECD in the South a real challenge (cont.)

• Some (additional) new challenges for South ME
  
• (i) institutionalisation of country-level ME -
  complex, integrated programming, with whole of
  government approach and high level targets
• (ii) higher level evaluations requiring new
  methodologies - including moving towards a
  systems evaluation approach
• (iii) joint or reciprocal evaluation efforts of
  the aid system, of development the system, of
  MDGs, of MDG 8 - towards more symmetrical (power)
  relationships and contributions
• (v) democratisation of ME
• (vi) evaluation (what works, what does not and
  why?) rather than an overly strong emphasis on
  monitoring (obsession with data)
• (vii) understanding the role and use of ME
  within the body of knowledge on, and practices of
  knowledge management

4
ECD in the South a real challenge (cont.)

• A slow and complex process - even in the North
• It is about creating a sound balance between
  demand and supply towards a culture of ME -
  unleashing, nurturing and retaining ME demand
  and capacity
• Begs some fundamental questions underpinning ECD
  - If ME is made our own by the South, how should
  it be viewed, marketed and used to increase
  demand? What are the implications for evaluation
  theory and practice as we know it?
• As a discipline and profession in its infancy -
  can it become as integrated into our work
  processes as planning, budgeting, auditing,
  communicating etc.?
• How can we best marry ME for (i) enhanced
  accountability, (ii) learning - for collective
  decision-making, and (iii) informed participation
  in development processes?
• And (i) independent evaluation, (ii)
  self-reflection and (iii) self-evaluation?
• How can we make all ME processes empowering to
  help ensure its sustained role in improving
  development?

5
Components of a poverty reduction strategy ME
system

• Poverty monitoring - progress in poverty
  reduction against national targets, MDGs - easier
  as capacities are concentrated and easily
  outsourced
• ME of progress in implementation of poverty
  reduction policies and programmes - difficult to
  get data from range of actors based on selection
  of indicators based on clear results chains
  provides most useful information in short term
• Expenditure tracking - needs parallel progress
  in budget and public expenditure management

6
Some lessons from experience institutionalising
country ME systems

• Main challenge The institutional setup to
  strengthen national capacity to provide policy
  makers etc. with feedback on policies and
  interventions
• Substantive government demand is a prerequisite
  for successful institutionalisation (created
  through increased awareness, understanding,
  evidence of high returns on investment)
• Incentives are important (support e.g. budget
  process, national planning, public sector reform)
• Powerful champions play a key role
• Synthesis from ECD Working Paper Series 15, IEG,
  2006

7
Some lessons from experience institutionalising
country ME systems (cont.)

• Start with diagnosis of existing ME
• Must be centrally driven by a capable ministry
  (planning, finance, O-(V)P)
• Try not to over-engineer the system
• Utilisation is the measure of success
• There are limitations to relying on government
  laws, decrees, regulations
• Structural arrangements have to ensure ME
  objectivity and quality
• Strength has to be built in analysis, not just
  data gathering
• Synthesis from ECD Working Paper Series 15, IEG,
  2006

8
Some lessons from experience progress in
building country ME systems

• Progress has been limited
• Practical difficulties with data collection
• Difficulties in coordination of activities -
  territoriality
• Lack of operational detail, costing and
  prioritization - cannot get meaningful indicators
• Weaknesses in public expenditure management
  systems - cannot track PRS expenditures
• Deficit in analysis and evaluation
• Low demand for information among policy makers
• From Beyond the numbers - Understanding the
  Institutions for Monitoring Poverty Reduction
  Strategies. 2006. The World Bank

9
Some lessons from experience organising country
ME systems

• Strong political leadership - ME system close to
  government centre or budget process - but
  evaluation should be independent of misuse by the
  leadership
• Coordination of actors - not burdensome, but with
  clear roles and responsibilities donors should
  be served through this system
• Links with line ministries - liaison points
  should be involved in ME for sector policy
  making should have authority, time and
  incentives
• Complementarity with existing statistical systems
  and statistical planning - should have resources
  and mandates
• Challenges of ME in decentralised system -
  capacity constraints, esp. in poorest areas

10
Some lessons from experience - making use of
country ME systems

• More attention has been paid to organising
  supply than to ensuring demand and use for
  improvements.
• Low levels of demand in turn impact negatively
  on the supply of information.
• Most promising strategy for strengthening demand
  is to tailor the ME system outputs to key points
  in the policy-making process where information is
  likely to be influential (e.g. budgeting, MTEF,
  planning cycles, updates of strategies,
  parliamentary sessions, public dialogue, donor
  strategy elaboration).
• These processes should guide system activities.

11
Some lessons from experience - making use of
country ME systems (cont.)

• Important elements in encouraging use of system
• Analysis and evaluation has to be
  institutionalised in the system - currently a
  striking deficit - focus is on monitoring
• Information and analysis have to be compiled into
  outputs and disseminated across government and
  the public
• Link the system to the budget process - but avoid
  perverse incentives
• Link with parliament to help facilitate their
  role of oversight
• Involve civil society actors - participation in
  committees and working groups, providing analysis
  and advice, interpreting and disseminating
  outputs to general public

12
Points of power - the drivers of development
evaluation quality, usefulness and use (and
demand) - and thus targets for ECD

• Commissioners / users of ME - government
  decision-makers
• Commissioners / users of ME - donor networks
  (currently mainly from the North)
• Academic institutions / research centres and
  networks / think-tanks (currently mainly in the
  North)
• Evaluators and their associations (North and
  South)
• Civil society (incl. Parliaments, CSOs mainly in
  the South)

13
Role of commissioners in ECD

• Co-guardians of evaluation quality and
  usefulness
• Can encourage empowering evaluation processes
  that stimulate use of evaluation information
• Can help ensure quality evaluations -
  methodologies, application of values / principles
  / standards, and meta-evaluation
• Determines who conducts evaluations - evaluation
  specialists and subject specialists
• Determines access to evaluation information /
  lessons
• Need to build own evaluation capacities, esp. in
  view of decentralisation of evaluation function
• Support to ECD in the South - but what?

14
Role of academic / research institutions,
think-tanks and networks in ECD(may include
evaluation practitioners)

• Research, develop and publish / push evaluation
  theory and practice
• Develop the discipline
• Responsible for variety of (development)
  evaluation paradigms
• Major differences between those who influence
  evaluation and those who influence development
  evaluation

15
Role of evaluators and evaluation associations in
ECD

• Key role in evaluation as a profession,
  discipline and career
• Co-guardians of evaluation methodology, quality
  and usefulness
• (Potential) developers of evaluation theory and
  practice
• Associations
• magnify evaluators voice, influence
• provide effective platforms for inter-sector
  cross-fertilisation and advocacy - creating
  awareness and understanding
• establish and encourage use of values,
  principles, standards
• can play a key role in capacity building
• provide a channel for concerted action across
  region / continent

16
Role of civil society in ECD

• A voice to demand government and donor
  accountability
• Use methods for self-reflection, creating
  evaluative thinking
• Use evaluation for improved management of local
  interventions and systems

17
Some key priorities for ECD

• Critical priority Mobilise and broaden the
  intellectual base for ME in the South. Drive
  this from the South, with North support.
• Establish new types of relationships and bodies
  between South and North (between donors and
  evaluators / associations / think-tanks /
  governments) to develop joint strategies - for
  quality use development of practice and theory
  evaluation of aid / development systems, etc.
• Short-term pull together a group of highly
  skilled South evaluation practitioners to
  strategise how to grow demand for key
  interactions, including with the North, and for
  critical independent analyses and research -
  including how evaluation should be integrated
  into the South

18
Some key priorities for ECD (cont.)

• Develop centres or networks of excellence in
  evaluation, in and driven by the South, that can
  focus on research on evaluation and on
  postgraduate studies in evaluation - with strong
  North / South and South / South ties mobilise
  African evaluators abroad
• Develop systematic tailor-made training for
  clearly specified audiences, focusing on
  evaluators and governments and training the
  trainers - transplant IPDET to the South, per
  region
• In the long term - work to mainstream evaluation
  effectively into other courses - management,
  international development, etc.
• Stimulate work on the interface between knowledge
  management and evaluation theory and practice -
  and use to create learning organisations

19
Some key priorities for ECD

• Strengthen evaluation associations, including
  their leadership / management, and use them to
  address key issues common to countries
• Use associations to share with governments
  strategically and systematically case studies and
  models, returns on investment show benefits
  create opportunities for interaction and exposure
• Build commissioners / donors capacities and
  approaches (i) methodology (ii) focus on
  standards, quality, and harmonisation with
  associations (iii) insist on using evaluation
  association members - not (only) subject
  specialists, policy analysts (iv) communication
  of results to diverse stakeholders
• Work with Parliaments to create awareness and
  understanding of evaluation