Methods for Assessing Policy Impact

1
Methods for Assessing Policy Impact

• Process and Partnerships for Pro-Poor Policy
  Change, Project Initiation Workshop 1
• ILRI, 21st February 2005

2
Workshop Outline

• Introduction to the RAPID Framework and ILRI/ODI
  Project
• Case Study Approach
• Episode Study Approach
• Outcome Mapping Approach
• RAPID Outcome Assessment (ROA) Approach
• Lunch
• Practical Sessions

3
Workshop Purpose Objectives
Purpose To familiarise the participants with
the general approach and specific methods to be
used in the SDP case study

• Objectives
• By the end of the workshop, participants will
• understand the Process and Partnership for
  Pro-Poor Policy Change projects purpose and
  general approach
• have the opportunity to contribute their own
  suggestions to improve the project
• understand, and have had the chance to try out
  the three key methods which will be used in the
  project
• assess the usefulness of the approaches in their
  own work.

4

• An introduction to the RAPID Framework and
  ILRI/ODI Project

5
Definitions

• Research any systematic effort to increase the
  stock of knowledge
• Policy a purposive course of action followed by
  an actor or set of actors
• Agendas / policy horizons
• Official statements documents
• Patterns of spending
• Implementation processes
• Activities on the ground

6
Policy Processes

• - Identify a policy problem
• - Commission research
• - Assess the results
• - Select the best policy
• - Establish the policy
  framework
• - Implement the policy
• - The problem is
  solved

7
Reality

• The whole life of policy is a chaos of purposes
  and accidents. It is not at all a matter of the
  rational implementation of the so-called
  decisions through selected strategies 1
• Most policy research on African agriculture is
  irrelevant to agricultural and overall economic
  policy in Africa2

1 - Clay Schaffer (1984), Room for Manoeuvre
An Exploration of Public Policy in
Agricultural and Rural Development, Heineman
Educational Books, London 2 Omamo (2003),
Policy Research on African Agriculture Trends,
Gaps, and Challenges, International Service
for National Agricultural Research (ISNAR)
Research Report No 21
8
Existing theory

• Linear model
• Percolation model, Weiss
• Tipping point model, Gladwell
• Context, evidence, links framework, ODI
• Policy narratives, Roe
• Systems model (NSI)
• External forces, Lindquist
• Room for manoeuvre, Clay Schaffer
• Street level bureaucrats, Lipsky
• Policy as social experiments, Rondinelli
• Policy Streams Windows, Kingdon
• Disjointed incrementalism, Lindquist
• The tipping point, Gladwell
• Crisis model, Kuhn
• Framework of possible thought, Chomsky
• Variables for Credibility, Beach
• The source is as important as content, Gladwell

• Linear model of communication, Shannon
• Interactive model,
• Simple and surprising stories, Communication
  Theory
• Provide solutions, Marketing Theory I
• Find the right packaging, Marketing II
• Elicit a response, Kottler
• Translation of technology, Volkow
• Epistemic communities
• Policy communities
• Advocacy coalitions etc, Pross
• Negotiation through networks, Sebattier
• Shadow networks, Klickert
• Chains of accountability, Fine
• Communication for social change, Rockefeller
• Wheels and webs, Chapman Fisher

www.odi.org.uk/rapid/lessons/theory
9
Existing theory a short list

• Policy narratives, Roe
• Systems of Innovation Model, (NSI)
• Room for manoeuvre, Clay Schaffer
• Street level bureaucrats, Lipsky
• Policy as social experiments, Rondene
• Policy streams and policy windows, Kingdon
• Disjointed Incrementalism, Lindblom
• Social Epidemics, Gladwell

• The RAPID Framework

10
An Analytical Framework
External Influences Socio-economic and
cultural influences, donor policies etc
11
Case Studies

• Sustainable Livelihoods The Evolution of DFID
  Policy
• The PRSP Initiative Research in Multilateral
  Policy Change
• The adoption of Ethical Principles in
  Humanitarian Aid post Rwanda
• Animal Health Care in Kenya Evidence fails to
  influence Policy

12
A Practical Framework
political context
Politics and Policymaking
Media, Advocacy, Networking
Research, learning thinking
evidence
links
13
What you need to know

• The external environment Who are the key actors?
  What is their agenda? How do they influence the
  political context?
• The political context Is there political
  interest in change? Is there room for manoeuvre?
  How do they perceive the problem?
• The evidence Is it there? Is it relevant? Is it
  practically useful? Are the concepts familiar or
  new? Does it need re-packaging?
• Links Who are the key individuals? Are there
  existing networks to use? How best to transfer
  the information? The media? Campaigns?

14
What researchers need to do

• Get to know the policymakers.
• Identify friends and foes.
• Prepare for policy opportunities.
• Look out for policy windows.

• Work with them seek commissions
• Strategic opportunism prepare for known events
  resources for others

• Who are the policymakers?
• Is there demand for ideas?
• What is the policy process?

• Establish credibility
• Provide practical solutions
• Establish legitimacy.
• Present clear options
• Use familiar narratives.

• Build a reputation
• Action-research
• Pilot projects to generate legitimacy
• Good communication

• What is the current theory?
• What are the narratives?
• How divergent is it?

• Build partnerships.
• Identify key networkers, mavens and salesmen.
• Use informal contacts

• Get to know the others
• Work through existing networks.
• Build coalitions.
• Build new policy networks.

• Who are the stakeholders?
• What networks exist?
• Who are the connectors, mavens and salesmen?

15
Policy entrepreneurs
Networkers
Storytellers
Engineers
Fixers
16
Practical Tools
Overarching Tools - The RAPID
Framework - Using the Framework -
The Entrepreneurship Questionnaire
Context Assessment Tools - Stakeholder Analysis
- Forcefield Analysis - Writeshops -
Policy Mapping - Political Context
Mapping
Communication Tools - Communications
Strategy - SWOT analysis - Message Design -
Making use of the media
Research Tools - Case Studies
- Episode Studies - Surveys -
Bibliometric Analysis - Focus Group Discussion
Policy Influence Tools - Influence Mapping
Power Mapping - Lobbying and Advocacy -
Campaigning A Simple Guide - Competency
self-assessment
17
Practical Application

• Within ODI
• Workshops for researchers, policy makers and
  activists.
• Advice to a DFID forest/ground water research
  project in India
• Less research
• More communication
• Developing champions in regional and national
  government
• Local, Regional National advocacy campaign

18
Further Information / Resources

• ODI Working Papers
• Bridging Research and Policy Book
• Meeting series Monograph
• Tools for Policy Impact
• RAPID Briefing Paper
• www.odi.org.uk/rapid

19
Can ILRI do it?

• Yes, but
• It this its role?
• Global Public Good Research vs Policy Advocacy
• Probably needs to do both
• How?
• Understand the political context
• Get the evidence package it well
• Strategic networking / lobbying / campaigning
• Collaboration.

20
ILRI International Livestock
Research Institute
Process and partnership for pro-poor policy change
The New DfID funded Project
21
ILRI International Livestock
Research Institute

• Project Leaders ODI / ILRI
• Key collaborators ECAPAPA
• Case study collaborators in Kenya
• MoLFD / KARI
• Range of NGOs other SDP partners

22
Why would I be interested?
ILRI International Livestock
Research Institute

• Not all research is expected or intended to lead
  to policy change, but there may be
• Specific cases where research is expected to
• provide evidence for policy change
• identify potential policies (or impact of)
• influence the policy making process (advocacy)
• Cases where speculative research becomes relevant
  because of changes in circumstance

23
The project
ILRI International Livestock
Research Institute

• Ideas for methods and approaches
• Lessons learnt from earlier activities
• Identification of appropriate communication tools

24
What will we be doing?
ILRI International Livestock
Research Institute

• Three case studies in three DIFFERENT countries
• A project considered to have influenced policy
  change
• A stream of research addressing a particular
  policy area
• A clear policy change
• New policy statement
• New law
• Irrefutable change in way something is done

25
What will we be doing?
ILRI International Livestock
Research Institute

• Three case studies
• SDP and impact on changed view of informal milk
  trade
• ????
• ????

26

• Discussion
• Can ILRI hope to influence pro-poor policy
  through research?
• Any good case studies?

27

• Case Study Approach

28
What is a Case Study?

• Definition
• " A systematic inquiry into an event or a set of
  related events which aims to describe and explain
  the phenomenon of interest" Bromley (1990)

29
Why is it useful?
Goal to describe as accurately as possible the
fullest, most complete description of the case.

• An ideal methodology when a holistic, in-depth
  investigation is needed
• Designed to bring out the details from the
  viewpoint of the participants by using multiple
  sources of data

30
Types of Case Study

• Types of case studies
• Exploratory,
• Explanatory,
• Descriptive (Yin, 1993)
• Stake (1995) included three others
• Intrinsic - when the researcher has an interest
  in the case
• Instrumental - when the case is used to
  understand more than what is obvious to the
  observer
• Collective - when a group of cases is studied.

31
Issues

• The unit of analysis is a critical factor
• Typically a system of action rather than an
  individual or group of individuals
• Tend to be selective, focusing on one or two
  issues that are fundamental to understanding the
  system being examined
• Case studies are multi-perspectival analyses
• The researcher considers not just the voice and
  perspective of the actors, but also of the
  relevant groups of actors and the interaction
  between them
• They give a voice to the powerless and voiceless.

32
Triangulation

• Data source triangulation, when the researcher
  looks for the data to remain the same in
  different contexts
• Investigator triangulation, when several
  investigators examine the same phenomenon
• Theory triangulation, when investigators with
  different view points interpret the same results
  and
• Methodological triangulation, when one approach
  is followed by another, to increase confidence in
  the interpretation.

33
Applications

• To explain complex causal links between research
  and policy
• To describe the real-life context in which policy
  has been influenced by research
• To describe the policy influencing process itself
  
• To explore those situations in which the policy
  intervention being evaluated has no clear set of
  outcomes.

34
Process

• Design the case study protocol
• determine the required skills
• develop and review the protocol
• Conduct the case study
• prepare for data collection
• distribute questionnaire
• conduct interviews
• Analyze case study evidence
• analytic strategy
• Develop conclusions, recommendations, and
  implications based on the evidence

35

• Episode Study
• Approach

36
What is an Episode Study

• A study that focuses on a clear policy change
  and tracks back to assess what impact research
  had among the variety of issues that led to the
  policy change.

37
What is the purpose?
Tracking backwards from policy change to any
particular research which influence policy

• an excellent way of investigating the influence
  of research on policy
• Can focus on a single episode or comparative
  episodes.

38
Advantage

• The process of working backwards in time gives a
  more realistic view of the broad range of factors
  other than research that influence policy
• Tracking forward probably overemphasizes
• the importance of research

39
Issues

• Policy processes are complex, multi-layered and
  change over time
• Often difficult to isolate the impact of research
  from other factors
• Actors may re-write history
• Important to seek the views of a wide range of
  informed stakeholders
• The process of preparing an episode study is
  iterative
• Key facts and / or inconsistencies need to be
  cross-checked with key informants

40
Process

• Identify a clear policy change.
• Identify key Research Questions (draw on RAPID
  framework)
• Explore how and why those policy decisions and
  practices took place
• Assess the relative role of research in that
  process by drawing on the framework.

41
Apply the RAPID Framework
political context
Politics and Policymaking
Media, Advocacy, Networking
Research, learning thinking
evidence
links
42
Key Questions

• The external environment Who are the key actors?
  What is their agenda? How do they influence the
  political context?
• The political context Is there political
  interest in change? Is there room for manoeuvre?
  How do they perceive the problem?
• The evidence Is it there? Is it relevant? Is it
  practically useful? Are the concepts familiar or
  new? Does it need re-packaging?
• Links Who are the key individuals? Are there
  existing networks to use? How best to transfer
  the information? The media? Campaigns?

43
Methods
Steps 3 and 4 can be done through a variety of
methods

• review of the literature
• interviews with key actors
• capturing the authors own experience and
• discussions at workshops.

44

• Episode Study Examples

45
Paravets in Kenya

• Professionalisation of Public Services.
• Structural Adjustment ? Collapse
• Paravet projects emerge.
• ITDG projects.
• Privatisation.
• ITDG Paravet network.
• Rapid spread in North.
• KVB letter (January 1998).
• Multistakeholder WSs ? new policies.
• Still not approved / passed!

• 1970s
• 1980s
• 1990s
• 2000s

• Professionalisation of Public Services.
• Structural Adjustment
• Privatisation
• ITDG Paravet network and change of DVS.
• KVB letter (January 1998).
• Multistakeholder WSs ? new policies.

ITDG projects collaborative research.
Dr Kajume
46
PRSPs Political Context

• Widespread awareness of a problem with
  international development policy in late 90s
• Failure of SAPs (and Asian financial crisis)
• Mounting public pressure for debt relief
• Stagnation of Comprehensive Development Framework
  idea
• Diverging agendas (UK Poverty, US Governance)
• WB/IMF Annual General Meeting, Sept 1999

47
PRSPs Evidence

• Long-term academic research informing new focus
  on poverty, participation, ownership, aid
  effectiveness etc
• Applied policy research
• ESAF reviews
• HIPC review
• SPA Working Groups
• NGO research on debt
• Ugandas PEAP

48
PRSPs Links

• WB, IMF, SPA, Bilaterals, NGOs all involved
• Formal an informal networks
• None of the players was more than two handshakes
  away from any of the others

49

• Outcome Mapping

50
What is it?

• an integrated PME tool
• a system to think holistically strategically
  about how we intend to achieve result
• an approach that focuses on changes in the
  behaviour, relationships or actions of partners
  (as outcomes)
• a methodology that characterizes and assesses the
  programs contributions to the achievement of
  outcomes
• an approach for designing in relation to the
  broader development context but assessing within
  your sphere of influence

51
Focus On Behavioural Change
52
How can it be used?

• For a program to tell its performance story in
  outcome terms by
• articulating its goals and designing its
  activities
• designing a monitoring system for assessing
  internal performance and outcomes of partners
• setting a use-oriented evaluation plan

53
Why use it?

• Focussing on changes in partners behaviour,
  relationships, or actions allows a program to
• measure results within its sphere of influence
• obtain feedback about its efforts in order to
  improve its performance
• take credit for its contributions to the
  achievement of outcomes
• show progress towards outcomes

54
Terminology

• Outcomes changes in behaviours, relationships,
  activities and/or actions of the people, groups
  and organisations with whom we work
• Vision the broad human, social and environmental
  betterment we desire
• Mission how we intend to contribute towards the
  achievement of the vision
• Boundary partners individuals, groups and
  organisations with whom we interact directly to
  effect change
• Outcome challenges changes behaviours of the
  boundary partners as identified by the vision

55
The Three Stages
56
Intentional Study Design

57
Performance Monitoring

• Provides a framework for a continuous monitoring
  of the initiative as a tool to achieve its
  outcomes.
• The program uses progress markers, a set of
  graduated indicators of behavioural change,
  identified in the intentional design stage to
  clarify direction with its primary partners and
  to monitor outcomes

58
Evaluation Planning

• Helps identify the evaluation priorities
  assessing the strategy at greater depth than the
  performance monitoring stage

59
Main Elements

60

• RAPID Outcome Assessment

61
What is it?

• A Visual Tool
• Combines the outcome mapping concept within a
  case study episode study approach
• Systematic approach to collecting information
  about changes in behaviour of key project
  partners that contributed to the policy change
• Assessment of the contribution of the project
  (programme, strategy, etc.) to observed changes
  in behaviour and ultimately to the policy change

62
Approach

• Describe policy environment at end
• Describe policy environment at the beginning
• Identify the key policy actors
• Identify key boundary partners
• Describe boundary partner behaviour at end
• Describe boundary partner behaviour at beginning
• Describe changes in BP behaviour
• Describe changes in project (strategic/opportunist
  ic)
• Describe external influences
• Determine level of impact of changes in project
• Determine level of impact of external influences
• Check through external interviews
• Write report

63
Sources Outputs

• Literature review
• - Project background, progress, (published)
  achievements
• Participatory workshop with staff (and BP)
• Gather detailed information
• Identify issues for further investigation
• Interviews with key informants to
• Triangulate the result of the workshop,
• Fill the gaps of information
• Clarify causality
• Report Writing
• Visual and Narrative

64
ROA Terms and Definitions

• Boundary partners individuals, groups and
  organisations with whom we interact directly to
  effect change.
• Outcomes changes in behaviours, relationships,
  activities and/or actions of the people, groups
  and organisations with whom we work.
• Behaviours the way we or our boundary partners
  do or think about things.

65
The key steps of the ROA framework

• Describe the policy environment at the end of the
  project

66
The key steps of the ROA framework

• 2. Describe the policy environment at the
  beginning of the project

67
The key steps of the ROA framework
3/4. Identify key policy actors and boundary
partners (that were influential at end)
68
The key steps of the ROA framework

• 5. Describe the behaviours of the boundary
  partners that contributed to the change in the
  policy environment or policy

69
The key steps of the ROA framework

• Describe the behaviours of the boundary partners
  at the beginning of the project

70
The key steps of the ROA framework

• 7. Map the key changes in behaviour for each
  boundary partner from the start of the project

71
The key steps of the ROA framework

• 8. Map the key changes in the project including
  organisational changes, outputs and changes in
  behaviour during the same period.

72
The key steps of the ROA framework

• Map the external influences including the actions
  f strategic partners and other exogenous factors
  during the same period

73
The key steps of the ROA framework

• 10. Determine the level of impact/influence of
  the project on the changes in behaviour of the
  boundary partners

74
The key steps of the ROA framework

• 11. Determine the level of impact/influence of
  external influences on the changes in behaviour
  of the boundary partners and the project

75
The key steps of the ROA framework

• 12. Refine conclusions with in-depth interviews
  and assess the real contribution of the project
  on the policy change