Four key tasks in impact assessment of complex interventions

1
Four key tasks in impact assessment of complex
interventions
Professor Patricia Rogers Royal Melbourne
Institute of Technology, Australia

• Rethinking Impact Understanding theComplexity
  of Poverty and Change
• 26 - 28 March 2008Cali, Colombia

2
Different types of impact assessment may need
different approaches

• Purpose
• Knowledge building for replication and upscaling
• Knowledge building for learning and improvement
• Accountability
• Timing
• Ex-ante
• Built into implementation
• Retrospective

3
Different types of intervention may need
different approaches to impact assessment

• Simple interventions that can be tightly
  specified and standardized intervention (e.g.
  product, technique)
• Complicated interventions that are part of a
  larger multi-component intervention
• Complex, emergent program or policy (e.g.
  community development, natural resources
  management, emergency situation)

4
Four key tasks in impact assessment

• Decide impacts to be included in assessment
  (conceptualisation of valued impacts)
• B) Gather evidence of impacts
• (description of actual impacts)
• C) Analyse causal attribution or contribution
• D) Report synthesis of impact evaluation

5
A. Decide impacts to include.

• Identifying and negotiating impacts to include
• Not only stated objectives also unintended
  outcomes (positive and negative)
• Values about good and bad impacts and about the
  distribution of impacts
• Prioritising information needs
• Adequate consultation and legitimisation
• Some approaches
• Program theory (impact pathway) - possibly
  developing multiple models of the program, eg
  Soft Systems
• Participatory approaches to values clarification
  eg Most Significant Change

6
B. Gather evidence of impacts.

• Balancing accuracy, utility, feasibility and
  ethics
• Dealing with time lags before impacts are evident
• Avoiding accidental or systematic distortion of
  level of impacts
• Evidence that is sufficiently comprehensive
• Making use of existing data as well as additional
  data
• Some approaches
• Program theory (impact pathway) identify
  short-term results that can indicate longer-term
  impacts
• Participatory approaches engaging community in
  evidence gathering to increase reach and
  engagement
• Real world evaluation mixed methods,
  triangulation, making maximum use of existing
  data, strategic sampling, rapid data collection
  methods

7
Analyse causal contribution or attribution

• Avoiding false negatives and false positives
• Systematic search for disconfirming evidence
• Distinguish between theory failure and
  implementation failure
• Understanding context implementation
  environment, participant characteristics and
  other interventions
• Some approaches
• Addressing through design
• eg experimental and quasi-experimental designs
• Addressing through data collection
• eg Beneficiary Assessment
• Addressing through iterative analysis and
  collection
• eg Contribution Analysis, Multiple Levels and
  Lines of Evidence (MLLE), List of Possible Causes
  (LOPC) and General Elimination Methodology (GEM)

8
D. Report synthesis

• Providing useful information to intended users
• Balancing overall pattern and detail
• Assisting uptake/translation of evidence
• Some approaches
• Layered reports (1 page, 5 pages, 25 pages)
• Scenarios showing different outcomes in different
  contexts
• Workshopping report to support knowledge
  translation

9
Following a Recipe A Rocket to the Moon
Raising a Child
Complicated
Complex
Simple

• Formulae are critical and necessary
• Sending one rocket increases assurance that next
  will be ok
• High level of expertise in many specialized
  fields coordination
• Rockets similar in critical ways
  
• High degree of certainty of outcome

• Formulae have only a limited application
• Raising one child gives no assurance of success
  with the next
• Expertise can help but is not sufficient
  relationships are key
• Every child is unique
• Uncertainty of outcome remains

• The recipe is essential
• Recipes are tested to assure replicability of
  later efforts
• No particular expertise knowing how to cook
  increases success
• Recipes produce standard products
• Certainty of same results every time

(Diagram from Zimmerman 2003)
10
(No Transcript)
11
Types of interventions
12
(No Transcript)