Evaluating Humanitarian Action EHA

1
Evaluating Humanitarian Action (EHA)

• Criteria and Frameworks

2
Criteria

• Effectiveness
• Efficiency
• Relevance/Appropriateness
• Impact
• Card exercise Define these 4 main criteria and
  suggest at least one sample generic question for
  each. You have 10 minutes

3
Effectiveness

• Effectiveness measures the extent to which an
  activity achieves its purpose, or whether this
  can be expected to happen on the basis of the
  outputs. Implicit within the criteria of
  effectiveness is timeliness.

4
Coverage (Effectiveness)

• The need to reach major population groups facing
  life-threatening risk wherever they
  are.Evaluators piloting the ALNAP Guide had
  differing opinions as to whether coverage should
  be a separate criteria, or included in
  effectiveness. For the time being it has been
  included as a separate criteria, with the links
  to effectiveness made clear in this Section.

5
Efficiency

• Efficiency measures the outputs qualitative
  and quantitative achieved as a result of
  inputs. This generally requires comparing
  alternative approaches to achieving an output, to
  see whether the most efficient approach has been
  used.

6
Relevance/Appropriateness

• Relevance is concerned with assessing whether
  the project is in line with local needs and
  priorities (as well as donor policy).
• Appropriateness is the tailoring of humanitarian
  activities to local needs, increasing ownership,
  accountability, and cost-effectiveness
  accordingly.

7
Impact

• Impact looks at the wider effects of the project
  social, economic, technical, environmental on
  individuals, gender and age-groups, communities,
  and institutions. Impacts can be intended and
  unintended, positive and negative, macro (sector)
  and micro (household).

8
Connectedness (Impact)

• Connectedness refers to the need to assure that
  activities of a short-term emergency nature are
  carried out in a context that takes longer term
  and interconnected problems into account.

9
Coherence (Impact)

• The need to assess security, developmental,
  trade and military policies as well as
  humanitarian policies, to ensure that there is
  consistency and, in particular, that all policies
  take into account humanitarian and human rights
  considerations.

10
Summary Definitions

• Effectiveness (achieving objectives - doing the
  thing right)
• Efficiency (doing it right, with as few resources
  as possible Effort, time, money, people,
  materials)
• Relevance (Appropriateness - doing the right
  thing)
• Impact (making a difference - changing the
  situation more profoundly)

11
Competing, Contradictory Criteria and Frameworks

• Tension or trades-off may exist between
  criteria, goals and principles
• Efficiency and Effectiveness, Impact and
  Sustainability?
• Time versus Quality versus Cost
• Independence and Effectiveness, Impact and
  Sustainability?
• in recovery or transitional aid programmes
• A matter of balance, wise choices and values?

12
From Criteria to HA Frameworks

• HA varies by
• Context (famine or earthquake versus war)
• Actor (specific mandate versus opportunistic)
• Time phase (emergency versus recovery)
• To evaluate a specific HA activity, we need a
  clearer definition of what effectiveness,
  relevance and impact mean in that specific
  context i.e. a baseline against which to judge

13
From Criteria to HA Frameworks

• To evaluate effect, efficiency, relevance
  and impact, we need to analyse HA
• Principles, policies, objectives, standards and
  good practices, specific to the particular
  context being examined, the actors involved and
  their aims (e.g. mission statements)

14
Types of HA Frameworks

• Legal frameworks (international and national law
  IHL IRL other Conventions, etc.)
• Organisational mandates, goals and objectives
• Non-legally binding principles (RC Code of
  Conduct GHD)
• Standards and indicators
• Statements of Good Practice

15
What is a Log-Frame (LF)?

• See ALNAP EHA Guide
• What is a log-frame and what are its uses?
• Links between log-frame logic and DAC criteria
• Creating a log-frame post-factum, in an
  evaluation
• Weaknesses of LFs
• Cartoon from IFAD website on LFs

16
Frameworks make wise choice!

• Understand context needs, actors, objectives,
  opportunities constraints (consult
  beneficiaries if possible)
• List possible frameworks, including HA actors
  own (especially if legally or administratively
  binding)
• Prioritise most relevant frameworks
• Consult stakeholders on options (be flexible -
  more than one may fit)
• Choose according to relevance, utility,
  feasibility
• State transparently which will be applied and why
• Avoid over-engineering again, ask beneficiaries
  for their framework

17
Match the framework to the HA context

• Exercise
• Imagine you are conducting an evaluation of HA
  activities in the following contexts.
• Choose a framework for each that would be most
  appropriate i.e. match the frameworks with the
  respective scenarios
• You have 10 mins.

18
SGVB Case

• Exercise
• Now using the Guinea video/case study note at
  least one EHA question or issue for each of the
  four criteria using the RC Code of Conduct as a
  framework
• You have 20 minutes
• (Answers to these and other questions could lead
  to an hypothesis about the situation, thus tying
  together findings, conclusions and
  recommendations, including lessons and
  accountability measures)

19
Possible questions for the sex-for-food abuse case

• 1. Are humanitarian needs met?
• 2. Was aid gender age-sensitive, including
  single-headed-households?
• 3. Was aid distributed for sex or cash?
• 4. Was aid controlled by vested interests?
• 5. Did staff apply community approaches to abuse
  cases?
• 6. Could better local knowledge have reduced
  abuse and inappropriate aid?
• 7. (How) Were (which) beneficiaries involved in
  programme management?
• 8. (How) Could alternatives to food and NFI aid
  have reduced vulnerability?
• 9. Were crimes committed? If so, by whom and why
  did accountability mechanisms fail (e.g. rule of
  law monitoring participation)?
• 10. Were beneficiaries treated as less than
  dignified human beings?

• 1 Humanitarian Imperative
• 2 Impartiality and need
• 3 Unconditional aid
• 4 Independence
• 5 Respect for cultures
• 6 Build on local capacities
• 7 Participation
• 8 Vulnerability Reduction
• 9 Accountability
• 10 Disaster victims as dignified humans

20
RC NGO Disaster Response Code of Conduct

• 1 Humanitarian Imperative (Relevance, Impact,
  Effectiveness)
• 2 Impartiality and need (Relevance)
• 3 Unconditional aid (Impact, Relevance)
• 4 Independence (Impact, Relevance)
• 5 Respect for cultures (Effectiveness, Impact)
• 6 Build on local capacities (Effectiveness,
  Impact, Efficiency)
• 7 Participation (Effectiveness, Relevance)
• 8 Vulnerability Reduction (Impact, Relevance)
• 9 Accountability (Impact, Relevance)
• 10 Disaster victims as dignified humans (Impact,
  Relevance)

21
Log-frames pros and cons

• Log-frames or Lock-frames?
• Advantages and disadvantages (see Des Gaspar
  paper)

22
Framework Issues

• Multiplicity
• More and more, often overlapping, even competing
  for clients
• Accountability and good practice
• Few are Mandatory (most just guidance)
• Process versus results
• Most address how, not what is achieved
  (outputs/outcomes)
• Efficiency
• Rarely covered explicitly

23
Framework Issues

• Appropriateness and Rigidity
• One-size-fits-all versus context-sensitive
  responses (e.g. Sphere and preparedness and
  recovery, which are more development than
  Emergency)
• Changing role and nature of HA actors (many no
  longer implement they play intermediary roles.
  Few frameworks are specifically for this role)
• Under0estimation of trades-off (assume it is
  possible to achieve all simultaneously)

24
Frameworks and Joint-evaluations

• Joint evaluations
• Programmes, more than projects
• Agreed frameworks?
• Multiple frameworks?
• Implications?