A range of approaches to Appraisal, monitoring and Evaluation in development work

1
A range of approaches to Appraisal, monitoring
and Evaluation in development work
What is useful? What isnt? What
approaches/tools/frameworks are appropriate in
different contexts? Demystifying some of the
evaluation terminology
2
Some context trends and changes in evaluation
over the last 30 years.. A changing field of
practiceMarco Segone (1998)
3
During these years, what essential changes have
we seen in evaluation practice?
4
(No Transcript)
5
What are the various approaches to A,M E ?

• A, M E confusing. Often lack of clarity about
  what approaches/frameworks/tools are useful..

• Briefly go through 6!
• Logical framework analysis
• Results based management
• Outcome mapping
• Participatory monitoring impact assessment
• Programme logic
• Most significant change (tool)
• What do each mean? What makes them different?
  What are the concerns/benefits of each.

6
What are all these approaches are essentially
for?
Appraisal, Monitoring evaluation is. 1)
Planning - knowing where you ( your partners
the people with whom you are working) are going?
2) Devising how will you (and they) get
there? 3) Assessing progress along your path 4)
Identifying, how will you (and they) know when
youve arrived? We do this all the time..
7
A M E is a essentially a way of dealing with
the richness/diversity of the development
experience synthesizing information into forms
we can understand.

• Not necessary for everyone to know the different
  approaches
• But is necessary for each organisation to
  consciously understand what approach (or set of
  approaches) it is using and very importantly
  to consciously evolve its own approach which is
  in line with its organisational values.

8
Logical Framework Analysis simple, common
sense approach for any design process (IFAD)

• This tool helps you
• Clear about WHAT you are trying to achieve and
  HOW this will be achieved.
• Decide how you know if you are achieving your
  objectives by ensuring you have a monitoring
  system.
• Makes explicit the conditions (and assumptions)
  outside the projects/programmes control that
  are critical for the project to succeed and
• Assess the risks for the project if problems
  arise.

9
Various steps in a LFA typically 6

• Establish the general scope of the project
• Agree on the specific planning framework,
  terminology and design process.
• Undertake the situation analysis
• Develop the project strategy (hierarchy of
  objectives, implementation arrangements and
  resources)
• Identify and analyse the assumptions and risks
  for the chosen strategies
• Develop a monitoring and evaluation framework

10
A logical framework matrix standard is 4 rows
4 columns

• The matrix summarises
• What the project should achieve, from the level
  of overall goal down to specific objectives.
• The performance questions/indicators that will be
  used to monitor progress.
• How the indicators will be monitored or how the
  data will be collected.
• The assumptions behind the logic of how
  activities will eventually contribute to the goal
• The associated risks.

11
(No Transcript)
12
Logical framework analysis - What are the main
benefits?

• Make planning easier. Neat, straightforward.
  Project teams have a clear plan on one piece of
  paper and can prioritise accordingly.
• Multilaterals, Donors and some NGOs like logical
  frameworks as they simplify complicated
  development initiatives into simple language and
  approaches. More appropriate for infrastructural
  initiatives.
• Enables many donors to work together as they can
  coordinate their goals/objectives with those of
  others.
• Indicators are set and monitoring them becomes a
  doable task.

13
Log frame analysis main criticisms

• Can lead to a rigidly controlled project design
  that becomes disconnected with the field
  realities and changing situations.
• The project/programmes can become over-simplified
  to such an extent that the matrix proves
  insufficient detail for effective ME.
• Outcomes, outputs and activities tend to become
  confused. The leads to disempowering processes of
  having to rewrite matrixes.
• There is often a lack of ownership of the
  project/programme. Very few log frames are done
  with beneficiaries. They are done on behalf of
  communities.
• The monitoring is often static. Indicators are
  often set from the beginning of the project and
  dont change.
• The monitoring and reporting information is
  written for those up the accountability chain.
  Not for those who are involved in the
  process/intervention.

14
2. Results based management

• Results based programme planning builds on the
  logical framework approach but puts more
  emphasis on achieving results.
• Approach from private sector Taken up by govts
  under pressure to show results to public.
• Results based management is different in that
• it ensures all available financial human
  resources support the planned results. Reduces
  diversions of resources (money, time and
  supplies) away from the planned results are
  minimised.
• Day to day management decisions are based on
  up-to-date data.
• 3 steps
• A clear and agreed results-chain is drawn up.
• Strategic results are set (goal or intended
  impact). These describe the longer term changes
  anticipated in peoples lives.
• Main agencies attempt to identify the elements of
  the results chain as input, output, short term
  outcome and longer term outcome and impact. They
  attempt to link these to the activities.

15
Language changes..
Often used for multi-agency intervention e.g
multi-laterals and bi-lateral (MDGs, World Bank,
AusAID etc)
16
UNICEF education programme
17
Results based management how does it help?

• Can help keep organisations aligned internally
  but also with other agencies. (e.g Joint work to
  eradicate diseases). Large UN programmes with
  Multilaterals involved. MDGs etc.
• Suitable for bureaucracies as everything is kept
  simple. Very understandable.
• Requires good management information about what
  is being achieved. So this information is
  collected and used.
• Keeps teams focused on the bigger picture, rather
  than on the minutiae of activities.

18
Main issues/criticisms of RBM

• RBM is once again based on linear planning and
  performance assessment - yet life and peoples
  situations are not linear.
• An overemphasis on targets to the detriment of
  relationships and reflective learning (Eyben).
• A model of doing development to others. Setting
  targets and expecting them to be recipients of
  aid.
• Emphasis on measuring performance which leads
  to recipients becoming deviant and secretive.
• Encourages a process of accountability upwards
  (from the government to the donor). Very little
  horizontal accountability . No accountability to
  the end user.

19
Group work (5 or 6 people per group)

• Questions related to Log frames or RBM? (post
  its)
• With each approach discuss.
• In what way would this approach (if used by your
  organisation) strengthen the values you aspire to
  in your practice?
• In what ways would this approach (if used by your
  organisation) threaten or undermine these values?

20
Outcome MappingParticipatory Monitoring
Evaluation
What systems or approaches are useful to you?
What can you take from each?
21
Outcome mapping

• Outcome mapping IDRC (Intern Dev Research
  centre in Canada).
• Very different. Focuses not on impacts or end
  results but on outcomes particularly
  changes in behaviour of people, groups and
  organisations with whom a programme is directly
  working.
• Through outcome monitoring an organisation can
  claim contributions to achievements of outcomes
  rather than claiming the achievement of impacts.

22

• Outcome Mapping
• Defines the program's outcomes as changes in the
  behaviour of direct partners
• Focuses on how programs facilitate change rather
  than how they control or cause change
• Recognizes the complexity of development
  processes together with the contexts in which
  they occur
• Looks at the logical links between interventions
  and outcomes, rather than trying to attribute
  results to any particular intervention
• Requires the involvement of program staff and
  partners throughout the planning, monitoring, and
  evaluation stages

23

• The Structure of Outcome MappingThe diagram
  illustrates the three stages of Outcome Mapping
  and the twelve steps of an Outcome Mapping design
  workshop.

• Design helps a program clarify what macro-level
  changes it would like to support and plan
  strategies it will use.
• 2) Outcome and performance monitoring provides a
  framework for monitoring the programs actions in
  support of its partners progress towards
  achievement of outcomes.
• Progress markers or graduated indicators of
  behavioural change used to monitor outcomes.
  (outcome journal). But open - whos changed, how
  have they changed? Etc.
• Strategy journal (to monitor strategies and
  activities)
• Performance journal (to monitor organisational
  practices)
• 3) Evaluation. - establish and evaluation plan.

24

• Outcome mapping how does it help?
• Focuses on relationships and how development
  agencies are changing the behaviour of partners
  (individuals, groups, and organisations) with
  whom they directly interact.
• Not based on a cause-effect framework, rather
  outcome monitoring recognises that there
  multiple, non-linear events which lead to change.
  
• No attempt to attribute change to a single
  intervention or a series of interventions.
• Requires the involvement of programme staff and
  partners throughout the planning, monitoring and
  evaluation stages.
• For an NGO working through and with partners,
  outcome mapping is useful as it enables
  organisations to actually engage with the added
  value (or not) that they are giving to their
  partners.

25
Outcome mapping Criticisms and issues

• 1. Good as far as it goes.but this tool does
  not help donors to understand changes/impacts in
  peoples lives. Often NGOs in NZ or donors
  need/want to know more (not just about boundary
  partners).
• 2. The language can be complicated for a simple
  process of mapping changes with partners.

26
Moving to systems and processes which encourage
what Chambers calls Reversals Not external
monitoring evaluation but internal
monitoring. Their perception, ideas, views. A
range of tools/processes encourage participatory
analysis.
Participatory monitoring evaluation
27

• ..Participation is essential for the
  development and fulfilment of the human
  personality.. Aristotle.
• ..If you are trying to transform a brutalised
  society into one where people can live in dignity
  and hope, you begin with the empowering of the
  most powerless. You build from the bottom up..
• - Adrienne Rich

28
It CAN flick the system over. Take for example
an evaluation process..
29
Participation from the beginning..
30

• Each participatory process different.
• Different stakeholders involved.
• Different processes/tools used.
• Important elements are that people themselves
  central to their own analysis of their problems.
  They are involved in seeking solutions to their
  concerns and to monitoring and assessing the
  results of interventions.

31
Many participatory techniques used

• Timelines often used to frame a discussion about
  what happened when, why. Can provoke a discussion
  on what different stakeholders perceive as
  significant.
• Maps often used to show location and types of
  changes in the area
• Venn diagrams show changing relationships and
  power relations between groups, institutions and
  individuals
• Flow diagrams often used to show direct and
  indirect impacts
• Photographs to depict changes through a sequence
  of images
• Matrix scoring or ranking to compare different
  groups preferences for a set of options or
  outcomes
• Oral histories and oral testimonies to track
  changes over time
• Network diagrams to how changes in the type and
  degree of contact between people and services.
• Participatory video or photography used to
  capture on film how people see and interpret what
  has happened in their lives and their
  communities.
• Direct observation
• Theatre or role-play often used as a tool to
  facilitate different groups to tell their own
  stories in their own ways.

32
Participatory monitoring and evaluation main
benefits

• Done well, it can really enable an empowering
  process. People themselves can take control of
  own development.
• Facilitated well. It can open a dialogue on power
  and create space for honest discussion about what
  is happening in peoples lives.
• Opens up the space for honest dialogue on
  differing perspectives of project/programme
  intervention there is no one view.
• Can lead to accountability not
• just to donors but to people in whose
• name you raise money. It is their lives,
• their ideas and their work.

33
Main criticisms/issues with Participatory
approaches.

• Can demand time from people who dont have time.
• Processes can produce challenging and
  unpredictable outcomes for donors.
• Some processes get caught up in producing
  pictures/diagrams forget the overall analysis.
• The processes can lead to the airing of
  conflicting opinions/ideas.
• Processes are often constrained by the clash with
  organisational structures and cultures and
  unequal power relationships.
• Much depends on the facilitation participatory
  processes can be done badly and processes can be
  extractive.

34
Group work (5 or 6 people per group)

• Questions related to Outcome mapping or
  Participatory M E (post its)
• With each approach discuss.
• In what way would this approach (if used by your
  organisation) strengthen the values you aspire to
  in your practice?
• In what ways would this approach (if used by your
  organisation) threaten or undermine these values?

35

• Last 2..
• Programme logic approach
• Most significant change tool

36
Programme Logic

• ½ way between dynamic (participatory) model a
  linear (static) model.
• What is programme logic?
• A simple graphic representation of your theory
  of how change will occur?
• Its a tool for building shared understanding
  of your programme
• A framework for planning, monitoring and
  evaluation
• A process for stimulating learning and critical
  reflection.

37
Steps for creating a logic model?

• No best way
• Depends on developmental stage of programme
• Existing programme start by asking What is it
  that we do? What do we hope to accomplish?Who
  are we reaching?What results are we seeking?
• New programme start by askingWhat do we expect
  to see in the long term term?What will be
  different as a result of this programme? Then
  work backwards asking what must happen to get
  there.
• Main points are to
• Determine the purpose of the logic model (who
  will use it? For what?)
• Explore and build understanding of the
  situation/context
• Explore research findings, build a knowledge
  base, investigate what others are doing/have
  done.

38
Example Parent education and support initiative
Situation
During a needs assessment, majority of parents
reported that they were having difficulty
parenting and felt stressed as a result
Outcomes
Inputs
Outputs
Parents increase knowledge of child dev
Parents identify appropriate actions to take
Develop parent ed curriculum
Staff
Improved child-parent relations
Targeted parents attend
Parents better understanding their own parenting
style
Money
Deliver series of interactivesessions
Parents use effective parenting practices
Strong families
Partners
Parents gain skills in effective parenting
practices
Research
Facilitate support groups
Assumptions
Contextual factors
39
Where does evaluation fit?

• From beginning to end

40
Parents increase knowledge of child dev
Develop parent ed curriculum
Parents identify appropriate actions to take
Staff
Targeted parents attend
Improved child-parent relations
Deliver series of interactivesessions
Money
Parents better understand their own parenting
style
Partners
Parents use effective parenting practices
Parents gain skills in effective parenting
practices
Strong families
Facilitate support groups
Research
EVALUATION What do you and others want to know
about this program?
41
Programme logic what are the main benefits?

• This framework has greater, more explicit focus
  on primary stakeholders what changes should be
  occurring in their lives?
• The process is more amenable to people-centred
  approaches to monitoring. The programme logic can
  be developed WITH primary stakeholders/partners
  and CAN easily changed. (on a wall with cards!)
• The process makes explicit the theory of change
  that you are expectingand helps you to alter the
  logic as time goes on.
• This process is more amenable to non-conceptual
  language ie. Looking for stronger families.
  Empowered communities. Can look at this in a
  variety of ways.
• Programme logic is good for all kinds of
  development work advocacy, rights based
  approaches and infrastructual development
  programmes.

42
Programme logic What are the main
criticisms/issues?

• There is a tendency to over structure this
  approach. It can become rigid and linear and
  fixed (and not involve partners or communities in
  the analysis).
• The logic focuses on expected outcomes only
  and therefore the monitoring sometimes misses out
  the unintended changes.
• This model doesnt address the question, are we
  doing the right thing?

43
Most Significant Change a tool

• A form of participatory ME. It is participatory
  because the projects main stakeholders are
  involved in deciding the sorts of changes to be
  recorded and in analysing the data.
• Essentially the process involves the collection
  of significant change stories from the field
  level and the systematic collection of the most
  important of these by panels of designated
  stakeholders or staff.

44

• MSC Steps
• Introducing MSC to a range of stakeholders and
  fostering interest in it as a process
• Significant Change stories collected from those
  directly involved.
• The stories are analysed and filtered up through
  the levels of authority. Each level of the
  hierarchy reviews a series of stories and sends
  them to level above.
• After the process has been underway for some time
  (e.g. year) a document is produced including all
  stories selected at the uppermost organisational
  level in each domain of change over a period of
  time.

45
MSC how does it help?

• Stories are a wonderfully rich way of expressing
  the changes which are happening in peoples
  lives.
• Gets away from pre-set indicators.
• Non-linear, open to the changes/outcomes which
  are seen by participants themselves in their
  lives.
• 4. A way of ordering a lot of information, to
  give an overview of some of the most significant
  changes which are happening as a result of
  interventions.

46
Main issues/concerns

• The process of feeding the information upwards,
  leads to over-emphasis on out-lyres (the best
  of the good and the worst of the bad) rather on
  the general, average changes caused by an
  intervention.
• The process of selecting stories is sometimes
  seen as culturally inappropriate.
• Most significant change stories are a compliment
  to other monitoring processes, but people often
  overemphasise MSC and forget to carry out regular
  monitoring processes.
• Generally the processes are once again about
  feeding information up a chain, rather than the
  information being owned and used by the end
  users or beneficiaries of the development
  process.

47
Final word..

• The important thing about ME is that it is used
  to open up honest reflection processes from which
  ALL stakeholders learn
• That development practice improves as a result of
  it
• That you align your organisational A, M E with
  the values of your organisation. You cant graft
  on processes.

48

• Acknowledgements
• Ann Braun
• Eleanor Taylor-Powell, University of Wisconsin
• Line drawings from Gonsalves et al (eds) 2005.
  Participatory research and development for
  sustainable agricultural and natural resource
  management.
• IDRC
• UNICEF
• Cartoons Julie Smith, World Vision, Australia.
• Jess Dart
• Outcome mapping http//www.outcomemapping.ca/resou
  rce/resource.php?id90
• www.MEnews.com

49
Group work (5 or 6 people per group)

• Questions related to Programme logic or MSC?
  (post its)
• With each approach discuss.
• In what way would this approach (if used by your
  organisation) strengthen the values you aspire to
  in your practice?
• In what ways would this approach (if used by your
  organisation) threaten or undermine these values?