An introduction to doing outcomes focused evaluations

1
Evaluation Summer School 2009
An introduction to doing outcomes focused
evaluations
2
What will the workshop cover?

• The difference between outputs and outcomes
• Setting outcomes for service users
• Identifying appropriate qualitative and
  quantitative outcome indicators
• Selecting a range of suitable sources of
  evidence, and identifying when and how to collect
  the evidence
• Writing an evaluation plan

3
Back to basics What is evaluation?

• A way of assessing whetherobjectives havebeen
  achieved
• A way of assessing theoutcomes of services
  forand with people whouse them
• A way of learning from successesand mistakes

4

• What are outcomes?

5
The jargon!Inputs, outputs, outcomes

• Inputs The resources used to create the
  services offered money, time, staff, premises
• Outputs The services and products made
  available to service users groups, sessions,
  activities
• Outcomes The changes, benefits, learning etc.
  that happen or are expected to happen as a result
  of our activity

6
What is an outcome?
Something that happens orchanges in service
users lives.
An end point of any intervention. It is a
result.

• Outcomes need to be
• About changes for people, not just about the
  servicethat has been delivered
• About the effects of what wehave done
• Shared by stakeholders

7
Example

• Reporting on outcomes

Within three months of starting the
programme, 65 of parents reported considerable
or greatimprovement in relations with their
children. The childrens school
attendanceimproved in 45 of cases.
8
Why focus on outcomes?

• Changing things for the betteris the purpose of
  our efforts
• Helps us stay focused onpositive change
  forservice users
• Helps to keep us accountableto service users,
  fundersand society

9
Activity inputs, outputs and outcomes

• Choose which statements are inputs, which are
  outputs, and which are outcomes

10

• Planning evaluations

11
Evaluation planningKey questions

• What is the evaluation for?
• Who is it for?
• What are the mainevaluation questions?
• Who should be involved?
• Who will do the work?
• What is the timescale?
• What ethical issues are there?
• What is the end product?
• How will you use your findings?

12
What is the evaluation for?

• Are you
• Measuring whether youveachieved what you set
  out to?
• Meeting the requirements ofyour funders?
• Convincing others that yourapproach is
  effective?
• Publicising your work?
• Sharing what you havelearnt with others?

13
Who is the evaluation for?

• Who is the intended audience?
• What will they wantto know?
• How will you tell them?

14
Evaluation questions
Outcome questions Whether your service has
brought positive change to your users
Effectiveness evaluation
Process questions Whether you are providing the
right quality services to the right people
Efficiency orProcess evaluation
Youll probably need both!
15
Stakeholder involvement in evaluation
Stakeholders can be involved as

• Commissioners
• Participants
• Planners
• Co-evaluators
• Feedback to peers
• Assessing implicationsof evaluation

16
Think about

• Who will take overallresponsibility to
  ensurethat it gets done?
• How will tasks beshared out?
• What is the timescale?
• What sort of end productdo you want?
• How will you use the findings?

17

• Setting outcomes

18
Stages in outcome setting
Stage 1Identify the overall aim from the
relevant outcomes framework
Stage 2Identify the intended long-term outcomes
ofyour work for service users
Stage 3Plan SMART outcomes which contribute to
these long-term outcomes
Stage 4Identify the types of services or
interventions that will help you achieve these
outcomes
19
Outcomes frameworks for children, for example
England Every Child Matters
Scotland Getting it Right for Every Child

• Be healthy
• Stay safe
• Enjoy and achieve
• Make a positive contribution
• Have economicwell-being

• Safe
• Nurtured
• Healthy
• Achieving
• Active
• Respected and responsible
• Included

20
Outcomes frameworks for adults, for example
Scotland National Outcomes Framework for
Community Care
England Independence, Well-being and Choice

• Improved health
• Improved quality of life
• Making a positive contribution
• Exercise of choice and control
• Freedom from discrimination and harassment
• Economic well-being
• Personal dignity

• Improved health
• Improved well-being
• Improved social inclusion
• Improved independence and responsibility

21
Long term outcomes
22
Think SMART

• Specific
• Measurable
• Achievable
• Realistic
• Time-based

23
Examples of SMART outcomes

• Children are receiving a nutritious diet
• Adults are able to achieve and maintain a healthy
  weight
• Children have reduced/safer consumption of
  controlled substances

24
For example
25

• Measuring outcomes

26
How do you know if you have achieved service user
outcomes?

• Develop outcome indicators
• Identify sources of evidence to measure these
  indicators
• Choose methods and tools for collecting this
  evidence

27
Outcomes arent always easy to measure
Some challenges tomeasuring outcomes

• Deciding what is causeand what is effect
• Measuring long-termoutcomes
• Maturation effects
• Unintended outcomes

28
Identifying indicators
We need to ask What would change for service
users if we were successful?
29
Identifying indicators cont

• Indicators help us to demonstrate the achievement
  of outcomes
• They can be qualitative or quantitative
• Remember, you need to be able to measure these
  indicators so BE REALISTIC and think carefully
  about what can be done within your services
  resources

30
Activity developing indicators

• Read the case study
• Think about the indicators you could use to
  demonstrate that the service is achieving its
  intended outcomes
• List the indicators you could develop
• Identify where, how and when you could collect
  this information

31
Measuring indicators

• A range or data or information can be collected
  to demonstrate or measure the achievement of
  outcomes, for example
• Existing scales and tools
• Questionnaires
• Individual orgroup interviews
• Creative methods
• Observation
• Practice tools
• Design your own?

32
Choosing information sources

• When choosing information sources, you need to
  think about

• What information is most relevant?
• Are some kinds of information more reliable than
  others?
• Is some easier to collect than others?
• Can evidence collection and practice go together,
  or are additional resources needed?

33
The Goldilocks principle

• Collecting information
• costs time and money.
• We should not

• Collect too frequently
• Collect information we have
• no use for
• Collect more than we need

The quantity, quality and timing of information
should be just right
34
Standardised tools

• There are a number of standardised tools
  available which can also help you to measure
  outcome indicators, for example
• Strengths and Difficulties Questionnaire
• Eyberg Child Behaviour Inventory
• Parenting Daily Hassles Scale
• Adult Wellbeing Scale
• Adolescent Wellbeing Scale
• Rosenberg Self-Esteem Scale
• See handout for further information

35
Standardised tools
Disadvantages
Advantages

• Can have increased reliability and validity
• Reduces the need to spend time developing your
  own tools
• Makes it easier to compare evaluation
  information across services/ organisations

• There may not a tool that is relevant to your
  service or the outcomes you wish to measure
• Many tools are expensive to purchase

36
Interviews
Useful when

• You need to know aboutpeoples experiencesor
  views in depth
• You are able to rely on afairly small numberof
  respondents
• The issue is sensitive andpeople may not
  speakfreely in groups
• Respondents dont expressthemselves well in
  writing

37
Interviews
Disadvantages
Advantages

• In-depth data
• Reasonable response rate
• Does not rely on written communication

• Costly and time consuming
• Large amount of data can make analysis
  complex
• Trained interviewers needed
• Some may find one-to-one intimidating

38
Group interviews
Useful when

• You need in-depth information about how people
  think about an issue
• Respondents are more confident talking in groups
• You have existing well-functioning groups as part
  of service delivery
• You want ideas about what could be better

39
Group interviews
Disadvantages
Advantages

• Time-consuming to set up
• Require skilled facilitator
• Difficult for some not used to groups
• Can be intimidating
• Sensitive issues may be avoided or minority
  views may not be expressed
• Confidentiality issues

• Allow a variety of views to be expressed
• Gather views from larger sample
• Interaction between people can generate more
  ideas
• Can be enjoyable
• Can carry out group exercises

40
Questionnaires
Useful when

• You need information froma large number
  ofrespondents
• You know exactly whatdata you need
• The information you want isstraightforward and
  in astandard format
• Respondents are comfortablewith filling in forms

41
Questionnaires
Disadvantages
Advantages

• Quick and efficient
• Cheap
• Can be sent to large numbers of people
• Can be less intrusive
• Can be quantifiable
• People feel able to make critical comments

• Often poor response
• Get varying amount of information
• Can be biased if only those with strong views
  answer
• Reliant on written communication

42
And finally.
Learning from evaluation Evaluation is only
worthwhile ifit helps to develop andimprove
practice

• Put time aside to considerthe findings of
  theevaluation with your team
• Consider what changes maybe necessary for your
  servicedelivery and how you willcommunicate
  these