Communicating Evaluation Results: Working Toward Sustainability

1
Communicating Evaluation Results Working Toward
Sustainability

• Michele Cummins
• SSRE
• Kim Netter
• NCMHPYVP

2
Overview

• Introduction
• Sustainability and Evaluation
• Evaluation Data for Sustainability
• Communications Plan
• Incorporating Evaluation Data into Communications
  Plan
• Effective Communication Strategies for Key
  Sustainability Decision-makers

3
Sustainability

• The capacity of a project to continue to deliver
  its intended benefits over a long period of time
  (World Banks definition in Bamberger and Cheema,
  1990).
• The ability of a project to deliver an
  appropriate level of benefits for an extended
  period of time after major financial, managerial,
  and technical assistance from an external donor
  is terminated (US Agency for International
  Development, 1988).

4
Evaluation

• Systematic collection of information about
  program activities, characteristics, and outcomes
  for use to reduce uncertainty, improve
  effectiveness, and make decisions.

5
Evaluation Based on What?

• You can never really say what youve
  accomplished, or whether youve accomplished
  anything at all, unless you have very specific
  quantifiable goals against which you can measure
  your effect.
• Carl Safina,
• National Audubon Society

6
What Can Evaluations Tell Us?

• Whether we are doing what we intended to do
  and/or serving those who we intended to serve
• Whether our services are appropriate
• Whether our clients are satisfied with our
  services
• Whether our services are achieving intended, and
  perhaps unintended, results
• Whether we are worth sustaining

7
Four Types of Evaluation Data
Process Outcome Impact Cost Effectiveness/Benefit
8
Process, Outcome, Impact, Cost
Effectiveness/Benefit Data

• Process
• Implementation (e.g., amount, nature,
  distribution)
• Outcome
• Short-term effects (e.g., changes in knowledge,
  attitudes, skills)
• Focus on quality and effectiveness of activities
• Impact
• Long-term effects (e.g., changes in behavior,
  system change)
• Focus on quality and effectiveness of strategies
• Cost Effectiveness/Benefit
• Relationship between program costs and program
  effectiveness or benefits

9
(No Transcript)
10
Characteristics of Evaluation Data

• Quantitative numerical (e.g., test scores,
  attendance rates, survey scales)
• Qualitative narrative (e.g., written
  descriptions, testimonials, success stories, case
  studies, focus groups, key informant interviews)
• Comparative compared to others
• Trend compared to the past
• Experimental mixture of comparative and trend
  (e.g., pretest and posttest with a
  control/comparison group)
• Sub-Group differences in effects (e.g., by
  gender, grade)
• Proxy Surrogate/substitute (used when cost,
  complexity or timeliness prevent a result from
  being measured directly)

11
Process Data for Sustainability

• Strategic Management
• Implemented on time
• Implemented as planned, or with changes that were
  based on sound reasoning
• Fiscal Management
• Implemented on budget
• Delivery
• Provided X number of services which was
  above/at/below the target level
• Served X number of people, X percent of whom
  were from the target population

12
Outcome Data for Sustainability

• Client Ratings
• Client assessments of satisfaction, utility,
  relevance, appropriateness, etc.
• Opportune use of qualitative information
• Immediate Client Changes
• Improvements in service-related knowledge,
  attitudes, skills, intended or current behavior

13
Impact Data for Sustainability

• Long-Term Client Changes
• Impact on service-related behavior
• Strongest evidence
• Not always available
• System Changes
• Evidence of enduring, system-level changes (e.g.,
  policy implementation or change)
• Often qualitative
• Not always available

14
Cost Effectiveness/Benefit Data for Sustainability

• Cost Effectiveness
• Showing that, dollar for dollar, certain
  programs, techniques, or strategies are as or
  more effective than others
• Cost Benefit
• Comparing the level of cost and associated
  benefit between programs
• Discovering whether program costs are less than,
  equal to, or greater than associated benefits
  represented monetarily

15
Communications Plan

• What are you trying to get done?
• Who can get it done for you?
• What do you have to convey?
• Who should carry the message?

16
Message Triangle
17
Incorporating Evaluation Data into Message
Triangle

• Set the Stage explain the problem
• demographics
• data
• paint the picture
• Describe the Solution and Results explain what
  you do and what youve accomplished
• program components
• program goals
• process, outcome, impact, and/or cost
  effectiveness/benefit data
• combine quantitative and qualitative when
  possible
• Detail What You Need call to action
• funding
• champions
• policy change

18
Communicating Evaluation Results Effectively

• Be honest and accurate
• Make it real
• Make it broad
• Make it clear and concise
• Layer information allow the audience to go as
  shallow or deep as they want
• Choose the appropriate method for your audience

19
Choosing Appropriate Methods
Adapted from (1) Borden, DeBord, and Snipes and
(2) Morris, Gibson, and Freeman.
20
Preferences of Policy Makers Who Make
Sustainability Decisions

• Want
• Material relevant to current issues
• Material related to real people
• Short bulleted paragraphs instead of large text
  blocks
• Charts or graphs to illustrate key points
• Brief reports/summaries or fact sheets, with more
  substantive back-up information available
• Dont Want
• Material that is too long, dense, or detailed
• Material that is too theoretical, technical, or
  jargony
• Material that is biased or not objective
• Audiotapes, audioconferences, e-mails, press
  releases

Adapted from Sorian and Baugh.
21
References

• Bamberger, M. and Cheema, S. Case studies of
  project sustainability Implications for policy
  and operations from Asian experience.
  Washington, DC The World Bank, 1990.
• Borden, L., DeBord, K., and Snipes, S. Beyond
  data. Department of Family and Consumer Sciences
  at North Carolina State University. Available
  online at http//www.ces.ncsu.edu/depts/fcs/beyond
  data/ prfig2.htm. Accessed April 23, 2004.
• Morris, L.L., Gibson, C.T., and Freeman, M.E.
  How to communicate evaluation findings. Newberry
  Park, CA Sage, 1987.
• Sorian, R. and Baugh, T. Power of information
  closing the gap between research and policy.
  Health affairs. 21(2) 264-273, 2002.
• US Agency for International Development.
  Sustainability of development programs A
  compendium of donor experience. Washington, DC
  USAID, 1988.