An Approach to Evaluation for Smaller and Grassroots Organizations

1
An Approach to Evaluationfor Smaller and
Grassroots Organizations
Level Best

• By Marcia Festen and Marianne Philbin
• With special thanks to the
• Pratt Richards Group
• for additional slides and content

2
Learning Objectives

• Become more familiar with basic principles of
  evaluation
• Learn an approach that will make ongoing
  evaluation easier

3
Our Discussion Today

• Based on the premises, principles and steps
  outlined in Level Best
• 5-Step Framework
• Worksheet

4
The Challenge

• It is easy to tell if you are succeeding
• in business you make money.
• In philanthropy, measuring performance
• can be fiendishly tricky and take a lot
  longer.
• Warren Buffet
• The New Powers In Giving
• The Economist, June 2006

5
Why Evaluate Your Program?

• To learn, plan and improve. With limited
  resources, you need to be as strategic as
  possible
• To better articulate the progress youre making
  toward your goals

6
But First
Pick a card
7
Your Experience

• Whats been the best and worst youve
  experienced relative to undertaking evaluation?
• Your assumptions, anxieties, realities?

8
A Few Basic Principles

• Evaluation becomes less daunting when you know
  that

9
Its Not Research

• Monitoring measurement, on an ongoing and
  regular basis, of program implementation, or
  results of a service.
• Research the systematic process of collecting
  data in a controlled environment in order to
  prove or disprove a hypothesis.

10
Definition of Evaluation

• Evaluation is the systematic process of asking
  questions, and then collecting information to
  help answer those questions, in order to improve
  the work of your organization, and often to
• tell the story of change.

11
Shift your thinking

• from prove to improve
• from judge to learn

12
from after to before

• Rather than thinking of evaluation as the test
  that follows the work, begin to think of
  evaluation as the measures you put in place
  beforehand to help you run your programs.
• How will you evaluate the work youre planning
  for this year, or next year?

13
Many methods, much jargon

• Process evaluation, outcome evaluation,
  summative, formative, participatory, Logic Model,
  quasi-experimental design etc.

14
Just start by asking

• What do we want to know?
• which is generally
• ARE WE MAKING PROGRESS TOWARDS OUR GOALS?

15
Quick Exercise

• I wish I knew whether
• _______________________ .

16
Level Best Evaluation Steps

• 1. Planning why/what you will evaluate
• Asking one-two key questions you want
  answered
• Tracking the activities that you conduct,
  the signs that youre making progress
  towards your goals
• Learning. from what you track and what it
  tells you
• Using. the insight you gain to shape
  your next program

17
Limit it

• Rolling evaluation our term for an evaluation
  process wherein smaller organizations choose one
  or two questions to ask, or areas per year to
  evaluate, and build evaluation learning year by
  year, over time.

18
For Example

• Although we are running 3 programs as part of
  our Leadership Development project, this year we
  will focus evaluation on our Parent Training
  Program.
• The purpose of this particular evaluation is
  to help us determine whether or not to invest
  more in 2007 in this particular program.

19
Step 1 Planning

• Identify a project or area youd like to
  evaluate. It can be.
• a priority, an idea, a particular program, a
  strategy within a program, an aspect of your
  process
• Worksheet

20
Avoid the BIGGEST Mistake

• Know what you want to evaluate?
• First step be absolutely clear on its GOAL.
• You cannot evaluate something for which no goal
  has been set !

21
You cant just stroll and gather

• It is a common misperception to assume that there
  is something simply there that will become
  apparent if you point the camera and shoot, or
  go looking in a general way.

22
How the Goal Shapes the Evaluation

• Example
• The Kids Will Keep Their Rooms Clean

23
For Every Program

• GOAL or DESIRED CHANGE
• Rationale
• SIGNS OF SUCCESS
• Well agree weve made progress towards this
  goal IF
• STEPS to get us there

24
Worksheet

• Fill in as we go
• Take a moment now to work together to fill in Box
  B, C, D
• (for now, skip A !)

25
SO---Remind Everyone of the PROJECTS GOAL

• What is the desired overall outcome for this
  project?
• Write down the goal statement in Box C
• Worksheet

26
Why evaluate this now?

• Box D. Write down one purpose you could imagine
  having for the particular evaluation you might
  conduct.
• Worksheet

27
Must-Have 1

• Agreement as to the CONDITION that your
  organization or program is addressing (is your
  youth theater program trying to change kids, or
  change local boring theater?). Clarity as to the
  CHANGE you hope to make.
• The question you ask will change depending upon
  the condition you see yourself addressing.
• Worksheet

28
Identifying the condition your program
addresses

• For example

29
Must-have 2

• A clear GOAL STATEMENT for any program or area
  you want to evaluate goal statement typically
  refers to the effect you want to have on the
  condition.

30
Step 2 Asking

• Once you determine the purpose of the evaluation,
  brainstorm with your team three or four questions
  youd love to be able to answer better.
• Select one or two to explore in this evaluation.
  Put the others on a calendar for next year or
  the following year.

31
Types of Evaluation Questions

• In Process Evaluation the guiding questions focus
  on the quality of a programs components or
  implementation.
• In Outcome Evaluation the guiding questions focus
  on the extent to which a program is achieving its
  desired results.

32
You
You do work. When you evaluate how well you do
what you do, its called a PROCESS EVALUATION.
33
Them

• Your work has results.
• It influences what people do or believe.
• When you evaluate what they do, its called an
  OUTCOME EVALUTION.

34
Impact

• Lots of work produces multiple outcomes over
  time.
• This equals IMPACT

35
Theory of Change

• The IF part of the statement often refers to
  what your organization will do (process)
• the THEN refers to what you hope will happen
  as a result (outcome). In some circles, this is
  referred to as your Theory of Change.

36
So.. are we? / did they?

• IF we offer reading tutors, THEN more people will
  be able to read.
• IF we educate teens effectively about sexually
  transmitted diseases, THEN fewer will engage in
  unsafe practices.

37
Theory of Change
THEN
IF we.
Which will lead to
Community or Social Change
Strategies
Immediate Outcomes
Intermediate Outcomes
Intended IMPACT (long-term)
38
Whats Most Important

• not whether youre doing process or outcome
  evaluation
• simply
• What do you want to know?

39
All flows from The Question

• You must ask a specific question what you track,
  consider and learn is dependent on the question
  you ask.

40
You Can Ask/Evaluate Anything

• Are teachers benefitting from the training we
  offer them? (How do we know?)
• Do visitors like our exhibits? (How do we
  know?)
• Are we serving girls as well as we serve boys
• in this program? (How do we know?)

41
But you must ask something!

• Evaluation is NOT about collecting massive
  amounts of information, and then attempting to
  sort through it later to see what it says

42
Step 3 Tracking

• List two or three things that if they occurred,
  would be signs of progress, examples that might
  show youre
• making progress towards the desired
  outcome for the project
• or
• that provide information that would help you
  answer the question youve posed.

43
Indicators

• What information would tell you if youre making
  progress toward your desired outcome?
• What would you track to help you answer the
  evaluation question you posed?
• Thats an indicator

44
Program Example

• Desired Outcome Our theaters main stage
  audience engagement increases
• How Well Know Were Making Progress/ 1
  Mainstage attendance as a percent of total seat
  capacity will increase to 65 for the 2010-11
  season.
• How Well Know Were Making Progress/ 2 90 of
  mainstage audience members completing the end of
  season survey will be very or somewhat
  satisfied with the season.

45
Organizational Example

• Desired Outcome Board engagement improves
• How we'll know we're making progress / 1 80 of
  Board members will contribute 500
• or more in FY2011.
• How we'll know we're making progress / 2 Each
  quarterly Board meeting will have 75 attendance
  or better in FY2011.

46
Golden Rule

• In evaluation we want to measure
• what we can control.
• So

47
Measure What Is in YOUR Control
Not everything that you want to achieve in your
program will be measurable.
48
Track Whats Closest to What YOU do

• The goal of our tutoring program is to help
  students
• --someday get jobs
• --graduate from high school
• -- better homework completion rate
• -- attend homework helper
  program
• -- be referred to this program by
  teacher
• Keep drilling down, ask what needs to happen
  before that can happen? You cant track it all.

49
S.M.A.R.T. Indicators

• Specific
• Measurable
• Attainable
• Relevant
• Time bound

50
Which Indicators to Track

• Refer back to the CONDITION youre hoping to
  change or address in the first place

51
Use What You Have

• Think creatively how might you be able to tell
  that you are achieving your desired results?
• Chances are, youre already evaluating on some
  level. What kinds of information are you already
  routinely gathering?
• There are no laws there is no one right way.
  There is no evaluation police

52
Peer Feedback

• Volunteers Share your question
• Colleagues What might your colleague track in
  order to answer that question?

53
Dont overdo it

• Most of the time, what funders and nonprofits
  really want to know is if an intervention can
  have positive outcomes given the right
  conditions, and if the results are worth the
  investment-- and they only need to know these
  answers beyond a reasonable doubt. Usually,
  this doesnt require a great deal of time or
  money. It does, however, require being very clear
  about what you want to know, and why you want to
  know it.
• ---Report from Gill Foundation

54
Sources of Information

• Agency Records
• Questionnaires/Surveys
• Interviews
• Focus Groups
• Direct Observation

55
Who Does It?

• Another good reason to have an Evaluation
  Working Group is to help think through the
  evaluation implementation.
• Who is going to
• Collect data
• Enter it
• Compile it for review?

56
Step 4 Learning

• Instead of thinking in terms of data analysis,
  helpful to think about data interpretation.
• You are looking for patterns and trends
• You are attempting to find correlation and not
  causation.

57
What did you learn?

• Do your findings speak
• for themselves?

58
What does our data tell us?

• How successful were we in reaching our desired
  outcomes/goals?
• What might we do next year to improve our
  programs/services?
• How might these improvements lead to greater
  impact in the future?

59
Challenges

• And if you dont like the results?

60
Step 5 Using

• Plug your new knowledge back into the cycle!

61
The Evaluation Cycle
62
Use Evaluation Results to

• Make Your Case
• Plan Your Next Move
• Gain Perspective
• Identify Resources Needed

63
Do NOT Use Results to

• Punish staff (though evaluation can identify
  staff/management issues that need to be
  addressed)
• Distract from other issues or cover up a problem
• Spend over budget
• Overconfidently dismiss alternate views or
  approaches.
• Prove to the world how good you are (While its
  good to toot your horn and show your results,
  dont do it at the expense of learning how to be
  even better.)

64
Where to Use Evaluation Findings

• Show and Tell Moments
• Proposals/Report to funders
• Annual report
• Board meetings

• Decision-Making Moments
• Staff meetings
• Board meetings
• Annual planning
• Strategic planning
• Budget planning

65
Rolling Evaluation

• Dont forget! The purpose of rolling evaluation
  is to isolate one or two key questions to ask PER
  year. So when you have successfully completed
  one cycle, begin again by revisiting some of the
  questions you shelved last year.

66
Level Best Evaluation Steps

• 1. Planning identifying your evaluation
  goals
• Asking one-two key questions you want
  answered
• Tracking the activities that you conduct,
  the signs that youre making progress
  towards your goals
• Learning. from what you track and what it
  tells you
• Using. the insight you gain to shape
  your next program