What Makes Good Evaluation? Gearing Up to Report Program

1
What Makes Good Evaluation?
Gearing Up to Report Program Impact
Developed by Susan Barkman , 4-H Department,
Purdue University
2
The systematic collection and analysis of
information to determine the worth of a
curriculum, program or activity.
Alkin 1990
3
Worth can be a measure of

• The quality or satisfaction level leading to
  program improvement.
• The effectiveness as a means to demonstrate the
  impact of the program or that the targeted
  outcomes have been achieved.

4
Program Improvement Evaluation

• What did you like best (least) about the program?
• How would you improve the program?

5
Program Improvement Evaluation

• Please rate the following sessions?
• 
  Poor Fair Good
  Excellent
• Opening Session 1 2 3 4
• Applying Pesticides 1 2 3 4
• Planning Your Budget 1 2 3 4
• Gardening with Kids 1 2 3 4
• Closing Session 1 2 3 4

6
Program Impact Evaluation

• Has the program achieved its targeted outcomes?
• How have the participants changed as a result of
  this program?

7
What Programs Should Be Evaluated?

• Are participants involved in at least 2 or more
  hours of educational programming?
• Does the program have clearly stated outcome(s)?
• Do your stakeholders care about whether this
  outcome is achieved?
• Is the program addressing an issue that the
  community cares about?

8
Targeted Outcome
The change in participants you would expect to
see as a result of participating in the program.
what are the participants going to know, feel
or do different after the program
9
Targeted Outcomes

LEARNING ACTION IMPACT
Awareness Created Knowledge Gained Attitudes
Changed Skills Developed Aspirations Sparked
Behavior Changed Practice Adopted Decisions
Made Policies Changed or Adopted Social Action
Initiated
Social Impact Economic Impact Civic
Impact Environmental Impact
10
Farmers adopt new management practices
that reduce risk to water quality
(ACTION OUTCOME practice adopted)
Water quality improved (IMPACT OUTCOME
environmental)
11
Teen mothers ate healthier and did not consume
alcohol, tobacco and other drugs (ACTION OUTCOME
behavior changed)
Babies live IMPACT OUTCOME social
Babies have to stay less days in hospital IMPACT
OUTCOME economic
12
You Be the Judge?

• Master Gardeners will increase their knowledge
  about garden management.
• Increase the number of schools that offer school
  enrichment programs.
• Adults will develop a more positive attitude
  toward foods that contain products derived from
  biotechnology.

13
You Be the Judge?

• Adults will adopt practices that will decrease
  the amount of drift when applying pesticides.
• The educator will create an awareness and
  knowledge of gang related symbols.
• Farmers will improve their farm financial
  management decisions.

14
You Be the Judge?

• Parents will use positive parenting techniques.
• Youth will improve their classroom behavior.
• Increase involvement of citizens in community
  affairs.

15
Evaluation Methodology(when and how you are
going to administer)

• Post-Test only
• Pre and Post-Test

16
Evaluation Methodology(when and how you are
going to administer)

• Post/Post Test
• Post-Retro Pre

17
Measuring Gains in Knowledge

• True-False
• Matching
• Multiple-Choice
• Completion

18
Measuring Attitudinal Change
BELIEF Genetically-modified foods are inferior
to foods from plants developed using conventional
methods. FEELING I like to do science
experiments.
19
Measuring Attitudinal Change
BEHAVIOR INTENTION or PREFERENCE Would I would
serve genetically-modified foods to my
family. Should The government should ban
production of genetically-modified foods.
20
Measuring Skills and Behavior Changes

• Pre/post or post/retro pre questionnaire
• Observe the participants

21
Measuring Skills Developed

• How comfortable do you feel about your ability to
  lead a small group activity.
• Very uncomfortable
• Somewhat uncomfortable
• Somewhat comfortable
• Very comfortable

22
Measuring Skills Developed

• Please circle the letter that best describes your
  ability to do the following things
• 1. Finding someones email address on the web.
• 2. Using HTML to design a simple web page.
• 3. Adding motion and sound to your web page.

Cannot Much help Maybe
Little help Can Do It
needed
needed Do It
23
Measuring Skills Developed

• Please rate your skills after the training and
  what you thought they were before the training
•  
•   BEFORE
  AFTER
• TRAINING
  TRAINING
• 
  
  None A lot None A lot
• Ability to design a 1 2 3
  4 5 1 2 3 4 5
• quantitative instrument
•  
• Ability to conduct a
• pilot test 1
  2 3 4 5 1 2 3 4 5

24
Measuring Behavior

• The following statements describe how you might
  communicate with others in your everyday life.
  Place an X in the box that best describe how
  often you have done what is described over the
  past month.

Never Rarely Sometimes Often Always
I find it easy to get my point across. I use
body language to help get my message across. I
use my tone of voice to reinforce what I am
trying to say.
25
Measuring Behavior Changes

• Behavior 1st
  Rating 2nd Rating

• Stays with task until completed.
• Participates in class discussion.
• Cooperates on tasks.
• Follows directions.

1 never, 2 occasionally, 3 50 of time,
4 frequently, 5 never
26
Measuring Intentions to Act
Do you plan to soil test for phosphorus?
Yes No
As a result of this program, how willing are you
to soil test for phosphorus this coming
season? Not willing Reluctant
Somewhat willing Quite willing Very
willing
27
Open-ended Questions

• Advantage reveals what is on the participants
  mind, instead of what the evaluator thinks is on
  their mind
• Disadvantage takes time to analyze data and
  participants dont always take time to fill out

Not very effective with youth
28
Open-ended Questions

• In the next 3 months, what is one thing you will
  do as a result of participating in this training?
• Name one or more ways you learned to economize
  your finances?
• What is one positive ways you can express your
  emotions?
• Describe any changes in your childs behavior
  that has occurred since he/she has been
  participating in the ____ program.

29
Hopefully this brief session has helped you to
GEAR UP to design good evaluation tools that
will yield valuable impact data.
For more info or help in designing
evaluation instruments
contact Susan Barkman,
4-H Department, Purdue University