Practical Ideas for Extension Program Planning

1
Practical Ideas for Extension Program Planning
Evaluation

• Western Regional Training
• August 19, 2004
• Joseph L. Donaldson
• Extension Specialist

2

• Good Morning!

3
Thank you

• for turning cell phones and pagers off!

4
First Things FirstPractical Ideas for
Extension Program Planning
5
Bennett/Rockwell TOP Model
6
(No Transcript)
7
If you were asked

• What is a program?
• What is program evaluation?
• What would you say?

8
What is a program?

• An Extension program is a planned educational
  response to an identified need.
• A program is a planned series of interrelated
  educational strategies.
• These strategies are designed to help people make
  social, economic or environmental improvements.

9
Program Planning Objectives

• Solve needs assessment problems
• Write measurable outcomes
• Use a planning/evaluation model
• Measure barriers to change
• The example programs I share today are not meant
  to describe model programs or recommended
  practices. I am using these purely hypothetical
  programs to teach best practices in program
  planning that could be applied to a host of
  different Extension programs.

10
Needs AssessmentTwo Kinds of Needs

• Needs that are known focus groups, surveys,
  observation, nominal group technique and
  interviews
• Needs that are unknown data indicators,
  investigative reporting and environmental
  scanning

11
Needs AssessmentQuestions to Ask

• What problems can be solved through education?
• Is this a need or a wish?
• Who has this need?
• With whom would instruction be the most
  effective?

12
Needs AssessmentWhat about surveys?

• Great technique, but
• Results often dont differentiate
• People dont help you set priorities
• So, add responses closest to the most popular
  responses to help people distinguish

13
In our county, how critical are these
agricultural and natural resource issues?
14
how critical?
15
Needs AssessmentRanking Issues

• After 4th place, rankings are not reliable
• Only ask folks to mark the top three needs
• Only ask folks to rank the top three needs

16
Needs Assessment Big Ideas

• Needs, not wishes
• Survey a random sample
• From needs to priorities
• From priorities to outcomes
• Determine outcome indicators

17
Needs AssessmentIdentify Needs

• The County Ag and Natural Resources Advisory
  Committee assisted in conducting a needs
  assessment survey of 30 producers that showed IPM
  to be one of their top three concerns.

18
Priorities to OutcomesAn IPM Outcome

• In a follow-up survey three months after the IPM
  program, producers will adopt at least two new
  IPM practices.

19
Outcomes

• Producers will be able to identify three pest
  resistant varieties for each crop they are
  growing.
• Producers will scout for economic threshold level.

20
Outcomes to IndicatorsEvaluation Plan

• 3 month follow-up survey
• Scale
• Before did before program
• Started started since program
• Plan plan within three month
• No Plan no plans
• NA Does not apply to me

21
Evaluation PlanFollow-up IPM Survey
Below are some IPM practices. For each practice,
please circle the answer to show what is true for
you.
22
Program Planning and Evaluation Model

• Help us to work in logical ways
• Logic Models
• Online course from UW
• Helps us separate outputs from outcomes

23
Example County Battling the Musk Thistle

• Issue Example County landowners devote
  incredible time, labor and money (from 10-25 per
  acre) to control the musk thistle, an exotic
  weed. The County Ag Advisory Council has
  identified the musk thistle as a major problem
  that reduces grazing productivity. Yet, the
  Council believes that decreasing dependency on
  herbicides is an important need to protect
  streams and ground water.

24
Example County Battling the Musk Thistle
25
Program Title____________________________________
____________________ Issue
26
Example County Parenting Education
Adapted from UW-Extension. Program development
in UW-Extension Getting started in Extension
programming. University of Wisconsin Distance
Education/Digital Media Unit.
27
Example County Parenting Education
28
Example County Parenting Education
29
Example County Parenting Education
30
What if I dont know the outcomes?

• Its a common problem
• One solution is to ask participants
• Give them a checklist
• Use their answers to build outcomes and outcome
  indicators

31
Practical Ideas for Extension Program
Evaluation
32
Create A New Survey
33
Today we will

• Improve evaluation techniques
• Match response scales to outcomes
• Use two tools for post-then-pre-test
• Use pictures as an evaluation tool

34
What is evaluation?

• Think of evaluation as asking questions for the
  purpose of learning about our program so we can
  continually improve and communicate program
  outcomes to our clients and stakeholders.

35
What should I evaluate?

• Planned (60)
• Most contact hours since thats where we have the
  most impact
• Likely to be offered again
• Measurable outcomes
• Important to your stakeholders
• Base programs as needed

36
Evaluation Plans

• Watch them Observation
• Examine what they did or said Records
• Ask them Interviews or questionnaires

37
Improve Evaluation Techniques

• Observation
• Records
• Interviews/Questionnaires

38
Observation

• What are you aiming for?
• Have criteria written down
• Checklist is an observation
• Scorecard is an observation

39
EXAMPLE Checklist for Prevention of Cross
Contamination Between Meats and Ready to Eat
Foods
? Uses separately labeled or color-coded cutting
boards ? Changes disposable gloves between
tasks ? Washes hands before and after, when
working with raw foods ? Cleans and sanitizes
food contact surfaces that touch contaminated
food before they come in contact with cooked or
ready to eat food ? Stores raw or contaminated
food below cooked or ready to eat foods to
prevent them from touching or dripping fluids
40
Examine Records

• Benchmarks
• But, records are often not available
• So, make your own
• Record/diary/checklist

41
Focus Group Interviews

• 5-7 people
• Ask them what happened as a result of the program
  
• Ask follow-up questions
• Record the answers
• Data can be hard to analyze, but its rich

42
Surveys

• Match question to scale
• Dont have bunches of scales and different
  questions
• As simple as possible
• Stick to what you have to know

43
Response Scales

• Another name is answer categories
• Match the responses to what you are measuring
  (outcomes)
• Measuring behavior or practice change
• Measuring attitudes or opinions
• Measuring skill improvements

44
How do you feel about soil testing before you
fertilize?

• 50 _____ 1. Strongly Oppose
• 0 _____ 2. Oppose
• 0 _____ 3. Neutral
• 0 _____ 4. Favorable
• 50 _____ 5. Extremely Favorable
• Mean 3.0 Neutral

45
How do you feel about soil testing before you
fertilize?

• 30 _____ 1. Strongly Oppose
• 20 _____ 2. Oppose
• 0 _____ 3. Neutral
• 20 _____ 4. Favorable
• 30 _____ 5. Extremely Favorable
• Mean 3.0 Neutral

46
How do we report means?

• Five part knowledge scale
• Pre-test mean of 3.5 Post-test of 4.5
• 4.5 minus 3.5 1
• 5 divided by 1 20
• Report that there was a 20 gain in knowledge

47
Pre-Test and Post-Test

• No need to say pre-test or post-test to your
  participants
• Its time to begin. First, you will complete a
  very short exercise that will help you prepare
  for our program.
• The last part of our workshop will allow you to
  review what we did today.

48
Post-Then-Pre Common Example
Please think about your beef marketing practices
now and before this program. Please check which,
if any practices you do now (left) and then check
which, if any, practices you were doing before
this program (right). NOW BEFORE THIS
PROGRAM ____ 1. I look for ways to add value to
my cattle. ____ ____ 2. I volunteer for group
marketing. ____ ____ 3. I follow best
management practices for marketing. ____
49
How important are beef marketing alliances to you?

• Some people feel that beef marketing alliances
  are no big deal. Others think they could be a way
  to earn greater profits for beef producers in our
  county. What is true for you? Please place a B
  on one step to show how important you thought
  beef marketing alliances were before our series.
  Please place an N on the step that shows how
  important you think beef marketing alliances are
  now.

Very Important
Not Important
50
Tally Results

• Four step increase 50 people
• Three step increase 18 people
• Two step increase 10 people
• One step increase 9 people
• Same or decrease 1 person

51
Changing Attitudes for Better Beef Marketing

• Beef producers in a recent UT Extension marketing
  program completed an attitude scale that
  described how important they felt beef marketing
  alliances are for higher profits. The scale
  measured their feeling both before and after the
  program. The attitude scale ranged from no change
  to a four level change, and 97 of participants
  now have a better attitude toward alliances.
• The majority of participants (50 people 56)
  increased four levels on the marketing attitude
  scale.
• 20 increased three levels.
• 11 increased two levels.
• 10 increased one level.

52
Using Results of Open-Ended Questions

• Turn your outcome into a question
• Obtain answers
• Later, write down answer categories
• Now list the participants answers
• Put answers in categories

53
Turn your objective

• At least 70 of participants will master all
  three basic tobacco practices including planting
  black shank resistance varieties, topping before
  the first bloom and marketing tobacco into four
  grades.

54
into a Question

• Did you learn anything new about tobacco
  production from UT Extension this year?
• If yes, what?

55
Write down answer categories

• The Outcomes
• Plant black shank resistance varieties
• Top before first bloom
• Market into four grades

56
List the participants answers

• Plant SN 2105 (black shank)
• No answer
• Topping at the right time (topping)
• Stripped all tobacco into four grades (marketing)
• Distance Diagnostics is fast

57
Figure Percentages

• Black Shank 20 of 50 40
• Topping 10 of 50 20
• Marketing 10 of 50 20
• No answer 5 of 50 10
• All other 5 of 50 10

58
Use Pictures for Evaluation

• TNCEP/SPIFFY example
• Injection site example

59
Impact Statements

• Relevance response - results
• Issue - what has been done - impact
• Impact learning, actions or conditions
• Stick with a line of logic

60
Share positive evaluation results

• County Ag Committee
• Advisory Committee
• Target audience
• Set goals move to the next level
• Elected officials
• Apply for awards/dossier/share with your peers

61
What do I do with less than positive evaluation
results?

• Improve programs
• Set goals
• Set priorities
• Eliminate ineffective programs
• Share with Advisory Committee
• Share with your peers

62
Measuring Barriers to Change - A

• Not interested in ____________________
• Program did not motivate me to get started
• Have not had enough time to get started
• No money to get supplies for the job
• Found the directions confusing
• Not a handy person
• Dont believe the changes are helpful
• Other reason (please list_____________)

63
Measuring Barriers to Change - B

• In your opinion, why do more child care providers
  not have a business plan?
• What could be done here in ___________
• County to make it easier for child care
  providers to write and use a business plan?

64
Thank you!

• Joseph L. Donaldson
• 865-974-7245
• jldonaldson_at_utk.edu

65
References

• Barkman, S.J. (2002). A field guide to designing
  quantitative instruments to measure program
  impact. West Lafayette, IN Purdue Extension.
• Bennett, C. Rockwell. K. (1995). Targeting
  outcomes of programs (TOP) An integrated
  approach to planning and evaluation. Retrieved
  August 29, 2003, from University of Nebraska TOP
  Web site http//citnews.unl.edu/TOP/english/index
  .html.
• Diem, K.G. (2002, Dec.) Using research methods to
  evaluate your extension program. Journal of
  Extension, 40(6). Retrieved from
  http///www.joe.org/joe/2003december/a1.shtml.
• Diem, K.G. (2003, Feb.) Program development in a
  political world its all about impact. Journal
  of Extension, 41(1). Retrieved from
  http//www.joe.org/joe/2003february/a6.shtml.
• Ferrer, M. Jacob, S., Ferraru, T.M. (2001, Oct.)
  Two (or more) heads are better than one An
  application of group process to developing
  extension evaluation tools. Journal of Extension,
  39(5). Retrieved from http//www.joe.org/joe/2001
  october/a2.html
• Kiernan, N.E. (2001). Measuring barriers to
  change Tipsheet 27. University Park, PA Penn
  satte Cooperative Extension. Available
  http//www.extension.psu.edu/evaluation/tipsheets/
  measuring barrierstochange.pdf.

66
References

• Kiernan, N.E. (2001). How many answer categories
  are correct? Tipsheet 38. University Park, PA
  Penn State Cooperative Extension. Available at
  http//www.extension.psu.edu/evaluation/tipsheets/
  howmanyanswercategoriesarecorrect.pdf.
• Kiernan, N.E. (2002). Calculating and displaying
  findings from a stepscale Tipsheet 47
  University Park, PA Penn State Cooperative
  Extension. Available at http//www.extension.psu.
  edu/evaluation/tipsheets/calculatinganddisplayingf
  indingsfromstepscale.pdf
• Kiernan, N.E. (2002). How to use evaluation data
  to write different types of impact statements
  Tipsheet 66. University Park, PA Penn State
  Cooperative Extension. Available at
  http//www.extension.psu.edu/evaluation/tipsheets/
  howtouseevaluationdatatowritedifferenttypesofimpac
  tstatements.pdf
• Taylor-Powell, E. (2002). Program development in
  UW-Extension. Retrieved August 28, 2003, from
  University of Wisconsin-Extension-Cooperative
  Extension, Program Development and Evaluation
  Unit Web site http//www1.uwex.edu/ces/pubs/pdf
  /G3658_1.PDF
• UW-Extension. (n.d.) Program development in
  UW-Extension Getting started in Extension
  programming. University of Wisconsin Distance
  Education/Digital Media Unit.