Title: Evaluating Your Program: How Do You Know If It Worked?
1Evaluating Your ProgramHow Do You Know If It
Worked?
- Susan Donaldson
- Cooperative Extension
- Bringing the University to You
2What is the purpose of the evaluation?
- To improve a program, or your own teaching
- To help others (including a funding agency)
understand the program and its results - To guide future actions/programs
- To determine if the program is worth the cost
- To win support and/or funding
- For academic interest (!)
3PLAN THE EVALUATION AT THE SAME TIME AS YOU PLAN
THE PROGRAM!
What do I want to know, and how will I know it?
4Design your program with specific learning or
action objectives in mind
- What results do you want from your program?
- Knowledge (understanding)
- Beliefs, attitudes and opinions (psychological
states) - Behavior (What actions do you want them to take?)
5Outputs are not the same as outcomes or impacts
- Outputs are things you produce, including
brochures, other printed materials, numbers of
events, etc. - Grantors often need these numbers
- Producing a brochure does NOT guarantee learning
or outcomes!
6Some questions you might ask
- What do people do differently as a result of the
program? - Who benefits, and how?
- Are the programs accomplishments worth the
resources invested? - What are the social, economic, environmental
impacts?
7Some questions you might ask
- What are the strengths and weaknesses of the
program? - How well does the program respond to the need
that caused you to design it? - How efficiently are resources being used?
- How well does the program fit in the local
setting?
8What information do you need to answer the
questions?
- Indicators how will I know success when I see
it? - Participation
- Reactions (maintain interest, accept leadership
responsibility) - Learning (sufficient knowledge and skills
developed) - Actions (knowledge is applied plans are made
collaboration occurs) - Impact (successful weed management planning and
implementation occurs decrease in infested
acres, etc.)
9Kinds of information
- Numerical (statistical)
- Narrative
- What will the recipients consider to be credible
information? What best meets their need?
10Sources of information
- Surveys
- Observations
- Interviews
- Tests
- Group techniques
- Case studies
- Photographs
- Document review
- Testimonials
- Etc
11The Truckee Meadows example
- City and County government
- Citizens and neighborhood advisory board
members - The general public
12City and County Government
- Message We have a problem. You have a legal
responsibility. Failure to take action will
increase future costs. - Desired action Form a functional CWMA
- Hook Forming a CWMA will help get funding.
13City and County Government
- Evaluation
- PARTICIPATION
- Does CWMA form?
- Do city and county personnel participate fully?
- REACTION
- Does participation continue over time?
- LEARNING
- Is the ability to manage weeds improved?
- ACTIONS
- Do collaborations develop among entities?
- Is budget allocated to weed management?
- IMPACT
- Are weeds managed?
14What happened?
- 36 city, county, agency, and citizen
representatives attended the first meeting - Attendees agreed that a CWMA was needed
- 12 weeds of concern were prioritized for group
effort - Attendees decided to continue to meet
15What happened (outcomes)?
- Staff now attend annual trainings
- An MOU and strategic plan were developed
- Washoe County and City of Reno are collaborating
- County has hired a Natural Resources Manager
position will coordinate weed management - Grants were obtained
- Seasonal mapping staff were hired
- Control budgets are still not adequate
- Issue is easily de-prioritized
16Indicators
- PARTICIPATION
- Does CWMA form? (YES)
- Do city and county personnel participate fully?
(NO) - REACTION
- Does participation continue over time? (YES)
- LEARNING
- Is the ability to manage weeds improved?
(SOMEWHAT) - ACTIONS
- Do collaborations develop among entities? (YES)
- Is budget allocated to weed management? (NOT
ENOUGH) - IMPACT
- Are there fewer weeds? (COULD BE A LOT BETTER!)
17CAB NAB members
- Message Weeds are affecting your neighborhood,
and you can help fight them - Action Incorporate weed cleanups into
neighborhood cleanup days - Hook Well supply bags, information and help
- Reaching the audience Personal contact with
each group via a canned presentation at a
regular meeting
18CAB NAB members
- Evaluation tools
-
- Number of champions recruited (PARTICIPATION)
- Number of projects incorporating weeds (ACTIONS)
- Neighborhoods have fewer weed problems (IMPACT)
19The campaign
- Presentations were made at 12 NABs and CABs in
April and May reaching 240 people - Each group was given a binder of information
- One champion was recruited many asked for
information and assistance - Number of events one in 2004, five in 2005
20General public
- Message Weeds are ruining your neighborhood
- Action Identify, pull, and dispose of yellow
starthistle) - Hook The ouch! factor this affects YOU!
21General public
- Methods PR campaign using movies, buses,
hotline - Money NDOA grant
- Evaluation tool Number of calls to hotline
(LEARNING ACTION) - Future evaluation tools Hits on website,
reports filed on website, random phone survey
(need funding)
22What happened?
- A 10,000 grant was obtained for an education and
outreach campaign - Campaign elements included
- presentations to advisory boards
- promotion of a weed hotline
- bus placards
- slides in movie theaters
232004 PR campaign
- Washoe County donated a phone line for a hotline
- Hotline magnets distributed at public events
- Advertised via news releases and PSAs
- Volunteers agreed to transcribe messages on a
monthly basis
24Bus Placards 9 buses for one month (potential
viewership 150,000)
Cost 1566.34
25Movie Slides 3 different slides shown on half of
all Reno/Sparks screens in movie theaters (21
total) for 6 weeks
26Movie Slides Potential viewership 126,600 people
27Movie Slides Number of times repeated
8,820/slide
Cost 2370.50
28Hotline Calls Logged by Month
Number of Calls
29Did it work?
- Increase in hotline calls was small relative to
amount of PR - Calls to hotline dropped off once PR stopped (and
fall came) - Need better method to evaluate public awareness
- Campaign elements were changed for 2005