Sowing the Seeds: Program Evaluation that Works for You - PowerPoint PPT Presentation

Loading...

PPT – Sowing the Seeds: Program Evaluation that Works for You PowerPoint presentation | free to view - id: 4cbbe4-ZGViN



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

Sowing the Seeds: Program Evaluation that Works for You

Description:

Sowing the Seeds: Program Evaluation that Works for You Self-Guided Training Module Please press or click the mouse to begin. Center drawing credit: – PowerPoint PPT presentation

Number of Views:144
Avg rating:3.0/5.0
Slides: 133
Provided by: abat5
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Sowing the Seeds: Program Evaluation that Works for You


1
Sowing the Seeds Program Evaluation that Works
for You
  • Self-Guided Training Module

Please press ltEntergt or click the mouse to begin.
Center drawing credit Josh Jetson, 16, Australia
2
Introduction
Welcome to the Sowing the Seeds self-guided
training module on program evaluation.
3
Introduction
The goal of this web-based module is to increase
your capacity to evaluate your program. The
examples used in this module focus on evaluation
of youth programs but the principles of
evaluation can be applied to any program.
4
Introduction
Individuals who will find this module useful are
program directors, staff members and volunteers,
youth participants, parents, or anyone interested
in quality programming.
Please press ltEntergt or click the mouse to begin.
5
Orientation
This module is an interactive and flexible way
for you to learn about the basics of program
evaluation. It will take approximately 30
minutes for you to go through the core module.
You will also have the option of going in-depth
to learn more about some of the topics.
6
Orientation
The module contains slides with descriptions and
definitions. Also, you will have the opportunity
to test your understanding with different
activities. The module uses an example of a
youth program to illustrate the steps of
evaluation.
7
Orientation
You also can move at your own pace Just press
ltEntergt, click the mouse, or press the forward
arrow when you are ready to continue or if you
would like to move quickly through a section.
Try it now to go to the next slide.
8
Outline of Training Module
This module consists of five parts
I. Introduction to Evaluation
II. Goals, Objectives, Measures
IV. Data Analysis Reporting
V. Evaluation Steps Planning
III. Data Collection Tools
9
I. Introduction to Evaluation
So, lets get started. Why do you want to
evaluate your youth program?
10
I. Introduction to Evaluation
  • Here are some reasons for conducting an
    evaluation.
  • Program evaluation helps you
  • assess changes in participant knowledge,
    attitudes, and skills
  • secure and maintain program funding
  • improve programming and
  • justify program continuation and adoption.

Are your reasons included?
11
I. Introduction to Evaluation
  • Whatever your reasons for wanting to know more
    about program evaluation, this module will
    provide you with basic information and tools to
    think about and plan your own evaluation.

12
Types of Evaluation
  • There are two types, or parts, of evaluation

Process Evaluation
Outcome Evaluation

13
Process Evaluation
Process Evaluation
  • assesses what your program does.
  • The process evaluation has to do with what you
    teach, how many sessions you hold, your
    attendance rates, the activities you conduct,
    etc.

14
Outcome Evaluation
Outcome Evaluation
  • describes how your participants will change as a
    result of the program.
  • The outcome evaluation has to do with the
    knowledge, attitudes, and skills you anticipate
    your participants will gain.

15
Types of Evaluation
  • As you go through the module and the optional
    sections, you will learn more about how to
    conduct the process and outcome evaluations of
    your program.

16
Types of Evaluation
  • If possible, it is important to conduct both
    types of evaluation. The information you get will
    help you to match up what you actually did in the
    program (process) with how participants changed
    as a result (outcome). The combination tells you
    what worked and what didnt in your program.

17
Special Note Youth Participatory Evaluation
  • A growing number of youth serving organizations
    are working collaboratively with youth
    participants to design and conduct program
    evaluations, an approach called youth
    participatory evaluation.
  • The results can be great for the program and for
    the youth!

For more information on youth participatory
evaluation, refer to the resource list at the end
of the training module.
18
Welcome to Part II
This part introduces the goals, objectives, and
measures (optional).
I. Introduction to Evaluation
II. Goals, Objectives, Measures
IV. Data Analysis Reporting
V. Evaluation Steps Planning
III. Data Collection Tools
19
II. Goals Objectives
When you evaluate your program, you measure
whether you are running the program and meeting
those served (e.g., youth) as you had
planned. Your programs goals and objectives
organize what you plan to achieve through your
program.
20
II. Goals Objectives
Goals objectives are important because they are
the destination on your evaluation roadmap.
Without them, it is hard to know what you want to
accomplish and if you are succeeding! As you will
soon learn, goals are more broad than objectives.
Objectives are more detailed and measurable.
21
II. Goals Objectives
  • A goal is the broad and overarching purpose
    toward which your program is directed.
  • For example, the goal of the YouthWorks program
    is
  • to contribute to a culture of free speech and
    social responsibility.

22
Goals
  • What is the goal of your youth program?

23
Objectives
  • Your programs objectives are the more specific
    things you plan to accomplish.
  • For example, one objective of the YouthWorks
    program is to involve 25 youth in 10 media
    workshops over 3 months.

24
Goals Objectives-Why both?
  • You may be wondering What is the purpose of
    having goals and objectives?
  • It is important to have both because while
    objectives help you assess short-term steps or
    milestones in running your program successfully,
    a goal helps you to strive toward your ultimate
    vision for the program.
  • Keeping your goal in mind can help you to write
    new objectives as your program changes and
    improves over time!

25
II. Goals Objectives
If you would like to learn more about goals and
objectives now, click on this button. If you
would like to continue with the core module,
click on this button.
26
Objectives
  • While a goal is general, an objective should be
    measurable, achievable, and specific.
  • Remember this and you will get MAS (more) out of
    your objectives!
  • Measurable Achievable Specific

27
Objectives
  • Measurable
  • means that you can set up a practical way to
    assess your objective.
  • Achievable
  • means your objective is realistic.
  • Specific
  • means that your objective is as detailed as
    possible.

28
Objectives
  • Here is an example of an objective with MAS.
  • One of YouthWorks programs objectives is to
    involve 25 youth in 10 media workshops over 3
    months.

achievable
measurable
specific
29
Types of Objectives
  • There are two types of objectives

Process Objectives
Outcome Objectives

30
Process Objectives
Process Objectives
  • describe what your program will do.
  • A process objective has to do with what you will
    teach, how many sessions you will hold, your
    attendance rates, the activities you plan to
    conduct, etc.

31
Process Objectives
Process Objectives
  • Remember, one of YouthWorks programs objectives
    is to involve 25 youth in 10 media workshops over
    3 months. This is a process objective because it
    describes what the program will do.

32
Outcome Objectives
Outcome Objectives
  • describe how your participants will change as a
    result of the program.
  • An outcome objective has to do with the
    knowledge, attitudes, and skills you anticipate
    your participants will accomplish.

33
Outcome Objectives
Outcome Objectives
  • Another objective of the YouthWorks program is to
    improve participants cooperation skills over the
    course of the 12-week program. This is an
    outcome objective.
  • You will learn later how to make this objective
    measurable, by identifying how you will know if
    cooperation skills have improved.

34
Process or Outcome?
Can you guess whether the following objectives
are process or outcome? Guess and then click the
mouse to see the answer.
Objective Instructors will cover material from
at least 6 of the 8 lessons in the curriculum
guide during one semester.
Process Objective
35
Process or Outcome?
Process or outcome? Guess and then click the
mouse to see the answer.
Objective All 15 participants will develop 5
pictures in 3 weeks.
Outcome Objective
36
Process or Outcome?
Process or outcome? Guess and then click the
mouse to see the answer.
Objective The program will have 85 attendance
on average during the summer session.
Process Objective
37
Goals Objectives Review
  • When you are clear about the goal and process and
    outcome objectives of your program, it will be
    much easier to evaluate your program.
  • Take the next five minutes to read about the
    Better Bodies program and develop some process
    and outcome objectives for the program.

38
Goals Objectives Review
  • The Better Bodies program has as its goal, to
    promote optimum lifelong physical health through
    weight management and education among youth.
  • The program involves about 50 youth (ages 9 to 12
    years) yearly in an after school program. The
    program involves two two-hour sessions weekly
    over six months and includes lessons and hands-on
    activities focusing on nutrition education, food
    preparation, shopping skills, physical education,
    and games.

39
Goals Objectives Review
You already know the goal of the Better Bodies
program. Can you come up with 1-2 process
objectives and 1-2 outcome objectives for the
program? STOP and take a minute to think about
and write down these objectives. Save them to
look at later.
  • The program involves about 50 youth (ages 9 to 12
    years) yearly in an after school program. The
    program involves two two-hour sessions weekly
    over six months and includes lessons and hands-on
    activities focusing on nutrition education, food
    preparation, shopping skills, physical education,
    and games.

40
Goals Objectives Review
Did you come up with 1-2 process objectives and
1-2 outcome objectives for the program? Save them
to look at later.
41
II. Measures
  • Measures are the pieces of information that tell
    you if youre meeting your objectives.
  • It is easy to remember what a measure is because
    it refers to how you will size up or measure if
    you are achieving your program objectives.

42
II. Measures
  • The idea of measures can seem tricky, but we
    use this concept in our everyday life.
  • For example, suppose you plan to get healthy this
    year. How would you keep track of whether you are
    meeting this goal?

43
II. Measures
  • You might want to collect data in a couple of
    areas to chart your progress. These categories or
    measures may be
  • Number of times per week you exercise for 30 or
    more minutes
  • Number of vegetables you eat each day
  • Number of pounds you lose each week

44
II. Measures
If you would like to learn about measures now,
click on this button. If you would like to
continue with the core module, click on this
button.
45
II. Measures
  • Each measure is related to an objective.
  • For example, if a process objective of the Better
    Bodies program is to involve the participants in
    25 minutes of vigorous exercise at each session,
    then your measure for that objective would be the
    information that you record to know if youth are
    really getting opportunities for exercise. What
    would that be?
  • The number of minutes provided for the youth to
    do vigorous exercise at each session.

46
Types of Measures
  • Just like objectives, there are two types of
    measures

Process Measures
Outcome Measures

47
Process Measures
Process measures
  • are types of information that you collect about
    what your program is doing or has done.
  • A process measure is a piece of information such
    as what you taught, how many sessions you held,
    your attendance rates, the activities you
    conducted, etc.

48
Process Measures
Process Measures
  • Remember, one of YouthWorks programs process
    objectives is to involve 25 youth in 10 media
    workshops over 3 months. The process measure for
    this objective is the number of youth in
    attendance at each media workshop.

49
Outcome Measures
Outcome Measures
  • are types of information that you collect to see
    how participants have changed as a result of the
    program.
  • An outcome measure is a piece of information that
    has to do with the knowledge, attitudes, and
    skills your participants accomplish in the
    program.

50
Outcome Measures
Here are definitions and examples of typical
outcome measures.
51
Outcome Measures
Outcome Measures
  • An outcome objective of the YouthWorks program is
    to improve participants cooperation skills over
    the course of the 12-week program. The outcome
    measure for this objective is the change in
    participants cooperation skills from the
    beginning to the end of the program.

52
Outcome Measures
  • You may have several options when deciding on the
    outcome measure for a particular outcome
    objective.
  • For example, for the outcome measure, change in
    participants cooperation skills, you will need
    to decide

53
Outcome Measures
  • how to put a value on cooperation skills
  • (so you can tell if there is a change)

54
Outcome Measures
  • whether you will use a self-reported measure,
    where the participant reports back on his/her own
    skills, or whether you will have a third person
    like an instructor assess the participants
    skills and

55
Outcome Measures
  • whether you will assess the change in the
    participants skills at the beginning and at the
    end of the program and calculate the change, or
    if you will just ask about the amount of change
    made, at the end of the program.

56
A Measures Story
  • You may not realize it but the fact that you know
    what a measure is means that you know A LOT!
    Lets consider the story of one measure
  • Change in participants knowledge of musical
    genres.
  • Lets start with some questions for you

57
A Measures Story
  • Change in participants knowledge of musical
    genres.
  • What type of measure is this?
  • If you said that it is an outcome measure, then
    you are right!
  • What is the outcome objective, then?
  • There are several ways of phrasing an appropriate
    outcome objective. Yours should sound something
    like, to increase participants knowledge of
    musical genres, by, on average, 2 additional
    genres, from the beginning to the end of the
    6-week unit on music through the ages.

58
A Measures Story
  • Change in participants knowledge of musical
    genres.
  • Now, from where will the information for this
    measure come?
  • Since were interested in changes in knowledge we
    may want to ask each participant to list the
    musical genres he or she knows at the beginning,
    and again at the end, of the unit. Then, we can
    calculate how many more genres each participant
    could name at the end of the unit, divide by the
    number of participants, and see if the average is
    below or above 2 (the number we wanted to meet in
    our objective).

59
Measures Review
  • It is now time to practice your measure-writing
    skills. Stop and refer back to the process and
    outcome objectives you wrote for the Better
    Bodies program. Write down a measure for each
    objective.
  • When you are ready, move to the next slide.

60
A Final Word about Measures
  • It may be tempting to skip the stage of figuring
    out and writing down your measures but dont
    succumb to that temptation!
  • By figuring out and writing down your measures,
    you ensure you will gather only the information
    that you need (and not a lot of extra!) to
    improve your program and to see if it is working.

61
Welcome to Part III
The next part looks at data collection tools.
I. Introduction to Evaluation
II. Goals, Objectives, Measures
IV. Data Analysis Reporting
V. Evaluation Steps Planning
III. Data Collection Tools
62
IV. Data Collection Tools
  • Quite simply, you use data collection tools to
    collect data on your measures. You can use one
    collection tool to collect data on several
    measures.
  • Just like the types of evaluation, there are
    process collection tools and outcome collection
    tools.

63
Process Collection Tools
  • Process collection tools provide information on
    process measures and process objectives. Some
    examples are
  • Attendance records
  • Lesson reports
  • Sign-in sheets
  • Activity logs

64
Process Collection Tools
  • Process collection tools will provide you with
    measures on your process objectives. For
    example, consider the following

Process Objective Process Measure Collection Tool
to have at least 50 of parents attend all of the parent information nights. the number of parents in attendance at each parent information night (out of total of participants for ). sign-in sheet
65
Outcome Collection Tools
  • Outcome collection tools provide information on
    outcome measures and outcome objectives. Some
    examples are
  • Survey
  • Participant interviews
  • Focus groups
  • Observations

66
Outcome Collection Tools
  • Outcome collection tools will provide you with
    measures on your outcome objectives. For
    example, consider the following

Outcome Objective Outcome Measure Collection Tool
to improve participants attitudes about their neighborhood by one point, on average, on a scale over the 10-week program. change in participants attitudes toward neighborhood on a 5-point scale (0-dont like it at all5-like it a lot) participant survey
67
Outcome Collection Tools
  • Lets look at some outcome collection tools in
    more depth

68
Surveys

Surveys are questionnaires given to to youth
participants, instructors, caregivers, and/or
audience members.
69
Participant Interviews

Participant interviews are face-to-face
discussions with individual participants, with
the aid of a pre-determined set of
questions. Interviews are great because you can
get in-depth information from each participant
but interviews also can be time-consuming.
70
Focus Groups

Focus groups are discussions with a small group
of participants, with the aid of a pre-determined
set of questions. You can gather feedback from a
large group of youth using focus groups, as long
as you ensure that it is a safe place for
participants to share their own opinions.
71
Observations
Observations look at participants involvement in
program activities. Typically, an instructor or
outside observer will record participants
ability to perform tasks or behavior on an
observation form at different points throughout
the program duration.

72
Portfolios

Portfolios are collections of participants work
(art, photos, writing pieces, etc.) in the
program. Portfolios provide concrete examples of
participants products but may not always reflect
the effort put into the project.
73
Surveys in More Depth

Since surveys are the most common type of data
collection tool, you may choose to learn more
about them now. If you would like to learn more,
click on this button. If you would like to
continue with the core module, click on this
button.
74
Surveys

Surveys are questionnaires given to to youth
participants, instructors, caregivers, and/or
audience members. Often, the questionnaires
themselves are called surveys.
75
Surveys

Surveys may be administered before (pre) and/or
after (post) the program.
76
Surveys

Surveys are great because you can ask a lot of
questions at one time and you can compare pre and
post surveys. However, the information is all
self-reported!
77
Survey Tips

Good questionnaires are 1. To the point - They
only ask for information that will be used later
(for the evaluation, reports, etc.). 2.
Developmentally appropriate - Instructions and
questions are at the appropriate reading
level. 3. Clear - Formatting is easy to follow
and text is large. 4. Straight-Forward - They
use simple language and common terms. 5.
Well-Ordered - The questions flow in a logical
order.
78
Survey Activity

Since you will likely construct a survey at some
point, here is a little activity to get you
thinking about well-constructed surveys. On the
following slides are some example survey
questions. Think about what might be problematic
about each and then click the mouse to see if you
are right.
79
Survey Activity

Problem survey question 1 During this program,
did you enjoy the classes on nutrition and the
classes on exercise? What is problematic about
this question?
80
Survey Activity

Problem survey question 1 During this program,
did you enjoy the classes on nutrition and the
classes on exercise? This question is
problematic because you wouldnt know from the
responses whether the respondents who said yes
liked both the nutrition and exercise classes or
if they only liked one, and which one! What
would be a better way to find out which classes
the participants enjoyed?
81
Survey Activity
  • One way to solve the problem is to split the
    question into two separate questions
  • During this program, did you enjoy the classes on
    nutrition?
  • During this program, did you enjoy the classes on
    exercise?

82
Survey Activity

Problem survey question 2 a) What was your
favorite part of the field trip to the university
athletic center? b) Did you go on the field trip
to the university athletic center? c) How could
the field trip to the university athletic center
be improved? What is problematic here?
83
Survey Activity

Problem survey question 2 a) What was your
favorite part of the field trip to the university
athletic center? b) Did you go on the field trip
to the university athletic center? c) How could
the field trip to the university athletic center
be improved? This set of questions is
problematic because their order does not make
sense. What would you do differently?
84
Survey Activity

Thats right, re-order the questions so they flow
sensibly! 2a) Did you go on the field trip to
the university athletic center? 2b) What was your
favorite part of the field trip to the university
athletic center? 2c) How could the field trip to
the university athletic center be improved?
85
Survey Activity

Problem question 3 How much fun did you have in
this program? A ton A lot Some A
little A tiny bit Almost none
None What is problematic with this question?
86
Survey Activity

Problem question 3 How much fun did you have in
this program? A ton A lot Some A
little A tiny bit Almost none
None This question has too many possible
responses. What is the difference between A
tiny bit and Almost none, and does it matter?
How would you improve the categories?
87
Survey Activity

With multiple choice responses it is good to
have between 3 and 5 categories. Most
importantly, make sure they make sense! 3. How
much fun did you have in this program?
A lot Some None
88
Survey Review

Although it may seem easy to write up a quick
survey for your participants, constructing a
useful survey takes thought and skill. Keep in
mind the survey tips presented earlier and
remember to make the questions and answers as
clear and to-the-point as possible. Also,
remember to avoid survey fatigue the shorter the
survey, the higher quality of your responses,
usually!
89
Data Collection Tools Review
  • Remember, you can collect information on more
    than one measure using one tool. You can also
    collect information on one measure from more than
    one tool.

90
Welcome to Part IV
This part will looks at data analysis and
reporting.
I. Introduction to Evaluation
II. Goals, Objectives, Measures
IV. Data Analysis Reporting
V. Evaluation Steps Planning
III. Data Collection Tools
91
V. Data Analysis
  • Data analysis involves looking at the information
    you have collected and coming up with useful
    statements about your program.

92
V. Data Analysis
  • Some examples of useful statements are
  • Numbers of lessons, classes, or units taught
  • Numbers of youth participants
  • Percentage of youth with improved knowledge,
    attitudes, and skills
  • Quotes by participants, parents, and staff
  • Response of community members

93
V. Data Analysis
  • In order to analyze your data, begin by
    tabulating (counting up) the data from your
    survey, interviews, observation form, or other
    collection tool.
  • You can then come up with numbers, percentages,
    and/or quotes that reflect your measures, which
    also will tell you if youre meeting your
    objectives.

94
V. Data Analysis
  • For example, if you have a survey question that
    asks participants to report how safe they feel at
    the after school program (0-not safe5-very
    safe), you might
  • add up the number who said they feel a little or
    very safe
  • divide that number by the total number of
    responses and then
  • write a statement such as (next slide)

95
V. Data Analysis
  • 87 of participants feel safe at the Downtown
    AfterCare Program.

96
V. Data Analysis
  • The statements you come up with during data
    analysis should sound like your objectives.
  • For example, if one of your objectives is to have
    an average attendance of 85 over a 12-session
    program, then the statement you make, after
    averaging the attendance rate recorded in
    attendance logs over the 12 sessions, might be
    (next slide)

97
V. Data Analysis
  • The program achieved a 91 average attendance
    rate over 12 sessions.

98
V. Reporting
  • When reporting the information from your program
    evaluation, you can summarize your statements in
    several ways, including
  • Tables
  • Graphs
  • Quotes
  • Case studies

99
V. Reporting

If you would like to learn more about each way of
reporting, click on this button. If you would
like to continue with the core module, click on
this button.
100
V. Tables
  • Tables are useful for showing information about
    your program or the participants, such as
  • Topics covered in each of the units or themes and
    number of sessions dedicated to each.
  • Age, gender, grade, or other demographic
    information about participants

Unit I. Nutrition II. Fitness III. Body Care
Topics covered ( of sessions) -Food groups (1) -Reading nutrition facts labels (2) -Benefits of fruits and vegetables (2) -Role of heart in health (1) -Heart rate (1) -Exercises to do at home (2) -Basic body physiology (1) -Importance of hygiene (1) -Tools for good body care (1)
101
V. Graphs
  • There are several types of graphs appropriate for
    reporting evaluation information. The most
    common and useful graphs are line graphs, bar
    graphs, and pie graphs.

102
V. Graphs
  • Line graphs are useful for showing changes in
    attendance, participation, etc. over time.

103
V. Graphs
  • Bar graphs are useful for comparing two or more
    groups, such as changes in outcomes for those
    with lower versus higher class attendance.

104
V. Graphs
  • Pie graphs are useful for showing parts of a
    whole, such as the percentage of participants by
    gender and age.

105
V. Quotes and Case Studies
  • Quotes are useful to support and explain points
    made from the data.
  • Quotes from youth may be collected on survey
    questionnaires, in interviews, or in focus
    groups.
  • Quotes from parents and community members may be
    collected at presentations or through evaluation
    forms.

106
V. Quotes and Case Studies
  • Case studies describe one participants story in
    the program. You may write a case study about an
    exemplar participant or a participant who has
    improved during the program.

107
Report Structure
  • Report Structure
  • Executive Summary
  •  I. Introduction
  • 1.    Program description
  • 2.    Evaluation purpose and overview
  • 3.    Report layout
  •  II. Evaluation Methodology
  • 1.    Goal objectives
  • 2.    Description of methods used
  •  III. Process Evaluation Findings
  •  IV. Outcome Evaluation Findings
  •  V. Conclusions and Recommendations
  •   Appendix

Written reports are an important way to share
what you found in your evaluation with funding
agencies, supporters, and board members. On the
right is a sample report structure outline.
108
Welcome to Part V
The final part focuses on evaluation steps.
I. Introduction to Evaluation
II. Goals, Objectives, Measures
IV. Data Analysis Reporting
V. Evaluation Steps Planning
III. Data Collection Tools
109
VI. Evaluation Steps Planning
  • A review of this training module provides the
    steps for your program evaluation.

110
VI. Evaluation Steps Planning
  • Before you begin your program
  • Establish your goal.
  • Establish your process and outcome objectives.
  • Decide on process and outcome measures.
  • Determine methods for collecting data.
  • Create forms for collecting data.

Involve program staff and youth. Construct an
evaluation timeline.
111
VI. Evaluation Steps Planning
  • During the program
  • 6. Collect data using pre-determined methods
    forms.
  • 7. Tabulate preliminary data.

Administer a pre-survey to track change.
112
VI. Evaluation Steps Planning
  • At the conclusion of the program
  • 8. Tabulate all data.
  • Decide on reporting modes.
  • Prepare, review, and present evaluation findings
    and use the information for program improvement.

113
VI. Evaluation Steps Planning
As you follow the evaluation steps, it may be
helpful to record your program objectives,
measures, methods, and reporting techniques in a
quick-reference table like the one on the next
slide.
114
Program Evaluation Planning Table
115
Congratulations!
  • You have completed this training module in
    program evaluation. With the knowledge you have
    gained you can begin thinking about and even
    planning your program evaluation. Refer back to
    slides in this presentation for help as you plan
    and carry out your evaluation.

116
Congratulations!
  • The final slides provide some additional guidance
    and resources on program evaluation, including
  • Tips for preparing reports and presentations for
    key audiences.
  • Glossary of terms
  • Resource list
  • Frequently Asked Questions about program
    evaluation and responses

117
Tips for Reports Presentations
  • When reporting to current funders
  • Describe how you applied their funds directly to
    benefit youth.
  • Provide numbers (of participants, attendance,
    etc.).
  • Include graphics and tables that make a point.
  • Use narratives to highlight how their support
    affected one youth.
  • When reporting to future funders
  • Provide positive impact of previous programming
    through sound evaluations.
  • Show youths, parents, and communitys desire to
    have additional programming.
  • Highlight total numbers potentially served. If
    that number is small, perhaps highlight total
    number of participant-hours, which shows
    intensity of the program.

118
Tips for Reports Presentations
  • When reporting to parents
  • Describe observed benefits to youth and their
    development.
  • Describe how program supported parents efforts.
  • Outline, with data, how parents can support
    and/or advocate for program.
  • When reporting to the press
  • Be concise Use short sentences (subject, verb,
    object).
  • Use personal stories of youth.
  • Tie results into current issues.
  • Tie results to resolving local problems.

119
Tips for Reports Presentations
  • When reporting to policymakers
  • Be concise.
  • Be prepared to use data to provide
    counterarguments.
  • Tie results to financial considerations (cost per
    child for program versus other programs or
    consequences.)
  • General tips on reporting your evaluation
    information
  • Know what is important to your audience.
  • Have both numbers and personal anecdotes on hand.
  • Use social math Describe numbers in terms
    relevant to the audience.
  • Focus demands sacrifice Make a point instead of
    just throwing in everything.

120
Glossary of Terms
  • Focus Group A data collection tool in which
    there is discussion by a group of people
    facilitated by a moderator. Focus groups can
    provide a lot of information about peoples
    opinions about the program in the short time of
    an hour or two.
  • Goal A broad expectation of what you or your
    program would like to accomplish.
  • Interview A data collection tool that that gives
    you information about how one person feels about
    a subject. In program evaluation, an interview
    can help you learn a young persons views on a
    program or how he/she has changed as part of the
    program.
  • Measure A measure is a category of information
    (data) that you can ask about (through logs,
    interviews, questionnaires, etc.) that will help
    you to understand if your program is working and
    if participants are getting out of the program
    what you want them to learn.

121
Glossary of Terms
  • Process Evaluation A type of evaluation in which
    you assess how the program is functioning and if
    it is implemented as intended.
  • Process Measure A measure that helps you to find
    out if you are running your program like you want
    (to assess if you are meeting your process
    objectives).
  • Program Evaluation Collecting information about
    a program or some aspect of a program to assess
    if the program is meeting its intended outcomes/
    objectives.
  • Objective A specific and measurable statement
    identifying the thing your program will do or
    what the youth will learn and do that will help
    you accomplish your broader program goal.

122
Glossary of Terms
  • Observation A data collection tool in which you
    document youth in the program to see what they
    are actually doing in the program.
  • Outcome Evaluation A type of evaluation in which
    you assess how participants have changed as a
    result of the program.
  • Outcome Measure A measure that helps you to find
    out if the program is making a difference in the
    lives of the participants, such as a change in
    their knowledge, attitudes, or skills (to assess
    if you are meeting your outcome objectives).
  • Record A data collection tool which gathers
    information during the program, such as
    attendance logs or curriculum checklists. Records
    chart the progress of the program
    (process-related) or the participants
    (outcome-related).

123
Glossary of Terms
  • Survey A survey or questionnaire is a written
    set of questions on a topic. A survey is one of
    the quickest ways to get information from a large
    number of people.
  • Youth Participatory Evaluation A type of
    evaluation in which young people engage in
    assessing their programs.

124
Resources
  • Tips on Evaluation
  • Innovation Network, Inc. provides planning and
    evaluation tools for nonprofits and funders
    (www.innonet.org)
  • The Evaluation Exchange-shares tools and tips
    from the program evaluation field
    (www.gse.harvard.edu/hfrp/eval.html)
  • SurveyMonkey-web service for creating internet
    surveys, options at no charge are available
    (www.surveymonkey.com)

125
Resources
  • Youth Participatory Evaluation
  • Checkoway B. and Richards-Schuster K.
    Facilitators guide for participatory evaluation
    with young people. Ann Arbor University of
    Michigan, 2002.
  • Johns Hopkins Center for Adolescent Health. Youth
    Participatory Evaluation Project (YPEP) Evaluator
    trainer manual and participant booklet.
    Baltimore Johns Hopkins University, 2005.
    Available at www.jhsph.edu/adolescenthealth
  • Sabo K. Youth participatory evaluation A field
    in the making. San Francisco Jossey-Bass, 2003.

126
Frequently Asked Questions (FAQ)
  • Q We have a lot of parts to our program. We
    offer clinical services and have youth
    programming. How do I know what to evaluate?
  • A It is a good idea to make a list of questions
    you want to answer about your program and then
    prioritize based on what makes sense funding
    needs, planning issues, problem areas, etc. It is
    common to not have money or resources to evaluate
    everything, particularly when you have lots of
    services/programs. Making this list will help you
    know what to assess first.

127
Frequently Asked Questions (FAQ)
  • Q I use surveys a lot in my programs. But,
    participants always leave lots of it blank. What
    should I do?
  • A Survey completion can be challenging. It is
    important to consider the length of your
    questionnaire and what makes sense for the age of
    you participants. You want to use a survey that
    is clear, easily understood, and as short as
    possible to ask your most important questions.
    Also, consider when you administer the survey. If
    it is at the end of day, the reason for
    incompletion may be fatigue.

128
Frequently Asked Questions (FAQ)
  • Q I like a lot of the data collection tools
    described in this module-surveys, interviews,
    portfolios, etc. How do I know what to use? I
    dont have money or time to do all of them.
  • A Ask yourself, What questions do I really need
    answered for this program evaluation? Based on
    these questions, pick a tool (or tools) where you
    can collect information to answer as many
    questions as possible. A good rule of thumb is
    2-3 tools. It will be hard to use more than that
    given the challenge of administration, time, and
    funding.

129
Frequently Asked Questions (FAQ)
  • Q I plan to use a pre-post survey design for my
    evaluation. But, kids enter my program not at one
    time, but on a rolling basis. Is it okay to give
    those kids the pre-survey when they enter, or do
    I have to do it all at one time?
  • A You can administer as they enter, but keep in
    mind that the content of your program evolves.
    So, it will be difficult to know the total
    impact of the program on all participants because
    they may have different experiences. You can take
    this into account with some sophisticated data
    analyses if that is available to you. If you
    administer the pre-survey at different times, you
    are able to chart change in individual
    participants, which may be useful information.
    And, it is always a good idea to document how
    your program is implemented (process evaluation)
    so that you can draw accurate conclusions about
    your program effect.

130
Frequently Asked Questions (FAQ)
  • Q I want to include young people in the
    evaluation planning, but I am not ready to use
    the youth participatory method that you mention
    in the module. What can I do?
  • A It is always a good idea to involve youth
    perspectives in some way as you plan your
    evaluation. You can ask a few students not part
    of the program (e.g., graduates) to review your
    questions to let you know if you are capturing
    areas of interest.

131
Frequently Asked Questions (FAQ)
  • Q I plan to use surveys and interview some kids
    in my program. What do I do with all of the
    information?
  • A You should have a plan for recording and
    tabulating the survey data (e.g., using a
    spreadsheet) and for reviewing notes/tapes from
    the interviews (e.g., writing down quotes and
    themes). It is helpful to write an outline for
    your report (i.e., what do you want to tell your
    audience) because this guide can help in knowing
    how to analyze and summarize your data.

132
Frequently Asked Questions (FAQ)
  • Q My funder wants numbers for my youth program,
    but my program doesnt look at outcomes like
    this. How can I include some numbers (or frame
    the impact of my program for my funders)?
  • A. Funders often like to report on the numbers of
    people served by their funded programs. Consider
    different ways of presenting information from
    your program total number of youth served
    during the year, average number of youth served
    per session, numbers of communities represented
    by your youth, etc. When numbers are small, a
    funder may like to include a case study that
    shows how one participant represents the
    experiences of many of the participants.
About PowerShow.com