Measure Your Success:

1
Measure Your Success Practical Program
Measurement at Therapeutic Horseback Riding
Centers By Sarah Seris, MPA Cedar Creek
Therapeutic Riding Center, Columbia, Missouri
2

• Greetings and Introductions
• Please ask questions!
• How did I end up here?
• From volunteer to program evaluator

3

• Part I Why measure?
• Part II Before we measure
• Part III How do we measure?
• Part IV What do we do with the data?
• Part V How does this apply to my center?

4
Why measure?
The role of Non-profits today

• Changes in funding sources
• Accountability
• Arent success stories enough?

5
Why measure?

• Measurement leads to
• More funding!
• Grant opportunities
• A stronger sense of organizational needs
• Identifies strengths
• Identifies weaknesses
• Demonstrates needs for new programs (and more
  funding!)
• Backs up all those great anecdotes

6
Why measure?

• You may already do some measuring
• Anecdotes
• Individual rider documentation
• Client/Volunteer Surveys

7
Why measure?

• Recap of Part I
• Why do we measure?
• Because we know Therapeutic Riding works
• Because we want everyone else to know how well it
  works
• Because if we present our successes in HARD DATA,
  funding will follow!

8

• Before we measure
• Program Organization
• Mission, Vision, Goals and Objectives
• Program Function
• Logic Models Assumptions, inputs, activities,
  target population, outputs and outcomes

9
Before we measure

• Expectations and Limitations
• Program evaluation data contributes to more
  rational decision-making on the part of
  managers
• Program evaluation does not replace judgment
  based on managerial experience or even values in
  decision-making
• Program evaluation does not replace politics
• Program evaluation DOES NOT replace passion or
  common sense

10
Before we measure

• Program evaluation starts with this
• Program Vision
• Program Mission
• Program Goals/Objectives

11
Before we measure

• Start with Vision
• Reflect on the vision of your center

12
Before we measure

• Vision of Success
• There is no more powerful engine driving an
  organization toward excellence and long-range
  success than an attractive, worthwhile, and
  achievable vision of the future, widely shared
• Quite simply a vision is a realistic, credible,
  attractive future for your organization.
• Burt Nanus, Visionary Leadership

13
Before we measure

• The visionary is the only realist.
• Federico Fellini, Filmmaker
• Vision is a clear and succinct description of
  what the organization or community should look
  like after it successfully implements its
  strategies and achieves its full potential.
  John Bryson

14
Before we measure

• Nonprofit Vision Statement
• Vision keeps your organization moving forward
• Vision encourages noble aspirations and
  motivation
• Vision states positive outcomes that are desired
• Vision concentrates on the end goal or desired
  future state of the organization, not on the
  means to the goal
• Vision emphasizes distinctive competence
• Vision must be shared with organizational members

15
Before we measure

• Characteristics of an Effective Vision
• Future-oriented
• Clarifying
• Identifying
• Ambitious
• Idealistic
• Energizing
• Meaningful

16
Before we measure

• Vision Statement Example
• Through strategic investments in the people,
  ideas and partnerships essential to outstanding
  research in strategic areas, Science Foundation
  Ireland will help build in Ireland research of
  globally recognized excellence and nationally
  significant economic importance.

17
Before we measure

• Vision Statement Example
• Building on more than 40 years of excellence in
  public administration and policy education,
  research, and public service, the Harry S Truman
  School of Public Affairs will be recognized as a
  leader in the study and practice of new
  governance. In all of our work, we will honor
  the legacy of Harry S Truman and his leadership,
  integrity, and commitment to public service.

18
Mission Statements
Before we measure
So may a thousand actions, once afoot, End in
one purpose, and be well borne without
defeat. William Shakespeare
19
Before we measure
Mission Statement
A mission statement addresses the organizations
fundamental reason for being and specifies the
functional role that the organization is going to
play in its environment. Leonard Goodstein
20
Before we measure
Formulating a Mission Statement

• In formulating a mission statement, an
  organization
• must answer four primary questions
• What function(s) does the organization perform?
• For whom does the organization perform this
  function?
• How does the organization go about filling this
  function?
• Why does this organization exist?

21
Before we measure

• NARHA is a membership organization that fosters
  safe, professional, ethical and therapeutic
  equine activities through education,
  communication, standards and research for people
  with and without disabilities.
• NARHA Website, www.narha.org

22
Before we measure

• NARHAs Mission Statement
• What?
• NARHA is a membership organization
• For whom?
• For people with and without disabilities
• How?
• Through education, communication, standards and
  research
• Why?
• To foster safe, professional, ethical and
  therapeutic equine activities

23
Before we measure

• Theory review
• Vision Mission Goals
• Something to Measure

24
Before we measure

• GOALS
• An effective goal focuses primarily on results
  rather than activity. It identifies where you
  want to be, and , in the process, helps you
  determine where you are. It gives you important
  information on how to get there, and it tells you
  when you have arrived. It unifies your efforts
  and your energy. It gives meaning and purpose to
  all you do. And, it can finally translate itself
  into daily activities so that you are
  pro-active.
• Stephen R. Covey
• The Seven Habits of Highly Effective People

25
Before we measure

• Examples of Goals
• To provide Missouri families with access to
  affordable child or adult day care
• To increase the amount of safe and affordable
  housing in the City of Columbia, Missouri
• To advance knowledge and practice of governance
  in Missouri by educating for ethical leadership
• To provide Missourians age 55 and older with
  services contributing to independent living

26
Before we measure

• Goals for Therapeutic Ridings Centers
• Goals for individual clients
• vs.
• Program Goals

27
Before we measure
Goals Are Achieved by Means of Objectives
Objectives Specific statements that detail the
desired accomplishments of a program.
Peter Rossi
28
Before we measure
The Evolution of a good Objective Stage 1 To
improve the reading skills of at risk
students Stage 2 To improve the reading skills
of at risk students, ages 14 to 18 Stage 3 To
improve the reading skills of at risk students,
ages 14 to 18, through tutoring
29
Before we measure
Stage 4 To improve the reading skills of at risk
students, ages 14 to 18, through tutoring,
as measured by performance on the school
districts reading comprehension test to be
administered before and after the program Stage
5 To improve the reading skills of at risk
students, ages 14 to 18, through tutoring,
as measured by an average increase of five
percent on the school districts comprehension
test to be administered before and after the
program
30
Before we measure
Stage 6 To improve the reading skills of 25
at risk students, ages 14 to 18, through
tutoring, and as measured by an average
increase of five percent on the school
districts reading comprehension test to be
administered before and after the program
31
Before we measure

• Therapeutic Riding Example
• Increase functioning in 65 of clients,
  according to their disability, through
  therapeutic riding, and measured by client
  outcomes surveys administered after each session.

32
Before we measure
Defining the Organization Vision
Statement Mission Statement Goals Objectives A
ctions
33
Before we measure

• Program Logic Models
• A catch phrase for program evaluation
• Useful, pretty, and dynamite in grant proposals
• Road map for organizational function

34
Before we measure
What is a Program Logic Model? A program logic
model is a picture of how your program works
the theory and assumptions underlying the
program This model provides a road map of your
program, highlighting how it is expected to work,
what activities need to come before others, and
how desired outcomes are achieved. Kellogg
Foundation Evaluation Handbook
35
Before we measure
Program Assumptions Inputs (Resources) Activitie
s Target Population(s) Outputs Outcomes Impact
s
36
Before we measure
Program Logic Models Assumptions
Assumptions Tacit ideas about how the world
works, or a mind-set that influences the
perceptions and interactions of organizational
members. Assumptions frequently emerge from
underlying values.
37
Before we measure

• Assumptions
• Therapeutic horseback riding is a viable and
  important part of therapy for many people with
  disabilities
• It is possible to find donated horses

38
Before we measure
Any other assumptions?
39
Before we measure

• Inputs include resources dedicated to or
• consumed by the program
• Examples
• Money
• Personnel
• Volunteers
• Facilities, equipment, and supplies

40
Before we measure
Other Examples of Program Inputs

• Specialized skills (for example, a MSW program
  manager)
• Education manuals, videos, and other teaching
  tools
• Federal, state, local laws and regulations
• Relationships with other organizations
  (collaborative efforts)
• Information resources

41
Before we measure
Any suggestions for inputs?
42
Before we measure

• Program Activities
• Program Activities The hands-on duties or tasks
  that are carried out by employees to fulfill the
  purposes of the program. Activities are what you
  do.

43
Before we measure

• Examples of Therapeutic Riding Program
• Activities
• Riding sessions
• Driving sessions
• Equine Management
• Volunteer education

44
Before we measure
Think about your program, what are its activities?
45
Before we measure

• Target Population
• Target Population The unit (individuals,
  family, community, etc.) to which a program
  intervention is directed. All such units within
  the area served by a program comprise its target
  population.
• Peter Rossi

46
Before we measure

• Target Population
• Those people or entities who directly interact
  with your program/services and staff
• Those whose behavior or condition you are
  intending to influence with your services and
  programs (product)
• They have a choice whether to use or benefit from
  your products
• They share certain behaviors and characteristics

47
Before we measure

• Examples of Target Populations
• Children aged 10 to 14 who are between one and
  three years below their normal grade in school
• Pregnant women and mothers of infants with annual
  incomes less than 150 of the poverty line
• Elementary schools (kindergarten to eighth grade)
  in which at least 30 of the pupils qualify for
  the federal free lunch program

48
Before we measure

• Therapeutic Riding Examples
• People with disabilities in central Missouri who
  have no contraindications to therapeutic
  horseback riding
• Children with autism ages 4-10
• Developmentally disabled Adults

49
Before we measure

• Target Populations
• To be effective, a program must not only know
  what its target population is but also be able to
  readily direct its services to that population
  and screen out individuals who are not part of
  that population.

50
Before we measure

• Target Populations
• Population in Need
• Population at Risk
• Incidence
• Prevalence
• Rates

51
Before we measure

• Identify the target population for your
• program

52
Before we measure

• Program Outputs
• Program Outputs The direct products of program
  operation that are usually measured in terms of
  the volume of work accomplished.
• United Way

53
Before we measure

• Examples of Program Outputs
• Number of classes taught
• Number of meetings held
• Program participation rates
• Number of hours of each type of service provided

54
Before we measure

• More Examples of Outputs
• Number of highway miles paved
• Graduation rates
• Test scores
• Number of people with disabilities served by a
  therapeutic riding center

55
Before we measure
List the outputs of your program
56
Before we measure

• Outcomes
• Program Outcomes Specific changes in attitudes,
  behaviors, knowledge, skills, status, or level of
  functioning expected to result from program
  activities and which are most often expressed at
  an individual level.
• Kellogg Foundation Handbook

57
Before we measure

• Program Outcomes
• Program Outcomes Benefits or changes for
  individuals or populations during or after
  participating in program activities. Outcomes
  are influenced by outputs. Outcomes may relate
  to knowledge, attitudes, values, skills,
  behavior, condition, or other attributes. They
  are what participants know, think or can do or
  how they behave or what their condition is, that
  is different following the program.
• United Way

58
Before we measure

• Program Outcomes Examples
• Financial Management Program
• Outputs
• Number of families counseled
• Number of financial planning sessions conducted
• Outcomes
• Developing and living within a budget
• Making contributions to a savings account on a
• monthly basis
• Increased financial stability

59
Before we measure

• Program Outcomes Examples
• Neighborhood Clean-up Campaign
• Outputs
• Number of organized meetings
• Number of weekends dedicated to the clean-up
  effort
• Outcomes
• Reduced exposure to safety hazards
• Increased feelings of neighborhood pride

60
Before we measure

• Program Outcomes Examples
• Therapeutic Riding Center
• Outputs
• Number of clients served
• Number of volunteers trained
• Outcomes
• Increased functioning for participants
• Increased awareness about people with
  disabilities

61
Before we measure
List the outcomes for your program
62

• Why do we measure?
• Because we want to demonstrate how well
• Our program works.
• What do we do before we measure?
• Look at Program Organization and Program
• Function, determine OUTCOMES
• How do we measure?
• Find a way to measure those OUTCOMES!

63

• How do we measure?
• Step 1 Determine what to measure
• Step 2 Choose a measurement tool
• Step 3 Complete the measurement
• Step 4 Extract the data
• Step 5 Make the data USEABLE!

64
How we measure

• The difficult history of Measuring
• Therapeutic Horseback Riding (THR)
• Not one published study measuring the OVERALL
  effects of THR
• Specific studies abound
• The mission Measure the entire CCTRC Program

65
How we measure

• Step 1 What are we measuring?
• Before participating in THR
• After participating in THR
• What changes?

66
How we measure

• Limitations
• Errors
• Validity
• Does this really measure what we do?

67
How we measure

• What are we measuring
• Perceptions of change
• After one session of riding
• Recorded by the rider/closest caretaker

68
How we measure

• Cedar Creek objective statement
• The main objective of the CCTRC program is to
• increase functioning for the target population
• according to his or her disability, and measured
  by
• the client outcomes survey.
• Cedar Creek research question
• How does a clients functioning change after one
• session of Therapeutic Horseback Riding?
• Cedar Creek measures
• Self-reported changes in client functioning after
  one
• session at Cedar Creek.

69
How we measure

• Step 2 Measurement Tools
• Surveys
• Focus Groups
• Other tools

70
How we measure

• The Cedar Creek Client Outcomes Survey
• Functional Independence Measure/Functional
  Assessment Measure, created by the Center for
  Outcome Measurement in Brain Injury
• http//www.tbims.org/combi/
• Rosenberg Self-esteem Scale
• http//www.bsos.umd.edu/socy/grad/socpsy_rosenberg
  .html

71
(No Transcript)
72
How we measure

• Volunteer Survey
• Changes in attitudes about people with
  disabilities
• Sufficient training for the Program

73
(No Transcript)
74
How we measure

• Step 3 Complete the measurement
• Practical survey distribution
• Costs
• Other options

75
How we measure

• Step 3 Complete the Measurement
• All confidential
• Receiving point
• Designated area

76
How we measure

• Step 4 Extract the data
• What the heck does this mean?

77
How we measure

• Step 4 Extract the data
• Look at each item
• How many clients out of 54 chose 3 on an item?
  How many chose a 4? A 5?
• Divide into percentages.

78
How we measure

• Data extraction for Balance
• 54 responses in the balance category
• 10 (18) said balance had significantly improved
• 13 (25) said balance had moderately improved
• 16 (30) balance had slightly improved
• 15 (27) balance had not improved
• Overall, 73 experience some positive change in
• balance that they attribute to Therapeutic Riding

79
How we measure

• Data extraction for Self-confidence
• 54 responses in the balance category
• 20 (37) said Self-confidence had significantly
  improved
• 7 (13) said Self-confidence had moderately
  improved
• 19 (35) said Self-confidence had slightly
  improved
• 8 (15) said Self-confidence had not improved
• Overall, 85 experienced some positive change in
• Self-confidence that they attribute to
  Therapeutic
• Riding

80
What do we do with the Data?

• Step 5 Make the Data useable
• Summaries
• Graphs
• Pick important results
• Dont overuse the data

81
Example of Table to Display Data
Complete findings from survey. Numbers are
percentages of all respondents who chose that
ranking for that specific function.    
82
Example of Pie Chart with data, using Microsoft
Excel
83
Example of Pie Chart with data, using Microsoft
Excel
84
What do we do with the Data?

• Step 5 Making data usable
• Using the data in grants
• Sample paragraph
• The main objective of the CCTRC program is to
  increase functioning for the target population
  according to his or her disability, dictated by
  the goals set before each therapy session. In
  the last program evaluation, 58 of clients
  evaluated experienced improvement in all 23
  categories measured. As a result of participating
  in the CCTRC program, more than one-third of
  clients evaluated experienced moderate or
  significant improvement in
• Anxiety level
• Self-confidence
• Attention span
• Emotional status
• Social interaction
• Verbal expression
• Fine motor skills
• Coordination
• Trunk strength
• Balance

85

• Quick (hypothetical) review
• Vision Provide THR to anyone who needs it and be
  the most outstanding center possible
• Mission This THR provides life-changing therapy
  to people with disabilities in this area through
  Therapeutic Horseback Riding
• Goal Increase functional outcomes for people
  with disabilities
• Output Provide THR to 100 people in 2005
• Outcome 58 of participants evaluated
  experienced positive change in functional
  ability.
• Measurement tool Client outcome survey

86
What do we do with the Data?

• Long term outcomes
• Potential is limitless
• Long-term riders
• Microsoft Access databases
• Infinite resources? (Yeah, right)

87
What do we do with the Data?

• No limits?
• Do this survey across several centers
• Do the real FIM/FAM survey with trained
  professional, before and after with a control and
  experimental group
• Measure different types of disabilities within
  the center
• Write a lot more grants

88
What do we do with the Data?

• Putting it all together
• Individual rider documentation (OT and PT)
• Client outcomes surveys
• Volunteer surveys
• Other

89
How does this apply to my center?

• How do you make it happen?
• Center size
• Availability of resources
• Interns
• Volunteers

90

• Conclusion
• Understanding program Measuring is vital for
  every not-for-profit organization
• The need for outcomes based measurement will
  increase
• Not-for-profits who understand its importance
  will survive.
• Measure your success!

91
Any Questions?