OutcomeBased Program Evaluation

1
Outcome-Based Program Evaluation

• Athena Education Initiative (AEI)
• Athena Software.net

2
What are Outcomes?

• Benefits or changes for individuals or
  populations during or after participating in
  program activities
• Changes sought in the knowledge, attitudes,
  motivation, skills, behavior, condition, status
  or other characteristics of a specific individual
  or group of individuals
• (Outcomes Measurement Resource Network, United
  Way of America)

3
Why Evaluate Outcomes?

• To improve your service
• To develop stronger proposals
• To promote the value of your service
• To celebrate your successes
• To be equipped to respond creatively to
  challenges
• Client outcomes are the bottom-line for mental
  health services, like profit is for business
  (SAMHSA, US dept. of health and human services)

4
Why Evaluate Outcomes?

• To Measure i.e., to gain a closer approximation
  of the truth about our work and the community
• To Improve i.e. , to act in a more informed and
  reflexive way to know how and why to change our
  practice
• To Impact i.e., to know that our work made a
  real difference
• To Communicate i.e., to demonstrate that our
  work made a real difference to stakeholders

5
Approach to Evaluation

• Defining goals identification of needs to be
  addressed by the organization, defining program
  (and client) goals
• Developing indicators
• Tracking information
• Reporting information
• Evaluating information

6
Approach to Evaluation

• Defining goals
• Developing indicators determining information to
  be tracked that will measure whether goals are
  reached
• Tracking information
• Reporting information
• Evaluating information

7
Approach to Evaluation

• Defining goals
• Developing indicators
• Tracking information encouraging compliance and
  consistency in tracking inputs,
  activities/processes, outputs and outcomes.
  Keeping it simple and a natural part of workflow
• Reporting information
• Evaluating information

8
Approach to Evaluation

• Defining goals
• Developing indicators
• Tracking information
• Reporting information gleaning meaningful
  statistics from mountains of data with the
  flexibility to compile information in ways that
  meet the unique requirements of each organization
  
• Evaluating information

9
Approach to Evaluation

• Defining goals
• Developing indicators
• Tracking information
• Reporting information
• Evaluating information interpreting, analyzing
  and utilizing information to improve service

10
Evaluating Information

• Using information obtained through monitoring to
• Foster and reinforce positive outcomes
• Identify barriers to performance
• Inform future program development and service
  delivery
• Evaluation is not about collecting mountains of
  data!
• An accumulation of facts is no more a science
  than a heap of stones is a house - Henri
  Poincare.

11
Evaluation in Penelope

• Data Elements of
• Program Evaluation
• (United Way of America)

inputs
activities
outcomes
outputs
Tracking outcomes information becomes a natural
part of workflow processes with an integrated
case management system
indicators
12
Evaluation in Penelope

• Questions for Managers (Wholey, 1979)
• What are the program objectives?
• How would you know if the program is
  accomplishing its objectives?
• Why will the inputs/activities lead to the
  program objectives?
• What are the challenges to meeting objectives?
• How do you get the performance information you
  need?

13
Evaluation in Penelope

• Using Penelope, you decide!
• The information to be tracked in the system
• The outside information to be stored in the
  system
• The indicators you feel will best assess outcomes
• The assessments you want to incorporate into the
  system
• THE SKY IS THE LIMIT

14
Evaluation in Penelope

• staff and staff hours direct and indirect
  service
• staff qualifications, certificates, areas of
  specialty, user-defined attributes
• volunteers and volunteer time
• staff caseloads, rates, peer groups
• resource utilization facilities, equipment and
  supplies

inputs
15
Evaluation in Penelope

• Individual, couple, family counseling sessions
• Group therapy sessions
• Educational and outreach activities (seminars,
  series)
• Prevention and advocacy activities
• Indirect service activities (prep. work,
  training time etc.)

activities
16
Evaluation in Penelope

• Number of classes, seminars
• Number of individual, couple, family and group
  counseling sessions
• Number of home visits, school visits, referrals
• Number of information kits, brochures, other
  program materials distributed, mailed, utilized
• Number of clients/participants/families served

outputs
17
Evaluation in Penelope
indicators
outcomes

• program-specific functional assessments/surveys
• program-specific assessments of knowledge
• psychological assessments and/or assessment
  scores
• client-specific progress towards client goals
• program/client specific follow-up surveys

• new knowledge
• new skills
• new attitudes
• modified behavior
• improved condition

18
Evaluation in Penelope

• Reporting Outcomes
• many inputs/activities/outputs and outcomes can
  be reported on using our 100 system reports
• ask more specific questions by exporting any and
  all relevant data to MSExcel

outcomes
19
Evaluation in Penelope

• Consider influential factors in reports
• demographic characteristics of the participants/
  clients
• level of difficulty presented by the participant/
  client
• geographic location of the participant/client
• organizational unit (site, office, staff member)
• type or amount of procedure used to deliver the
  service
• other characteristics of the participants/clients,
  service, staff member relevant for the
  particular program/organization/funder

20
Athena Education Initiative

• Where Theory meets Practice
• AEI is an innovative new program to provide an
  information technology component to education in
  human services.

21
Athena Education Initiative

• The Athena Education Initiative provides
  University and College course instructors with
• Access to an integrated case management software
  system with a built in evaluation component
• Course tutorials on the use of a case management
  system for mental health services including a
  program/client evaluation tutorial

22
Athena Education Initiative

• The Athena Education Initiative provides
  University and College course instructors with
• Support for instructors interested in
  incorporating a tech component to existing course
  materials.
• Additional expertise through partnership with
  outcomes evaluation professionals

23
AEI Outcomes Tutorial

• Students may develop program goals for one of
  several programs in a fictitious social service
  agency
• They may decide what information they will need
  to obtain and/or track, including outcomes
  indicators in order to evaluate whether the
  program goals are being met

24
AEI Outcomes Tutorial

• They may track this information as appropriate in
  an integrated case management software system as
  part of their service provision as clinicians at
  the organization
• They may evaluate the information, by defining
  the data to be queried, exporting and analyzing
  the data and writing a report

25
AEI Find out More

• If you want to learn more about the Athena
  Education Initiative
• visit the AEI website
• www.athenasoftware.net/AEI/athena_education_initia
  tive.htm
• call Athena Software at 888.826.0811 ext 1
• email Dana Fox at dana_at_athenasoftware.net