Conducting Successful Program Evaluations: Researchers Reveal Their Insights

1
Conducting Successful Program Evaluations
Researchers Reveal Their Insights

• Sue Palsbo, PhD
• Thilo Kroll, PhD
• NRH Center for Health Disability Research

2
Why Do an Evaluation?

• Does the program work?
• Why?
• How?
• Who benefits?
• Who loses?
• Who pays?
• How does it compare to other programs?

3
Evaluation Process

• State the
• Performance Measure

Determine the Outcome
Collect data
Analyze data
4
Who Cares?

• Interested in different outcomes
• Different outcomes require different methods
• Different methods require different data sources

5
Funding Considerations

• Public or private?
• Contracts or grants?
• Single source or multiple?
• Lead time
• Scope and /or size
• Funding organizations priorities

6
Diverse interests for outcomes
Diverse Stakeholders
Requires diverse set of measures
qualitative
quantitative
Blended together for total picture
7
Evaluation Types

• Formative
• Immediate feedback allows program modifications
  and mid-course corrections
• Summative
• Post-program perspective
• Comprehensive overview of outcomes
• Total impact of the intervention or program

8
Design Considerations

• Unit of analysis
• Person, Provider, State
• Funding for the study
• Single vs. multi-method approach
• Short-term vs long-term outcomes
• Legacy impacts e.g. sustainability and
  duplication

9
How to Select the Evaluators

• Your evaluation identifies the questions and
  methods these indicate the skill set you need
• In-house or external?
• Sole source contracting
• Competitive bidding

10
Example
11
Stakeholders
12
Evaluation Funding
13
Evaluation Design

• Consortium model
• Prospectively designed
• Contemporaneous (formative)
• Highly leveraged
• 3-year cumulative

14
MnDHO Goals

• 1. Create and maintain satisfaction
• 2. Promote overall well-being of enrollees
• 3. Meet cost and utilization goals.

15
Measurement Domain

• 1. Satisfaction
• 2. Quality of care
• 3. Utilization and patterns of care
• 4. Costs and rate setting

16
Measurement Selection Criteria

• 1. Use existing data
• 2. Avoid duplication of information.
• 3. Avoid recall bias.
• 4. Minimize respondent burden.

17
Preliminary Findings From Mixed-Methods Evaluation

• Longitudinal CAHPS-based survey
• At MnDHO enrollment
• 1 year post-enrollment
• Focus groups
• MnDHO enrollees
• Eligible people who chose not to enroll

18
Survey Content

• General satisfaction
• Care coordination experience
• Access to a wide variety of health and
  long-term-care services
• Self-directed care
• Quality of interactions with health care
  providers
• Quality of interactions with MA or AXIS staff

19
Longitudinal Survey

• 100 MnDHO participants
• 35 enrollees have completed baseline and
  follow-up surveys.
• Baseline
• Health care experiences in the year before
  enrollment.
• Follow-up
• Health care experiences in MnDHO.

20
4 Focus Groups

• October, 2002
• 2 MnDHO enrollees
• 2 fee-for-service Medical Assistance program

21
Care Coordination. In the year before / after
you enrolled in AXIS/UCare Complete, did anyone
help manage the health care services you received
from different doctors, nurses, therapists,
PCAs,or equipment providers?
22
Focus Group Statements

• The MnDHO / AXIS Experience
• Shell (health coordinator) line me up with my
  appointments and stuff. Shell find my therapists
  and my physical equipment, all that stuff.
• My stress level has been relieved somewhat. Im
  able to focus on more vocational and future
  issues, as opposed to the day-to-day healthcare
  issues

23
Focus Group Statements

• The fee-for-service experience
• I spend most of my time on the phone calling
  people, setting up appointments, or trying to get
  services myself, and that is very tiring. I get
  exhausted because I do have MS.
• Our experience was so bad we had to hire our own
  coordinator. It seems like if the person is
  stable, people dont return their calls, people
  dont seem to care

24
Self-direction in health care. In the year
before / after you enrolled in AXIS/UCare
Complete, were you involved as much as you wanted
in making decisions about your health care?
25
Focus Group Statement

• My healthcare coordinator, she always involves
  me. It is our lives. They can give us all the
  information, but ultimately it comes down to us,
  what we want to do... it comes to us making the
  final decision... The healthcare coordinators
  are always there to give you the pros and cons.

26
Recommendations

• Its worth the time and money to do a program
  evaluation
• Do a formative evaluation so you can make changes
  in real time
• Important lessons from a summative evaluation can
  tell you what worked and what didnt, to improve
  the next effort
• Mixed methods provide more information together
  than a single method