Measuring Consumer Perception of Care

1
Measuring Consumer Perception of Care

• Challenges Opportunities
• John Bartlett, M.D.,M.P.H.
• March 20-21 Meeting with Californias Division of
  Alcohol and Drug Programs

2
Goals for Presentation

• Review consumer perception of care as a concept
• Review success criteria for its measurement
• Review the development testing of CSATs
  Modular Survey

3
The Concept

• Measuring consumer response
• core business function
• In healthcare tied to growth of consumerism
• CQI initiatives
• A NOMS domain
• Consumer perception of care ? satisfaction

4
Measuring Consumer Perception of Care

• Approaches to measurement differ
• depending on scope purpose
• For purposes of comparability, improvement over
  time, benchmarking measures must be
• Meaningful
• Scientifically-sound
• Actionable

5
The Problems with Satisfaction

• No evidence linking the measurement of
• satisfaction to client outcomes
• Few satisfaction surveys scientifically
  validated
• Data is not actionable (ceiling effect)

6
The Modular Survey

• SAMHSA supported initiative
• Conducted under the auspices of the Forum
• on Performance Measurement
• the Washington Circle
• Conducted in 2 phases
• Phase 1 in conjunction with mental health Rx
• Phase 2 substance abuse specific

7
Design Requirements

• Phase 1 focus on commonality,
• not comprehensiveness
• Short
• Scientifically sound
• Actionable
• Use of existing, widely-used, non-proprietary
• surveys
• Consensus-driven

8
Modular Survey Flow of Common Questions for
Individual Respondent
Common Design Template
Adult Mental Health Core Measures
Adult Common Measures
Adult Substance Abuse Core Measures
Field-level Common Measures
C/Adol Mental Health Core Measures
Child/ Adolescent Common Measures
Adol Substance Abuse Core Measures
9
Approach to Phase 1

• 4 workgroups to develop consensus
• Selection of instruments
• Identification of concerns
• Identification of potential items
• Ranking of items
• Final item selection (modified Delphi)
• Pilot testing

10
Phase 1 Pilot Testing

• Conducted during summer/fall 2004
• Primary data collection in Cincinnati
• United Way agencies (N 1157)
• Secondary analysis using MHSIP data
• (16 state LA County data sets)
• Final N gt 22,000 respondents
• Pool of items reduced from 28 to 11
• All items common to both fields, both populations

11
Approach to Phase 2

• Stand-alone SUD Rx initiative
• Under Washington Circle with Forum as
  subcontractor
• New item development (no existing SUD survey)
• Content work group co-chaired by
• Tom McLellan (TRI) Doreen Cavanaugh
  (Georgetown)
• Support from Forum Methods Work Group
• Ann Doucette (George Washington
• Public Provider Consumer Advisory Groups

12
Phase 2 SUD Initiative

• Closely coordinated with NOMS
• Identification of concerns
• Relationship to treatment program
• Self awareness of problem/commitment to change
• Perceived outcomes
• Social connectedness
• Generation of items (35 in testing pool)

13
Phase 2 Pilot Testing

• OMB IRB approval spring 2006
• Conducted in 3 rounds
• Round 1 Adult Adolescent (summer 2006)
• 14 programs, N 1207
• Round 2 Adult Adolescent (winter 2006 07)
• 6 programs, N 585
• Round 3 Adolescent (spring-summer 2007)
• 8 programs, N 268
• Final adult N 1549 (2 samples)
• Final adolescent N 492 (1 sample)
• All demographic groups covered except Native
  American

14
Phase 2 Completion

• Analysis recommendations by Ann Doucette PH.D.
• Use of IRT
• For a copy of the technical report, e-mail
• jbartlett_at_avisagroup.com
• Review by Forum Methods Work Group
• (November 2007)
• Review Approval by SUD Content Committee
• (November 2007)

15
Final SUD Modular Survey

• 21 items (11 from Phase 1, 10 from Phase 2)
• Quality 6 items
• Perceived Outcomes 6 items
• Social Connectedness 7 items
• Commitment to Change 2 items
• 10 demographic background items
• Spanish translation available

16
Modular Survey Flow of Common Questions for
Individual Respondent Final Version
Phase 2 Mental Health Items
Phase 1 Common Items
Phase 2 SUD Items
Common Design Template
All populations, all fields
17
Convergence with NOMS

• In Spring 2006 NOMS Technical Consulting Group
  convened
• Recommended 17 items from 8 different
  instruments
• 9 of the 17 from the Modular Survey
• 5 are in the Final Modular Survey

18
In summary

• Consumer perception of care key measurement
  domain
• Its measurement must meet certain criteria in
  order to be worth the effort
• The Modular Survey is the only current
  instrument measuring consumer perception of
  care that is
• SUD Rx specific
• Product of both consensus and empirical analysis
• Short and actionable