Improving Patient Care: How Medical Practices Are Using CAHPS Surveys for Ambulatory Settings

1
Improving Patient Care How Medical Practices
Are Using CAHPS Surveys for Ambulatory Settings

• Presented by the CAHPS Survey Users Network
• April 6, 2005 130 pm ET

2
Todays Speakers

• Susan Edgman-Levitan, PA, Executive Director,
  John D. Stoeckle Center for Primary Care
  Innovation, Massachusetts General Hospital
• Dana Gelb Safran, ScD, Director, The Health
  Institute, Tufts-New England Medical Center
• Richard Marshall, MD, Chief Medical Officer,
  Harvard Vanguard Medical Associates
• John Ingard, MD, Chief of Internal Medicine,
  Wellesley Practice, Harvard Vanguard Medical
  Associates
• Charles J. Hipp, MD, President, Stillwater
  Medical Group
• Lawrence E. Morrissey, MD, Medical Director,
  Stillwater Medical Group

3
Susan Edgman-Levitan, PA Executive Director,
John D. Stoeckle Center for Primary Care
Innovation, Massachusetts General Hospital
4
CAHPS Surveys Grounded in Four Principles

• Reports and ratings of experiences -- not just
  satisfaction
• Standardization to enable valid comparisons
• Evidence basis for design, protocols, and
  language
• All CAHPS products and services in public domain

5
CAHPS II Consortium A Public-Private Research
Team
AHRQ
Harvard
AIR
CMS
Federal Agencies

• Grantee Teams

CAHPS II Consortium
CDC
RAND
NIDRR
Westat
6
Public Reporting Creates Need for Improvement

• Widespread adoption of CAHPS surveys requires
  that they serve multiple purposes whenever
  possible
• Because of widespread adoption and the use of the
  plan level data for accreditation by NCQA and for
  public reporting by CMS, everyone wants to
  improve their scores.
• For providers and health systems, using the data
  for QI is 1 priority.

7
Limitations of the Health Plan Survey

• Plan-level data collection too imprecise for
  practice level improvements members vs.
  patients, large samples
• Surveys need to focus on the processes that are
  most relevant to consumers/patients at the plan
  level and the practice level, e.g. customer
  service for a plan, office staff interactions and
  the clinical experience for a practice.

8
How Were Addressing Those Limitations

• Requires the right unit of analysis
• Group and individual provider-level surveys
• Health plan surveys
• Requires new questions
• More questions about plan customer service
• More questions about doctor-patient communication

9
Why a Survey for Clinicians and Groups?

• It is the right unit of analysis for QI
• Consumers have more choice of practices than they
  do of plans
• Data can be actionable for an individual practice
  or site
• Plans/payers can identify benchmark performers
  more precisely.

10
Why a Survey for Clinicians and Groups?

• Data collection at the practice/ provider level
  survey helps support multiple uses of the same
  data
• Maintenance of certification for doctors, ABMS
  activities
• External use by plans/ payers
• CMS Doctor Office Quality project

11
CAHPS Clinician Group Survey Content Areas
(Composites)

• Doctor Communication
• Shared Decision-Making
• Health Promotion and Education
• Coordination/Integration
• Other Clinicians in the Practice (Care Team)
• Access
• Office Staff

Visit Module

12
CAHPS Health Plan Survey Composites

• Getting Needed Care
• Getting Care Quickly
• How Well Doctors Communicate
• Customer Service
• Office Staff
• Health Plan Customer Service
• Home Health and Preventive Services

13
Schedule for the CAHPS Clinician Group Survey
2005
2006
Fall
Spring
Winter
Spring
Summer
Cognitive testing
Field testing at select sites
Release Clinician Group Survey Submit to NQF
Psychometric testing of field test data Final
revisions
14
CAHPS Surveys for Every Level of the Health System

• Ambulatory Level
• Health Plans
• Group Practices
• Individual Clinicians
• Behavioral Health Organizations (ECHO)

• Facility Level
• Hospitals
• Dialysis Facilities
• Nursing Homes

15
CAHPS Quality Improvement Resources

• Web and print-based resources
• Tools, best practices, case studies
• The CAHPS Improvement Guide funded by CMS

16
The CAHPS Improvement Guide

• A resource manual for health plans and medical
  groups seeking to improve their CAHPS scores
• Funded by CMS (Medicare) and developed by Harvard
  Medical School CAHPS Team
• Over 2 dozen strategies mapped to CAHPS core
  questions

17
Another Resource The National CAHPS Benchmarking
Database
18
DanaGelb Safran, ScD Director, The Health
Institute,Tufts-New England Medical Center
19
Why the Focus on Physicians

• Survey-based measurement of patients experiences
  with physicians is not new.
• Whats new efforts to standardize these measures
  and potential widespread use.
• IOM report Crossing the Quality Chasm gave
  patient-centered care a front row seat.
• Methods and metrics have been honed through 15
  years of research and through several recent
  large-scale demonstration projects.
• But putting these measures to use raises many
  questions about feasibility and value.

20
Demonstration Projects to Measure Experiences
with Individual Clinicians and Their Practices
2001
2002
2003
2004
2005
HVMA QI (MA)
MHQP Statewide Pilot (MA)
IOM Chasm Report
PBGH (CA)
ICSI QI Collaborative (MN)
CMS DOQ Pilot
21
1st Generation Questions Moving MD-Level
Measurement into Practice

• What sample size is needed for highly reliable
  estimate of patients experiences with a
  physician?
• What is the risk of misclassification under
  varying reporting frameworks?
• Is there enough performance variability to
  justify measurement?
• How much of the measurement variance is accounted
  for by physicians as opposed to other elements of
  the system (practice site, network organization,
  plan)?

22
Physician-Level Reliability A Measure of
Concordance Among Patients
Good Reliability
Poor Reliability
0
1.0
0.7
0.85
0.5
Perfect agreement among a physicians patients
No reliable information Just noise
23
Sample Size Requirements for Varying
Physician-Level Reliability Thresholds
24
What Is the Risk of Misclassification?

• Not simply 1- ?MD
• Depends on
• Measurement reliability (?MD)
• Number of cutpoints in the reporting framework
• Proximity of score to the cutpoint

25
Risk of Misclassification Model with 3
Performance Categories
Pre-publication data Not available for
circulation
26
Risk of Misclassification Model with 5
Performance Categories
Pre-publication data Not available for
circulation
27
Variability Among Physicians (Communication)
MD Mean Score,
Number of Doctors
28
Variability Across Practice Sites (Communication)
Group Mean Score,
Group Mean Score,
Western Region
Eastern Region
Central Region
Eastern Region
25th-75th percentile range of group scores
Group Mean score
25th-75th percentile range of group scores
Group Mean score
29
Variability Among Physicians Within Sites
(Communication)
Site and MD Mean Score
Site A-1
Site A-2
Site A-3
Site A-4
25th-75th percentile range of MD scores
25th-75th percentile range of site scores
Site Mean score
MD Mean score
30
Allocation of Explainable Variance
Doctor-Patient Interactions
Pre-publication data Not available for
circulation
31
Allocation of Explainable Variance
Organizational/Structural Features of Care
Pre-publication data Not available for
circulation
32
Changing Rates of Preventive Care Processes,
1996-2001
33
Focusing on Improvement Identifying Priorities
Office Staff
Knowledge of Patient
Clinical Team
Relationship Duration
Percentile Rank Adjusted
Interpersonal Treatment
Integration
Patient Trust
Health Promotion
Visit-based Continuity
Communication
Organizational Access
Priority Improvements
Correlation to Measure of Willingness to Recommend
34
Larry Morrissey, MD Medical Director,
Stillwater Medical Group
35
Charles Hipp, MD President, Stillwater Medical
Group
36
Overview of the ICSI Project

• Collaborative quality improvement project
  between
• ICSI (Institute for Clinical Systems Improvement)
• Brings medical groups together to work on quality
  improvement through clinical guideline
  development and projects
• Action Groups take a specific area and target
  it for improvement
• Harvard Medical School CAHPS Team

37
Objectives of the ICSI Project

• Learn more about customer service
• Implement test survey to identify priorities for
  improvement
• Implement process improvements
• Assess and monitor impact

38
Richard Marshall, MDChief Medical Officer,
Harvard Vanguard Medical Associates (HVMA)
39
John Ingard, MD Chief of Internal Medicine,
Wellesley Practice, Harvard Vanguard Medical
Associates
40
MHQP Measuring Patients' Experiences with
Individual Physicians and Sites

• MHQP measures technical quality and patients
  experiences for groups, practice sites, and
  individual physicians

2002 2005 2006
Statewide project evaluated patients
experiences with individual physicians.
Practice site results to be publicly reported.
MHQP measures patients' experiences with
individualphysicians and practices (primary
care). Practice sites receive their results.
41
Initiation of the ICSI Project

• The idea of addressing these issues was well
  received.
• There was enthusiasm for the project among staff
  and providers.
• Needed to proceed in an a manner that did not
  generate significant costs to the group.
• Change always creates a bit of concern.

42
Stillwater Medical Group (SMG) Baseline Survey
Results (n164)
Access Getting Needed Care
Office Functioning Scheduling Visit Flow
Communication Interpersonal Care
Integration
Percentile Rank Adjusted
Preventive Care
Priority Improvements
Correlation to Measure of Willingness to Recommend
1/9/04
43
HVMA Identifying the Need to Improve

• Performance on CAHPS measures For some measures,
  practice ranked in the 40th percentile.
• Confirmed with focus groups, staff interviews.
• Began process of continuous measurement to track
  progress.
• Goal To be above the 75 percentile.

44
Harvard Vanguards Interventions to Improve
Patients Experiences

• Improve visit-based continuity by increasing the
  percentage of the time that patients see their
  own primary care physician when they come to the
  office for care.
• Improve visibility of clinical team by carefully
  structuring practices into teams with staff known
  to the patients.

45
Stillwater Medical Group Results of the
Intervention

• Priority aim
• Improve doctor communication scores
• Intervention
• 3-day doctor training course facilitated by
  American Academy on Physician and Patient (AAPP)
• 56 of 58 doctors attended in May 04
• Results
• Notable short-term improvement in question scores
  related to doctor communication
• Mixed results in question scores over the longer
  term
• Verbal support for Patient-centered care and
  need for improvement to help sustain and increase
  our business

46
HVMA Challenges at the Practice Level

• Some level of distrust regarding the validity of
  the results of the patient survey.
• Nurse Practitioners, who are key element of our
  health care delivery system, felt excluded by
  this survey.
• The major goal of the HVMA practice improvement
  initiative was to improve visit based continuity
  scores.   It was hard to promote this goal in an
  internal medicine department with a history of
  well-functioning teams and senior clinicians with
  large panel size.

47
HVMA Cost of Improvement Offset by Benefits

• Costs
• Survey costs
• Analysis costs
• Investment in practice change
• Benefits
• Better retention of patients
• Better retention of more satisfied staff
• Practice growth

48
Harvard Vanguard Medical Associates What We Have
Learned

• We learned a lot about what our patients think of
  their experience with us.
• We learned that our staff were also worried about
  our practice, and frustrated that their patients
  werent more satisfied.
• We learned how hard it is to change practice
  infrastructure and culture. Health care systems
  have complicated infrastructures that keep things
  running along quite conservatively.

49
Stillwater Medical Group What We Have Learned

• Everyone wants to provide good service, but we
  need to constantly work on improving.
• Our patients really like us and once they start
  to come here they keep coming back.
• Data is a powerful motivator, but it needs to be
  individualized to really impact providers
• We have raised awareness about this issue.

50
Advice from Stillwater Medical Group

• Ensure that the culture of the organization
  supports being Patient Centered. Thats
  critical to customer service.
• Stick with it. Culture change is a slow
  laborious process that requires passion,
  persistence, and a sense of proportion.

51
Advice from Harvard Vanguard Medical Associates

• Develop absolute clarity about the goal you want
  to achieve and the key changes you need to make
  to achieve that goal.
• Dont underestimate the investment in resources
  and time you will need to change your practice.
  There are many, many structural features of the
  care process that impede the physician-patient
  connection.

52
Harvard Vanguard Medical Associates Plans Going
Forward

• Continue ongoing survey of patients experiences.
• Extend survey to all disciplines in the practice,
  including specialty care.
• Continue use of patient feedback to monitor
  success of patient-centered care initiatives.

53
Stillwater Medical Group Plans Going Forward

• Creating a culture of quality project with ICSI
  to address the cultural issues that stand in the
  way of sustained results in improving customer
  service
• Continued use of survey to monitor our progress
  in customer satisfaction
• Creation of a patient advisory council

54
Summary of Key Points

• With sample sizes of 40-45 patients per
  physician, measures have achieved physician-level
  reliability of .7-.85.
• With a 3-level reporting framework, risk of
  misclassification is low except at the
  boundaries, where risk is high irrespective of
  measurement reliability.
• Individual physicians and practice sites
  accounted for the majority of system-related
  variance on all measures.
• Within sites, variability among physicians was
  substantial.
• Encouraging evidence regarding the feasibility of
  achieving improvements though serious and
  ongoing commitment of senior leadership appears
  critical.

55
Factors That Contribute to Measurable and
Sustained Improvement

• Leadership is committed and engaged
• Strategic goals are aimed at organizational
  transformation
• Internal communication and action are aligned
  with strategic goals
• Motivation through external rewards and incentives

56
Some Tips for a Successful QI Initiative

• Begin improvement work by
• Focusing on involving patients and families in
  redesign and improvement activities.
• Maximizing the quality of the work environment
  for clinicians and staff

57
Some Tips for a Successful QI Initiative
(continued)

• Follow with specific interventions
• Focus on care team communication
• Pre-visit preparation
• Improving the quality of decision making
• Information follow up, post-visit
• Shared care plans
• Continuous access to the information and healing
  connections (clinicians, group visits, on-line
  groups, support groups, etc) necessary to improve
  health outcomes

58
Need Help? Contact the CAHPS Survey Users Network

• Email cahps1_at_westat.com
• Phone [phone]
• Website www.cahps-sun.org

59
Dana Gelb Safran, ScD
Susan Edgman-Levitan, PA
Lawrence E. Morrissey, MD
Charles J. Hipp, MD
John Ingard, MD
Richard Marshall, MD
Harvard Vanguard Medical Assoc.
Stillwater Medical Group
60
Questions or Comments? Contact the CAHPS Survey
Users Network

• Email cahps1_at_westat.com
• Phone [phone]
• Website www.cahps-sun.org