Understanding the measurement of the quality of diabetes care using the DARTS dataset: implications

1
Understanding the measurement of the quality of
diabetes care using the DARTS dataset
implications for the new GP contract

• Bruce Guthrie
• Tayside Centre for General Practice and on behalf
  of the DARTS/MEMO collaboration

2
Introduction and background

• NHS RD/CSO funded project recently started
  examining the purpose and validity of quality
  measurement
• Quality measurement increasingly common in public
  services, and linked to NHS modernisation agenda
• The new GP contract as a framework for discussion
  today using the DARTS clinical database to
  explore some challenges of quality measurement at
  practice level
• Complete data on HBA1c and cholesterol
  measurement for 9064 patients registered with 67
  Tayside practices (prevalence 2.25)

3
Properties of a good quality measure

• Is fit for its purpose or purposes
• Measures something that is important
• Robust enough to reliably and fairly distinguish
  good and bad performance (eg chance effects
  and case-mix)
• Attributable and responsive
• Avoids perverse incentives resistant to gaming
• Timely, usable, feasible and cheap enough

4
Distribution of contract quality measures for 67
Tayside practices in 2001
5
Chance effects and apparent variation

• All measures are subject to chance effects,
  especially with small numbers
• No difference in mean performance by practice
  size
• Smaller practices have greater apparent
  variability within each year
• Risk of over-interpretation of this kind of data

6
Chance effects and the new contract

• Smaller practices have greater variability in
  measured performance between years
• Under the new contract, this may translate into
  variability in income, and so hamper planning

7
Variation and chance effects

• Conventional league table with 95 CI
• On the face of it, considerable variation
• 9 practices better than average, 6 worse
• Presentation exaggerates variation
  intra-cluster correlation coefficient 3.9

8
Control charts

• Cross sectional control chart with 3 sigma
  control limits
• Identifies most variation as due to chance
• Only identifies 1 practice as better than
  average and 3 as worse

9
Chance effects require analytical choices

• League table with 3 SE confidence intervals
  (99.3 CI)
• Identifies same practices as different as the
  control chart
• The two techniques wont always agree, and
  neither account for hierarchy or clustering in
  the data
• Statistical significance chosen should be fit for
  the purpose of measurement

10
Adjusting for case-mix (age, sex and SES)

• Case-mix adjustment routine in national
  indicators
• Maximum differences in measured quality of
• 3 for HBA1c measured 4 difference in payment
• 7 for cholesterol measured 11 difference in
  payment
• 3 for HBA1c ? 7.4 14 difference in payment
• 5 for cholesterol ? 5 13 difference in
  payment
• May be mitigated by contract exclusions
• What should be adjusted for?
• May make true inequity less visible
• May make for fairer comparison, but greater
  complexity and opaqueness may reduce
  effectiveness of an incentive system needs to
  be fit for purpose

11
Attribution and responsiveness

• Large variation in the proportion of patients
  attending hospital
• The new GP contract lacks incentives for
  integrated care
• Most quality measures dont recognise the
  interdependency of general practice and hospital
• Whats the right level to measure quality of
  diabetes care at? Needs to be fit for purpose

12
Potential perverse incentives and unintended
consequences

• Exclusions likely to variably interpreted
• Focus effort on those near outcome thresholds
• Refer more patients to hospital
• Differences in how incentives work in different
  kinds of practice eg prevalence
• Other important areas may be ignored
• Unknown effects on trust and morale

13
Conclusion

• Quality measurement is complex because
• Quality is a contested concept and
  measurement serves many purposes
• There are methodological problems with measures
• New GP contract quality framework is a clearly
  reasoned and rational approach striving to
  achieve multiple aims, but it is hard to predict
  the equity or effectiveness of such a complex
  system
• Evaluation will need data from a large,
  representative sample of practices to allow
  analysis of quality under different assumptions
• A unique opportunity to measure population
  quality for common diseases at different levels

14
(No Transcript)