Health Uses of Cumulative Summation Charts Relevant to Beds and Herts

1
Health Uses of Cumulative Summation Charts
Relevant to Beds and Herts

• Rachael Williams
• 11th November 2005

2
CUmulative SUMmation Charts

• Developed during WWII
• Quality control test for munitions production
  lines
• Is process in control?

3
Method of CUSUM

• Plots the CUSUM value against the number of
  attempts
• Success decrease in CUSUM value
• Failure increase in CUSUM value
• Use for monitoring and providing early warning
  changes in performance
• Very similar to scoring in golf!

4
Scoring in golf

• Number of shots (observed) compared to par
  (expected)
• Better golfers lower scores
• BUT can also identify where things go wrong for
  a player by steep increase in score

5
(No Transcript)
6
A statistical approach to measuring the
competence of anaesthetic trainees at practical
proceduresI. G. Kestin, British Journal of
Anaethesia, 1995 (75 805-809)
A statistical approach to measuring the
competence of anaesthetic trainees at practical
proceduresI. G. Kestin, British Journal of
Anaethesia, 1995 (75 805-809)
7
Setting

• Anaesthetic Directorate Derriford Hospital,
  Plymouth, 1994-95
• Not practical to have observer assessing
  technique
• SHOs and Registrars asked to plot CUSUM
• Four practical procedures

8
Failure

• Failure well defined (11 possibilities)
• (Un)acceptable failure rates defined by
  consultants
• Based on whether further supervised training
  would be required
• Ranged from 10 to 40 based on procedure
• Trainees instructed how to use the graph
• Told to see Kestin if CUSUM crossed a boundary

9
Results

• Unacceptable failure rates in
• Procedure I 0/2
• Procedure II 0/4
• Procedure III 2/8
• Procedure IV 3/12
• Maximum number of attempts made before crossing a
  boundary 23

10
Registrar X

• Failure rate unacceptably high
• Particular problems after
• Taking exams (lack of practice)
• Working for 18 months in different environment
• Change in practice
• Trainee required 185 attempts to achieve and
  demonstrate satisfactory failure rate

11
Advantages of CUSUM

• Demonstrated unacceptable failure rate at much
  earlier date
• Hospital were previously unaware
• Trainee was given advice and supervision
• Demonstrated that success may not be maintained
• Highlighted problems of change in technique /
  lack of practice
• Demonstrated wide variation in the speed at which
  trainees become proficient

12
Limitations

• Relies on honesty of trainees
• Relies on trainees interpretation of success and
  failure
• Does not assess other important aspects (e.g.
  safety)
• Large number of procedures needed before
  acceptable success rate possible
• Due to too stringent definitions of failure?

13
Monitoring mortality rates in general practice
after ShipmanR. Baker, British Medical Journal,
2003 (326 274-276)
14
Shipman Inquiry

• Convicted of murder of 15 elderly female patients
  in January 02
• Review of death certificates from 74 onwards
• 236 excess deaths compared to local GPs
• 197 compared to England and Wales
• 176 compared with manufacturing districts
• 152 compared with Tameside (local health
  district)
• Greatest amongst elderly women
• Requirement gt routine monitoring system for
  mortality of GP patients
• Detecting illegal behaviour
• Help GPs to plan improved methods of care

15
But

• Would Shipman have been detected by monitoring
  excess deaths?
• Only if threshold for investigation was set
  relatively low
• This would increase the false-positive rate
• Is there an appropriate CUSUM alternative /
  variation?

16
Yes!!!

• Risk-adjusted sequential probability ratio test
  (SPRT)
• Complex name simple technique
• Very similar to CUSUM in look and use
• Different statistics going on in the background
• Shipman would have crossed threshold in 1985 (3
  years and gt130 excess deaths earlier than he was
  arrested)

17
Requirements of Monitoring System

• Clear priority and purpose
• High sensitivity and specificity
• Problems with small numbers
• False positives???
• Provide meaningful data
• Require minimum of expertise and resources to
  maintain
• Be acceptable to GPs and patients

18
Other Helpful Data

• Age
• Sex
• Cause of death
• Circumstances of death
• Adverse events / non-fatal outcomes
• Comparative datasets (e.g. other local GPs)

19
In Conclusion

• Range of monitoring purposes of individual level
  data over time
• Trainees and experienced practitioners
• Morbidity and mortality
• Perceived complexities and limitations should not
  prevent CUSUM / variations being a part of
  monitoring systems in the future