Australian Masterclass

1
Australian Masterclass

• Sally Batley Deputy Director of Analysis,
• NHS Modernisation Agency (UK)
• Working in partnership with the Patient Flow
  Collaborative (Victoria AU)

2
So what are we going to cover

• Measurement for Improvement
• What is Statistical Process Control (SPC)
• Understanding Variation
• Benchmarking
• Build you own SPC charts

3
So what are we going to cover

• Measurement for Improvement
• What is Statistical Process Control (SPC)
• Understanding Variation
• Benchmarking
• Build you own SPC charts

4
Measurement for Improvement
5
Sir Josiah Stamp

• Public agencies are very keen on amassing
  statistics - they collect them, add them, raise
  them to the Nth power, take the cube root and
  prepare wonderful diagrams. But ...
• what you must never forget is that every one of
  those figures comes in the first instance from
  the village watchman (or admissions clerk?) - who
  puts down what he damn pleases.

6
There are three kinds of lieslies, damned lies
and statistics

• After Mark Twain

7
Collecting your data
8
How good is your data?

• Is the routine data you collect and distribute
  100 accurate?
• Is it complete rubbish?
• So it must be somewhere in between

9
Issues

• Definitions
• Accuracy
• Consistency
• Timing

10
The information vicious circle
Information is not used
Information is Inaccurate Incomplete Late Inconsi
stent
11
Task

• In groups you have to describe the people in the
  room so answer these questions
• How many people are there in the room?
• How many are wearing something red?
• How many are tall?
• How many types of footwear are there?
• Find one word to describe the group?

12
Issues

• Timing
• Definitions
• Accuracy
• Consistency

13
Data types
14
Types of data

• Routine v special collection
• Qualitative v Quantitative
• Soft v hard
• Descriptive v numeric
• Example of current performance
• Patients are satisfied v waiting time is 4
  hours
• Example of change
• Communication with patients has improved v
  Average X-ray waits reduced by 20 minutes

15
Which types are you collecting?
16
Types of measurement
17
Different types of measurements

• measurements for judgement
• league tables

18
Performance Indicators

• Measure probability not certainty
• Are better in groups
• Are better at identifying poorer performance
• Should not be used for league tables

19
Different types of measurements

• measurements for diagnosis
• to show where the problems are
• lots of measures
• comparative data useful
• measurements for improvement
• to show if improvement are being made
• linked to the project objectives and aims
• a few specific measures

20
Measurement for Improvement

• or how do we know
• that a change is an improvement?

21
(No Transcript)
22
project aims
23
project aims global measurements
24
project aims global measurements change
principles
25
Building Improvement Knowledge
Changes that result in improvement
Improvement
Time
26
Measurement for improvement

• Answers the question
• How do we know change is an improvement ?
• Is linked to the project objectives or aims
• usually requires no more than five to seven
  measures
• crosses the whole process of care
• measures change over time

27
Change areas, aims and measures should be related

• Area - Effective Delivery of Health Care
• Aim - To improve access to the appropriate
  treatment
• Measure - Reduce the number of days between
  referral and first definitive treatment

Example from Action On programme
28
Measuring quantitative outcomes
29
Measuring quantitative outcomes

• A descriptive goal
• eg reduce DNAs
• But by how much?
• Quantify the starting point (baseline)
• Set an objective (improve by x)
• How will you measure that? (methods)
• Monitor progress

30
An example - hospital cancellations
Baseline 15
Target 5
31
How are we doing?Setting the baseline

• Baseline period must be representative
• Small numbers issue
• Baseline period can be greater than monitoring
  frequency

32
Over what period to measure baseline?
Average 8.7
33
Over what period to measure baseline?
Average 8.7
34
How will we know?Tips on measurement

• Measurement periods
• Census point (particular time of day - eg 12pm)
• Period of time (eg 24 hour period)
• Dont mix the two!
• Use routine data where possible to allow
  cross-checking
• Specify method precisely
• eg process time in hours for patients from triage
  to admission onto appropriate ward

35
How much will we improve? - Expressing the
measurement of change

• Be realistic in your expectations
• Dont think you can reduce error rate from 50 to
  0
• Mostly express values to one decimal place
• DNA rate 5.6 (not 6)
• Express target as a value not as an improvement
• If baseline is 5 patients/hour and you want to
  improve by 10 then state target as 5.5
  patients/hour
• Avoid confusion over percentages
• Baseline is 10 and you want to improve (reduce)
  by 25 then state target as 7.5

36
Process Mapping

• Understand the process before settling on your
  measures

37
Route A - Self-referral
W4
W3
W1
Indicative waits W1 - 5 minutes W2 - by
category W3 - 1 hour W4 - 1 hour W5 - 4 hours
W5
W2
38
We want to improve the overall patient journey
Global measure patients seen within
recommended waiting times at three key identified
stages in care
39
But Changes are made at specific points
Global measure patients seen within
recommended waiting times at three key identified
stages in care
40
The Measurement Paradox

• We want to improve the whole patient experience/
  journey but we make changes at specific points.
  How do we cope with measuring the change?
• Specific measures
• can be temporary
• to monitor change ideas
• Global measures
• are permanent
• to monitor overall improvement

41
Measurement at specific points

• In addition to reported global measures
  plotted, additional measures may be required
  during changes
• specific measures related to the change
• results for sub-groups of patients
• results by consultant groups
• results for patients experiencing a particular
  clinical process

42
Impact of changes on global measures (hopefully!)
Average waiting times across the care
pathway in days
60
Change 1
50
40
Change 3
30
20
Change 2
10
0
Jul
Jan
Jun
Oct
Nov
Jan
Mar
Apr
Feb
May
Aug
Dec
Sept
43
Setting the baseline Or how are we doing right
now?

• Baseline period must be representative
• Watch out for small numbers!
• Baseline period can be greater than monitoring
  frequency

44
Measurement guidelines

• key measures plotted and reported each month
  should clarify your project teams aim and make
  it tangible.
• be careful about over-doing process measures.
• consider sampling to obtain data.
• integrate measurement into the daily routine.
• plot data on the key measures each month during
  the programme

45
Task Creating measures for your project aims

• Your Project is Improving Patient Flow
• what is your measurement strategy?
• what are you aims
• what quantified measures could be used?
• Data collection method
• what baseline are you going to use?
• what is the potential performance?
• frequency of measurement?
• How are you going to feed it back and to whom?

46
Patient experience monitoring
47
Why?

• To use patient feedback to improve services

48
Agenda

• evaluating patient experience
• quantitative versus qualitative
• rating versus reporting
• practical hints and tips

49
Task

• On your table, brainstorm ideas for measuring and
  monitoring patients experience with a service
• How can we measure what patients think of the
  service?

50
Approaches to monitoring

• quantitative
• structured
• questionnaires
• tick box
• surveys

qualitative

• semi-structured
• interviews
• questionnaires that
• combine tick box
• with comment spaces

• unstructured
• interviews
• patient focus groups
• critical incident
• technique

51
Task

• contrast quantitative and qualitative patient
  feedback approaches
• what are the advantages and disadvantages of
  each?
• in what circumstances would you choose to each
  approach?

52
Report experience dont rate satisfaction

• How satisfied were you with the consultation you
  received with the doctor?
• Please answer the questions by ticking the
  response which most closely matches your
  experience.
• All the treatment options were fully explained to
  me.
• I was given as much as much information as I
  wanted to know
• Treatment options were very briefly discussed
  with me
• The doctor did mention different treatments, but
  I did not really understand
• I did not feel that I was given a choice about
  treatment

X
53
Designing a questionnaire or survey

• goal of the research
• research method
• questionnaire design
• patient sample
• frequency of data collection
• data collection methods
• systems for analysis
• reporting systems

What do you want to know? How will you find
out? What sort of questions? How many will you
ask? How often will you ask them? How will you
ask them? How will you analyse the data? How
will you report the results and to whom?
54
Designing a questionnaire or survey

• keep it simple
• plain English
• small patient sample and track changes over time,
  little and often (run chart)
• combine quantitative and qualitative
• pilot first
• involve patient / user representatives in
  questionnaire design, data collection and
  analysis of results

55
Leave room for comments

• How satisfied were you with the consultation you
  received with the doctor?
• Please answer the questions by ticking the
  response which most closely matches your
  experience.
• All the treatment options were fully explained to
  me.
• I was given as much as much information as I
  wanted to know
• Treatment options were very briefly discussed
  with me
• The doctor did mention different treatments, but
  I did not really understand
• I did not feel that I was given a choice about
  treatment
• Add any other comments you wish to make in the
  box below

56
The power of a good quote
57
Back to you measurement strategy
58
So what are we going to cover

• Measurement for Improvement
• What is Statistical Process Control (SPC)
• Understanding Variation
• Benchmarking
• Build you own SPC charts

59
What do you know?
60
Task

• What do you know about the following
• Mean
• Variation
• Special causes
• Standard deviation

61
What is SPC?

• P is for Process
• We deliver our work through processes
• S is for Statistical
• because we use some statistical concepts to help
  us understand our processes
• C is for Control
• And by this we mean predictable

62
What is SPC for?

• A way of thinking
• Measurement for improvement - a simple tool for
  analysing data
• Better way for making decisions
• Evidence based management
• Easy, sustainable

63
What Can It Do For Me?

• To identify if a process is sustainable
• are your improvements sustaining over time
• To identify when an implemented improvement has
  changed a process
• and it has not just occurred by chance
• To understand that variation is normal and to
  help reduce it
• To understand processes - This helps make better
  predictions and improves decision making

64
What about this?
65
Where have we come from?

• Compare to some arbitrary fixed point in the past
• the average (median) waiting time of those on the
  list, at 2.97 months, fell slightly over the
  month, and remains lower than at March 1997 (3.04
  months).
• Show percentage change this month and to some
  arbitrary fixed point in the past
• the number of over 12 month waiters fell this
  month by 3,800 (7.4) to 48,100, and are now
  24,000 (33) below the peak at June 1998

66
Comparing this year to last year
67
Waiting time performance

• What can you tell me about the following data?

68
Is this better?
69
Or better still?
70
Common management reactions to data

• take 3 different numbers
• 6 possible ( random) sequences

71
3 points can give 6 possible ( random) sequences
72
Unacceptable decision-making

• Develop polite impatience with
• guesswork - single figure decision making
• shooting from the hip
• anecdotal data
• debate
• known solutions
• ?arbitrary targets and standards

73
What else can it do for me?

• Recognise variation
• Evaluate and improve underlying process
• is it stable? can it meet targets?
• Help drive improvement
• has the process really improved or is it just
  chance?
• is it sustainable?
• Prove/disprove assumptions and (mis)conceptions
• Use data to make predictions and help planning
• Reduce data overload

74
What is a control chart
Upper process limit
Mean
Lower process limit
75
So what are we going to cover

• Measurement for Improvement
• What is Statistical Process Control (SPC)
• Understanding Variation
• Benchmarking
• Build you own SPC charts

76
What is Benchmarking?

• Benchmarking compares practice and performance
  across organisations in order to identify ways to
  improve
• It is in essence, the identification,
  understanding, dissemination and implementation
  of best practice

77
Benchmarking encompasses

• Regular comparison of aspects of performance
  (functions and processes) with different
  practitioners
• Identifying gaps in performance
• Seeking fresh approaches to bring about
  improvements in performance
• Following through with implementation of
  improvements
• Monitoring progress and reviewing the benefits

78
Why is Benchmarking important?

• Benchmarking can be used to improve the overall
  performance of organisations through sharing and
  developing different practices

79
What are the benefits of Benchmarking?

• Improving quality and productivity
• Improving performance measurement
• Learning from others and greater confidence in
  developing and applying new approaches
• Greater involvement and motivation of staff

80
Comparing performance of different people or
services
81
Measuring for judgement

• The minister has decided that prescribing aspirin
  for patients on the CHD register is a Good Thing
• Non-compliance will henceforth be a hanging
  offence
• But who to hang?
• He has been given the latest data on several
  Health Services

82
Whos doing well?
Gold stars to Health Services A B
Hanging for Health Services I, J K
83
Why not traditional?
84
Remember whos doing well?
Gold stars to Health Services A B
Hanging for Health Services I, J K
85
A different way of presenting it
86
Control limits added
87
So what are we going to cover

• Measurement for Improvement
• What is Statistical Process Control (SPC)
• Understanding Variation
• Benchmarking
• Build you own SPC charts

88
What is Variation?

• Everything varies - no two things are alike
• Recognising this is a start but not enough must
  understand its effect on customers and then
  manage it as appropriate

89
Task

• In pairs think of reasons why your journey
  driving to work may be delayed on a morning.
  Write on post its
• You have a few mins and well come back to this
  later.

90
Different Types of Variation

• Common Cause Stable in time therefore
  relatively predicatable
• For example traffic lights which hold us up today
  would probably hold us up in the next week.

91
Different Types of Variation

• Special Cause Irregular in time and therefore
  unpredictable.
• For Example a police convoy escorting a wide load

92
Practical interpretation of the Standard Deviation
93
3s and the Control Chart
UCL
Mean
LCL
6s
94
Reducing Variation

• Walter Shewhart - Statistician 1920s
• Bell Telephones every failure led to an
  alteration to the telephones.
• Good idea?
• Started to look at limits and Common Special
  Causes

95

• A phenomenon will be said to be controlled
  when, through the use of past experience, we can
  predict, at least within limits, how the
  phenomenon may be expected to vary in the future
• Shewart - Economic Control of Quality of
  Manufactured Product, 1931

96
Task

• Back to the Task-Journey to work
• Which are common causes of variation?
• And which are special causes?

97
My trip to work
tyre had puncture
Stopped by police for speeding
Accident on motorway
average
School holidays
Borrowed helicopter
COMMON CAUSE VARIATION - Points within the yellow
lines is variation you would expect - normal
variation of the process (my trip to work) E.G.
traffic lights, pedestrians, rush hour
98

• CONTROLLED VARIATION
• stable,consistent pattern of variation
• chance/constant causes

Upper process limit
Mean
Lower process limit
99

• UNCONTROLLED VARIATION
• pattern changes over time
• assignable/special causes

100
2 Ways to improve a process

• If controlled variation
• process is stable and predictable
• variation is inherent to process
• therefore, process must be changed
• If uncontrolled variation
• process is unstable and unpredictable
• variation caused by factor(s) outside process
• cause should be identified and sorted

101
2 dangers to beware of

• Reacting to special cause variation by changing
  the process
• Ignoring special cause variation by assuming its
  part of the process

102
PauseThink of some examples in your own
area- Common cause variation- Special cause
variation
103
So what are we going to cover

• Measurement for Improvement
• What is Statistical Process Control (SPC)-the
  math
• Understanding Variation
• Benchmarking
• Build you own SPC charts

104
How to interpret the results
105
Rules for special causes
RULE 1
Any point outside one of the control limits
RULE 2
A run of seven points all above or all below the
centre line, or all increasing or all decreasing.
RULE 3
Any unusual pattern or trends within the control
limits.
RULE 4
The number of points within the middle third
of the region between the control limits differs
markedly from two-thirds of the total number of
points.
106
SPECIAL CAUSES - RULE 1
Point above UCL
X
UCL
UCL
X
X
X
X
X
X
X
X
X
MEAN
X
MEAN
X
X
X
X
X
X
X
X
LCL
LCL
X
Point below LCL
107
Rules for special causes
RULE 1
Any point outside one of the control limits
RULE 2
A run of seven points all above or all below the
centre line, or all increasing or all decreasing.
RULE 3
Any unusual pattern or trends within the control
limits.
RULE 4
The number of points within the middle third
of the region between the control limits differs
markedly from two-thirds of the total number of
points.
108
SPECIAL CAUSES - RULE 2
Seven points above centre line
UCL
UCL
X
X
X
X
X
X
X
X
X
X
MEAN
MEAN
X
X
X
X
X
X
X
X
X
X
X
LCL
LCL
Seven points below centre line
109
SPECIAL CAUSES - RULE 2
Seven points in a downward direction
UCL
UCL
X
X
X
X
X
X
X
X
X
X
X
X
MEAN
MEAN
X
X
X
X
X
X
X
X
X
LCL
LCL
Seven points in an upward direction
110
Rules for special causes
RULE 1
Any point outside one of the control limits
RULE 2
A run of seven points all above or all below the
centre line, or all increasing or all decreasing.
RULE 3
Any unusual pattern or trends within the control
limits.
RULE 4
The number of points within the middle third
of the region between the control limits differs
markedly from two-thirds of the total number of
points.
111
SPECIAL CAUSES - RULE 3
Cyclic pattern
Trend pattern
UCL
UCL
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
LCL
LCL
112
Rules for special causes
RULE 1
Any point outside one of the control limits
RULE 2
A run of seven points all above or all below the
centre line, or all increasing or all decreasing.
RULE 3
Any unusual pattern or trends within the control
limits.
RULE 4
The number of points within the middle third
of the region between the control limits differs
markedly from two-thirds of the total number of
points.
113
SPECIAL CAUSES - RULE 4
Considerably more than 2/3 of all the points fall
in this zone
Considerably less than 2/3 of all the points fall
in this zone
UCL
UCL
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
LCL
LCL
114
NOW FOR SOME MATHS!
115

• Use individual values to calculate the Mean
• Difference between 2 consecutive readings, always
  positive Moving Range, MR
• Calculate the Mean MR
• One standard deviation/sigma (Mean MR) d2
• s or s
• Upper Process Limit (UPL) Mean 3 s
• Lower Process limit (LPL) Mean - 3 s
• d2 is a constant for given subgroups of size n
  (n 2, d2 1.128)
• H.L. Harter, Tables of Range and Studentized
  Range, Annals of Mathematical Statistics, 1960.

116
Construction and Interpretation of (X, Moving R)
Chart
Run chart, running record, time order
sequence Calculate the mean Calculate upper and
lower process limits Interpret the chart for
process control Find the causes of real change
act to improve
117
Calculation of the mean
X19 X20
X1 X2 X3 X4 X5 X6 X7 X8
1.5 2
5.9 0.4 0.7 4.7 2 1.3 0.8 0.7
Mean
X1 X2 X3 X4 X5 X6 X7
X8 X19 X20
20
5.9 0.4 0.7 4.7 2
1.3 0.8 2
20
Mean 2.545
S means sum of
50.9
S
X
X

n
20
n number of results
SPC 33
118
Calculation of mean moving range
R19
R18
R1 R2 R3 R4 R5 R6 R7 R8
5.5 0.3 4 2.7 0.7 0.5 0.1 1.8
0.8 0.5
Moving Range
R1 R2 R3 R4 R5 R6 R7
R8 R19
19
5.5 0.3 4 2.7 0.70.5 0.1 1.8 0
0.8 0.7 3.7 0.5 3.8 0.1 0.2 0.6 0.8
0.5
19
27.3
19
S
R
27.3
MR

1.437
n
19
S means sum of
n number of moving ranges
SPC 33
119
Calculation of s 1 standard deviation
s
Never use the standard deviation key on a
calculator to get this figure
Calculate
From the formula
R

d2
1.437
1.128
1.274
d2 is always 1.128 for a sample size of 2
(difference between 2 values)
SPC 33
120
Calculation of control limits
Calculate UCLX (Upper Control Limit) for X
X 3 0
2.545 3.822
6.367
Plot on graph
Calculate LCLX (Lower Control Limit) for X
X - 3 0
2.545 - 3.822
-1.277 cant have negative so take to be 0
Plot on graph
SPC 33
121
And thats how you get one of these! - a Control
Chart
Upper process limit
Mean
Lower process limit
122
(No Transcript)
123
Things to remember

• only need 20 data points to set up a control
  chart
• standard deviation
• this is not the one used in formulae in Excel or
  on calculators.
• d2 constant
• sample size of 2 refers to the sample size for
  moving range (which is nearly always 2) - NOT the
  number of data points
• 20 data points produces 19 moving ranges

124
Remember the 2 ways to improve a process

• If controlled variation
• process is stable
• variation is inherent to process
• therefore, process must be changed
• If uncontrolled variation
• process is unstable
• variation is extrinsic to process
• cause should be identified and treated

125

• CONTROLLED VARIATION
• stable,consistent pattern of variation
• chance/constant causes

Upper process limit
Mean
Lower process limit
126
Remember the 2 ways to improve a process

• If controlled variation
• process is stable
• variation is inherent to process
• therefore, process must be changed
• If uncontrolled variation
• process is unstable
• variation is extrinsic to process
• cause should be identified and treated

127

• UNCONTROLLED VARIATION
• pattern changes over time
• assignable/special causes

128
DEFINING LACK OF CONTROL

• A single point falls outside the 3-sigma control
  limits
• 2 out of 3 successive values fall on the same
  side of, and more than 2-sigma units from the
  central line
• 4 out of 5 successive values fall on the same
  side of, and more than 1-sigma unit from the
  central line
• 8 (or 7??) successive values fall on the same
  side of the central line, or all increasing or
  all decreasing

129
Variation
We live in a world filled with variation - and
yet there is very little recognition or
understanding of variation
WILLIAM SCHERKENBACH
130
So what are we going to cover

• Measurement for Improvement
• What is Statistical Process Control (SPC)
• Understanding Variation
• Benchmarking
• Build you own SPC charts

131
SPC Spreadsheet Formulae
132
Example Data Set
Table 1. Shows what the data should look
like. Table 2. Shows how the formula should
look. Average, Lower limit and Upper limit should
only have the formula in the first row and the
value pasted for the entire dataset.
133
Example SPC Chart
Within this process Trust x could expect to see
between 0 and 58 admitted Inpatients per day,
with and average of 22. Therefore, there needs to
be 58 inpatient beds available everyday to match
current demand.
134
Task

• Split into equal groups around each laptop
• At least one analyst in each
• Let someone use the computer who is not use to
  working with excel
• Others can coach them on how to use it
• You have a data file on your computers called
  example.xls
• Compose a SPC chart and feedback

135
Thats all Folks !!!Any Last Questions?
136
Useful references

• Donald Wheeler. Understanding Variation.
  Knoxville SPC Press Inc, 1995
• Walter A Shewhart. Economic control of quality of
  manufactured product. New York D Van Nostrand
  1931.
• American Society for Quality www.asq.org/about/his
  tory/shewhart.html
• WE Deming. Out of the crisis. Massachusetts MIT
  1986
• Donald Wheeler. Advanced topics in statistical
  process control. The power of Shewhart's charts.
  Knoxville SPC Press Inc, 1995
• Donald M Berwick. Controlling variation in health
  care a consultation from Walter Shewhart. Med
  Care 1991 29 1212-25.