Internal QC Programs a quantitative approach

1
Internal QC Programs- a quantitative approach

• Graham Jones
• Chemical Pathology

2
Quality Terminology

• QA - Quality Assurance
• Planned, systematic actions providing confidence
  that a quality output will be produced
• Laboratory Procedures
• IQC - Internal Quality Control (QC)
• Procedures use to assess validity of results in
  real time, controls release of results
• QC material run with patient samples
• EQA - External Quality Assessment
• Procedures operated by an external agency which
  allow retrospective review of performance
• RCPA-AACB

3
Internal QA

• An Internal Quality Assurance (IQA) program is a
  tool to assist with maintenance of correlation
  between different analysers in the same
  laboratory.
• Our IQA program includes the following features
• frequent sample distributions (2-3 pairs per
  week)
• the use of patient samples to reduce matrix
  effects
• unknown values at the time of measurement
• a range of values to cover the measurement range
• result review occurring at some stage after
  analysis
• We have developed a spreadsheet application to
  allow user friendly review of results.

4
QC Issues

• Selection of material (matrix)
• Selection of levels (decision points)
• Setting of targets and ranges
• Decision on frequency (batch vs RA)
• Decision on number of QC samples (n)
• Response to out-of-range values
• Quality planning

5
What is the aim of QC?

• To ensure that results released are OK
• To identify when the assay is not performing
  satisfactorily
• Question
• How good do we need to be to be OK?

6
QC Variables

• Batch assays
• With run
• Accept or reject whole batch
• Random Access
• Hold results, run QC, then release results
• Run and release results, run QC
• If QC fails, may need to re-run previous results.

7
QC Design

• Bench Level
• Simple, reproducible procedure to ascertain
  acceptability of current functioning.
• Knowledge of basic procedures to correct problems
• Supervisor Level
• Knowledge of whole Quality Assurance
• Link effects of QC to patient results
• Set up and administer system

8
What does a result mean

• A single result means that the actual value has a
  95 chance of lying within /- 2SD of that result
  (SD of assay at that level)
• Performing replicates narrows the range within
  which the actual value is likely to lie.
  (Standard error of the mean, SD/vn)
• Applies to QC as well as patient samples

SEM
/- 2SD
9
Levy-Jennings Plots
3SD

• When in control
• Centered around Mean
• Only outside 2SD 5
• Outside 3SD lt1

2SD
Mean
-2SD
-3SD
TIME
10
Levy-Jennings Plots
3SD
2SD
Mean
-2SD
Shift in Mean
-3SD
Shift in SD
11
Assay failure

• Assay change by
• Change in Mean (A)
• Change in SD (B)
• Blunders
• Usual QC programs best at detecting shifts in
  mean

A
B
12
Interpreting Results

• One result 2 SD from the mean

3SD
2SD
Mean
-2SD
-3SD
13
QC results - Examples

• A QC result at 2 SD of assay performance
• 5 chance that the result reflects normal assay
  performance
• 50 chance that true result (mean) is above
  that value (ie assay has changed by gt2SD)

14
QC results - Examples

• A QC result at 10SD of assay performance
• 0 chance that the result reflects normal assay
  performance
• 100 chance that true result represents change
  in assay mean (ie assay mean has changed by gt7 SD)

15
QC - improving power to detect changes

• To improve the probability of detecting a change
  assay performance
• one QC further from the mean (eg gt3SD)
• see more QC values gt2SD
• Principle Certainty of a true change is
  increased by
• A result further from the mean
• More results showing the same deviation

16
Being sure that the assay has changed

• Many results a bit different
• One result very different

3SD
2SD
Mean
-2SD
-3SD
17
Multi-rules

• Looks at many results
• eg
• 1 x 3S (one value gt3SD from mean)
• 2 x 2S (2 values gt2 SD from mean)
• 4 x 1S (4 values gt1 SD from mean)
• All may indicate a significant shift in the assay
  mean
• Increases chance of finding an error.

18
Summary thus far

• Big Shifts
• Easy to detect
• High probability of detection
• Can use simple rules with few QC samples
• Little shifts
• Hard to detect
• Low probability of detection
• Use multi-rules with many QC samples
• Hard to detect shifts of less than 2.5 SD

19
Westgard - Quantifying QC

• Process of quantifying power of QC to detect
  changes
• Based on Levy-Jennings plots
• If the mean shifts by X, what chance is there
  that my QC process will detect the change
• or
• With my current QC process, what changes can I
  reliably detect

20
Terminology

• n - number of QC samples run together
• (commonly n2 for Chemistry, often n3 for
  Immunoassays)
• 1 3s - 1 QC result gt3 SD from the mean
• 2 2s - 2 QC results gt2 SD from mean
• 4 1s - 4 QC results gt 1 SD from mean
• Multi-rules can be within run (across material)
  or same material (across runs)
• 3of4 1s - 3 out of 4 gt1 SD from mean

21
Original Westgard Multi-rules
22
www.westgard.com/
23
Westgard - QC Quantitation
3SD
2SD
Mean
-2SD
-3SD
24
Power Function Graphs
P- ED
P- FR
25
Power Function Graphs
P- ED
P- FR
26
Power Function Graph (n2)
12s
12.5s
MR
13.5s
27
Power Function Charts (1 3s)
n8
n4
n1
28
Power Function Charts - Summary

• Allows quantitation of assay shifts which can be
  detected for various QC protocols
• Variables
• n, rules
• Shows probability of detecting change
• Can allow choice of QC protocols to detect
  certain errors

29
Shifts and Results

• An assay may produce results up to 2SD from the
  true value when working well.
• If the mean shifts by 2 SD, results may be
  produced up to 4 SD from the true value
• If we can detect a shift of 3SD 90 of the time,
  then when undetected, a 5SD error may be released
  2.5 of the time

2SD
30
QC Goals

• Do we want to identify statistically significant
  changes in assay performance?
• or
• Do we want to measure clinically important
  changes in assay performance?
• Eg is a 2 change in an amylase result worth
  fixing!

31
SydPath QC Protocol

• 4) Interpretation
• In response to the QC results, one of three
  options can be chosen
• 1. CONTINUE - continue without change
• 2. PAUSE - Stop releasing results -
  troubleshoot assay and continue when fixed
• 3. STOP - Stop releasing results - troubleshoot
  assay, rerun previous samples if assay changed
  (Re-run not required when problem is shown to be
  due to QC material).

32
SydPath QC Protocol

• If both QC in Range (lt2LSD) CONTINUE
• If either (of 2) QC ? 3LSD STOP
• If 2 (of 2, High and Low) QC ? L2SD STOP
• If 1 (of 2) QC ? 2LSD and lt3LSD - stop releasing
  results, repeat High and Low QC for that analyte
• If, of the repeat QC results
• Either QC ? 3LSD STOP
• Both QC ? 2LSD STOP
• Either QC ? 2LSD PAUSE
• Otherwise CONTINUE
• These rules have the power to cause a STOP 90 of
  the occasions when there is a shift in the assay
  of 2.8 x LSD and cause a PAUSE 90 of the
  occasions when there is a shift in the assay of
  2.6 x LSD.

33
SydPath QC

• Rerun Protocol
• Choose 5-10 previous samples measured since the
  previous in-control QC.
• Re-analyse once assay fixed.
• Compare original results and change in computer
  if results significantly different.
• If all samples re-run are different, re-run
  earlier samples until correct results obtained.
• Decisions about significant differences to be
  based on CAL Re-Run Protocol . Refer to CAL
  Result Change Assessment Chart (based on
  RCPA-AACP EQA Allowable Limits of Performance).
• If a change is required to a previously released
  result which may have been seen by a clinician
  1) Inform the requesting Doctor of any
  clinically significant change of results.
• 2) Enter the new result in place of the
  previous result and insert the following
  footnote
• "AAAAA Following re-analysis, result changed
  from previously reported value of BBB at CCCC
  on DD/DD/DD. EEEE informed of change in result."

34
Change Protocol
In the event of re-running an assay following the
suspicion of an analytical error, significant
changes must be changed in the computer and
notified to the requesting/treating doctor.
Changes equal to, or greater than those shown
below may be considered significant (these values
based on the RCPA-AACB Allowable limits of
performance). If in doubt, consult the
Pathologist or Senior Scientist.
35
The Reverse Process - Capability

• Identify Analytical Goals
• Agreed clinical targets
• RCPA-AACB Allowable Limits of Performance (ALP)
• CLIA targets
• Set up a QC program which has a high probability
  of detecting an error this size together with a
  low false-rejection rate.

36
Capability - Quantitative Tests
/- 2 SD

• Actual variation compared to
• clinically required variation
• SD of QC material compared
• to Allowable Limits of
• Performance (ALP)
• CpALP/SD

/- ALP
37
Capability
Cp 6
2.5 SD Shift
2 SD spread
Cp 4
Cp 3
ALP
38
Capability

• Good assays (capable) have an analytical
  performance (SD) which is much less than the
  clinically important change.
• This can be quantified as the Capability index
  CpALP/SD
• (ALPAllowable limit of Performance)
• gt6 great
• 4-6 OK
• lt4 poor

39
Capability in practice

• Capable assays can use different QC to less
  capable assays
• Eg
• Simpler rules
• Less QC runs
• Can also set tighter limits to correct a true
  change in the assay performance before it is bad
  enough to stop releasing results.

40
Capability

• Assays may be poor because of
• The laboratory
• The method
• The industry
• If
• Lab trouble-shoot
• Method consider changing method
• Industry wait!

41
Setting Ranges for QC

• Actual SDs are difficult to determine exactly
  (which results to exclude)
• Actual SDs change over time
• But need to put limits into instruments as mean
  /- 2SD to fixed sig figs
• Suggest
• Actual SD (ASD, measured SD)
• Limit SD (LSD, put into QC package)

42
Chemical Pathology

• Run-in new QC for urine and serum
• Set target ranges
• Assess power to detect errors of current process
• Future
• Review performance
• Target poor assays
• adjust n, rules, frequency

43
Summary

• Some aspects of QC can be quantified
• This process allows us to use appropriate QC
• can either
• Know how good we are
• or
• Aim to reach certain targets