On the Role of Proficiency Testing Schemes in Laboratory Accreditation

1
On the Role of Proficiency Testing Schemes in
Laboratory Accreditation
2
ISO/IEC 17025 and PROFICIENCY TESTING
5.9 Assuring the quality of test and calibration
results The laboratory shall have quality
control procedures for monitoring the validity of
tests and calibrations undertaken. The resulting
data shall be recorded in such a way that trends
are detectable and, where practicable,
statistical techniques shall be applied to the
reviewing of the results. This monitoring shall
be planned and reviewed and may include, but not
be limited to, the following a) regular use of
certified reference materials and/or internal
quality control using secondary reference
materials b) participation in interlaboratory
comparison or proficiency-testing programmes c)
replicate tests or calibrations using the same or
different methods d) retesting or recalibration
of retained items e) correlation of results for
different characteristics of an item. NOTE The
selected methods should be appropriate for the
type and volume of the work undertaken.
3
EXTERNAL QUALITY CONTROL PROFICIENCY TESTING

Inter-laboratory comparison Target value and
standard deviation based upon results of
participants Proficiency testing Organizer
decides on target value and target standard
deviation
4
PROFICIENCY TESTING, WHY
The prime purpose is to allow participating
laboratories to become aware of unsuspected
errors in their work and to take remedial
action. The participant can make three
comparisons of performance - with an externally
determined standard of accuracy- with that of
peer laboratories- with its own past performance
5
ISO/IEC 17025 and PROFICIENCY TESTING
PLEASE NOTE Interlaboratory comparisons may be
designed for purposes other than proficiency
testing - validation of methods-
characterization of candidate reference
materials- self-assessment of a laboratory in a
performance test A precondition for selection is
that the scheme is operated according to ISO/IEC
Guide 43 and/or ILAC Guide G-13.
6
ISO/IEC 17025 and PROFICIENCY TESTING
Why mandatory PT activities ? 1) ILAC requires a
policy of the Accreditation Bodies related to
the participation of accredited laboratories in
PT. 2) ILAC makes recommendations 3) A National
Accreditation Body may transfer this
recommendation into its own policy, e.g. one PT
activity prior to gaining accreditation and one
activity relating to each major sub-area of major
disciplines of a laboratorys scope of
accreditation at least every four years where
applicable and where practical.
7
ISO/IEC 17025 and PROFICIENCY TESTING
Does any PT-scheme exist for the tests and
samples normally analyzed by the laboratory
? Sources - National accreditation
bodies - Peer laboratories already
participating - PT-organizers in the own
country - Internet e.g. www.eptis.bam.de
- http//www.ianz.govt.nz/aplac/documents/web_do
cs/APLACPT003.pdf - Accreditation
Quality Assurance journal
8
ISO/IEC 17025 and PROFICIENCY TESTING
9
ISO/IEC 17025 and PROFICIENCY TESTING

• PT-testing is not about passing or
  failing a test.
• PT-testing is about taking part and
  learning from the results
• One bad result does not make a laboratory
  bad. The challenge is to prevent that
  it repeats
• One good result does not make a laboratory
  supreme. The challenge is to repeat this
  performance
• Goal consistent satisfactory performance

10
ISO/IEC 17025 and PROFICIENCY TESTING
Auditors KEEP IN MIND

• PT participation done under daily conditions ?
• Performance over sevral years with same PT
  organizer
• PT operated in accordance to ISO/IEC Guide 43-1
• The scheme documentation and statistical
  protocol available ?

11
ISO/IEC 17025 and PROFICIENCY TESTING
Does any PT-scheme exist for the tests and
samples normally analyzed by the laboratory
? What IF NOT ?
- Analysis of blind samples - Analysis of spiked
samples - Analysis of multiple sub-samples -
Exchange of samples with other laboratories - Use
of more than one analytical method
12
WRONG OBJECTIVES
QC/QA and method validation ?. PT IS OFTEN
A WRONG CHOICE Auditor Check the method
validation !! For assessment of sources of
error ?.PT CAN BE A WRONG CHOICE Auditor
Check how and who did the PT measurement Because
their clients say so ?.BUT WILL THEIR
CLIENTS JUDGE ON BASIS OF 1 TEST ONLY
? Auditor Check which client asked for this
To get an impartial and independent view of
the laboratorys performance ?. PT DOES NOT
INDICATE THE REAL (DAY-BY-DAY)
PERFORMANCE Auditor Explain the PT
report Because they have no suitable
calibrants ?. POSSIBLY, BUT CREATING A NEW RM BY
PT IS A TIME-CONSUMING PROCESS Auditor
What did they do so far ?
13
Proficiency testing insight in the REAL
performance ?
Ref. R.Forman, Pittcon 2002
As PT test A B C D E Aroclor 1242 120 128
102 102 111 benzo(k)fluoranthene 210
0 0 0 210 chrysene 139 94 77
72 158 arsenic 138 106 91 103
112 cadmium 138 100 89 104 102
lead 132 91 99 112 115
As double-blinded study A B C D E Aroclor
1242 0 62 0 0 143 benzo(k)fluoranthene
0 55 119 0 55 chrysene
0 59 73 0 49 arsenic 84 55 99
57 61 cadmium 10 12 98 20 25
lead 27 15 88 16 34
14
ISO/IEC 17025 and PROFICIENCY TESTING

• RELEVANT DOCUMENTATION
• Planned action
• List of PTs in which is participated
• SOPs for selection participation,administration
  , evaluation and
  follow-up

15
ISO/IEC 17025 and PROFICIENCY TESTING
VALIDITY OF RESULTS OF PT The period of
validity of the result obtained by a laboratory
in a single round of a PT scheme is limited to
the time that the laboratory performed the test
!! If a laboratory achieves a satisfactory
result in a single round, the result shall not be
used to support a claim that the laboratory
obtained reliable data on any other occasion !!!
16
ISO/IEC 17025 and PROFICIENCY TESTING
17
ISO/IEC 17025 and PROFICIENCY TESTING
18
ISO/IEC 17025 and PROFICIENCY TESTING
19
ISO/IEC 17025 and PROFICIENCY TESTING
20
ISO/IEC 17025 and PROFICIENCY TESTING
CHECKLIST FOR AUDITORS

• Is the PT-scheme in which they participated
  relevant in view of the scope of
  accreditation ?
• Check
• The normal IQC and results thereof frequency,
  control charts, criteria, evaluation,
  remedial/corrective actions
• Type of material in daily measurement vs. IQC
  material vs. PT material
• Frequency of measuring routine samples of the
  type supplied as PT
• Measurand
• Measurand value in comparison with typical
  daily range
• Deadline for reporting (turn-around time in
  comparison with routine work)
• Who analyzed it (senior staff member vs.
  technician ?)
• (Non) availability of IQC material -what else
  for routine control ?
• (Non) availability of suitable calibrants - how
  method validation ?
• Number of participants
• Expected number of different techniques and/or
  number of techniques, similar as applied in
  the laboratory
• Sample dimensions comparable to daily work
  conditions ?
• How may the majority have measured it best
  measurement capability or routine?
• Frequency of this type of PT testing
• Results of previous PTs on same material

21
ISO/IEC 17025 and PROFICIENCY TESTING
DONT FORGET

• If the PT score is poor (z gt 2 or zgt3)
• Assess how the target uncertainty was
  determined compare this with e.g. the
  associated value predicted by the Horwitz
  equation Sometime target uncertainties are
  unrealistic small, set by metrologists under
  best measurement conditions, not taking
  instead of routine conditions
• Check your own measurement uncertainty

22
ISO/IEC 17025 and PROFICIENCY TESTING
STUDY THE OVERALL PERFORMANCE OF ALL
PARTICIPANTS Unsatisfactory where the
majority is satisfactory ?Unsatisfactory
where a significant number performed poorly
? Both indications of problems regarding
methodology ? then re-validate
criteria for satisfactory performance
incorrectly set ? then check performance
of analysts.
23
ISO/IEC 17025 and PROFICIENCY TESTING
IS IT JUSTIFIABLE TO EXCLUDE OUTLIERS ?
ISO-34 Clause 5.15.1 The reference material
producer should never rely entirely on a
statistical analysis of the characterization data
when assessing the property values of interest.
Outliers should not be excluded on purely
statistical evidence until they have been
thoroughly investigated and, where possible, the
reasons for the discrepancies identified.
Alternatively, the use of robust statistics may
be appropriate in some cases.
24
ISO/IEC 17025 and PROFICIENCY TESTING
DO WE NEED STATISTICS IN PT ? Yes - For
objective demonstration of performance - For
objective comparison of laboratories/ techniqu
es/methods - If PT material has to be upgraded
to RM No - If participants use results for
finding deficiencies - If fast turn-around times
are necessary
25
ISO/IEC 17025 and PROFICIENCY TESTING
Bad newsMany things in nature are not
normally distributed.Good newsMuch of what
is not normally distributed in biology would be
normally distributed if you took the logarithm
of each data item. Bad newsMuch of what is in
books about statistics has to do with normally
distributed data. Good newsRobust statistics
provide useful information even if applied to not
normally distributed data.
26
ISO/IEC 17025 and PROFICIENCY TESTING
WHY DO ACCREDITED LABS PERFORM NOT BETTER THAN
NON-ACCREDITED LABS?
BECAUSE THESE COMPARISONS BETWEEN
LAB-PERFORMANCES ARE BIASED !!! 1. Quality
conscious laboratories join PT as part of their
quality improvement strategy other, less
conscious laboratories are less likely to
participate. 2. Non-accredited laboratories that
perform badly may more likely to drop
participation in PT than accredited labs, who
need it to maintain their accreditation. So there
is an inclusion of only good non-accredited
laboratories with both good and bad
accredited laboratories !! 3. Accredited
laboratories may even have no motivation to
perform super goodresults are fit for the
(clients) purpose4. Accredited laboratories may
have much more pressure on routine performance
(large throughput) than non-accredited
laboratories
27
Thank you for listening