ISOIEC 17043 Conformity assessment General requirements for proficiency testing

1
ISO/IEC 17043Conformity assessment General
requirements for proficiency testing

• Hong Kong Accreditation Service
• 9 September, 2009
• Daniel Tholen, M.S.

2
ISO/IEC 17043 Overview

• Standard and Project History
• Changes from ISO Guide 43 and ILAC G132007
• Scope/intent of document
• Introduction, definitions
• Technical requirements
• Management requirements
• Annexes

3
History of Standards for Proficiency Testing

• 1945-1982 Interlaboratory Comparisons
• Method validation
• Material verification
• 1964 PT (US medical CAP, 1968)
• 1984 ISO/IEC Guide 43 1st Edition
• 1993 IUPAC Harmonized Protocol
• 1997 ISO/IEC Guide 43-1 and 43-2

4
History of Standards for PT

• 2000 ILAC G13
• Management system from ISO/IEC Guide 25
• Technical requirements from Guide 43-1
• Replace should with shall
• Edit informative discussion in Guide 43
• 1997-2005 ISO 13528 on Statistical Methods for PT
  by ILC
• 2006 ILAC proposals
• Revise G13
• Start ISO 17043 development

5
2006 ILAC Proposals

• Revise G132000 (completed, 2007)
• Update Management Requirements
• Remove redundant requirements
• Add language for medical EQA/PT
• NWIP for Revision to Guide 43-1 43-2
• Use G13 as base document
• Use IUPAC 2006
• Reference ISO 13528

6
Background ISO/IEC 17043

• Work Proposal from ILAC, approved by CASCO 30 P
  members Approve, 2 P members Not approve
• CASCO solicited experts for Working Group 28
  (WG28)
• ILAC nominated 6 members to WG28
• 57 experts on WG28, 31 CASCO member countries and
  6 liaison organizations.

7
WG28 Americas, Asia and Africa

• Argentina
• Brazil
• Canada
• Columbia
• Mexico
• Peru
• South Africa
• Trinidad and Tobago
• United States

• Australia
• China
• India
• Indonesia
• Japan
• Malaysia
• Pakistan
• Singapore
• Thailand

8
WG28 Europe Liaisons

• Austria
• Belarus
• Czech Republic
• Denmark
• France
• Germany
• Ireland
• Italy
• Netherlands
• Spain

• Sweden
• Switzerland
• United Kingdom
• Liaisons
• BIPM
• Eurolab
• IEC
• ILAC
• UILI
• CODEX Alimentarius

9
WG28 Progress

• 3 years, 5 meetings, 3 ballots
• 1. December 2006 WD1 proposed
• 2. May, 2007 WD3 proposed
• WD3 sent to WG28 for comment
• 211 comments, no serious objections
• 3. January, 2008 CD proposed
• CASCO CD ballot March-June, 2008
• 49-0-6 ballot, 484 comments

10
WG28 Progress

• Recent meetings
• 4. September, 2008 DIS proposed
• ISO and IEC ballots Nov 08 Apr 09
• ISO vote 57-3-5
• IEC vote 19-2-7
• 400 comments
• 5. July, 2009, FDIS proposed
• Resolved comments

11
ISO/IEC FDIS 17043

• Ballot as Approved International Standard in
  October 2009, 2 month review
• Only editorial comments permitted
• If ballot is successful, no additional meetings
  required
• Approved Standard, early 2010
• If ballot is not successful, re-ballot FDIS
• 6th meeting required

12
Next Steps

• Approval of FDIS ?
• Publish International Standard
• Adoption by PT Providers
• Adoption/acceptance by Accrediting Bodies
• Extension of the ILAC MRA
• Revision of 17043 after 17025 is revised

13
Structural changes from G13 and Guide 43 parts 1
and 2

• Technical Requirements come before Management
  Requirements
• One part with 3 Informative Annexes rather than
  two parts
• A Types of proficiency testing
• B Statistical methods for proficiency testing
• C Selection and use of proficiency testing

14
17043 General Format

• Language and requirements in alignment with
  ISO/IEC 17025
• Management System Requirements conform with ISO
  PAS 17005 for content (ISO 9001)
• Structure not consistent with ISO 17005
• Option B not included
• CASCO neutrality Policy
• No reference to a particular means of recognition
  of competence

15
17043 Coverage

• Intended to apply to traditional PT
  (interlaboratory comparisons) for testing and
  calibration laboratories
• Applies also to non-traditional settings
• Inspection
• Sampling
• Sensory evaluation
• Personal competence

16
Content changes from Guide 43-1 and ILAC G13 -
2007

• Added text to cover PT for Inspection Bodies
  (non-destructive testing)
• Enhanced requirements to promote educational
  potential of PT.
• Management System Requirements consistent with
  ISO 90012008

17
Content changes from Guide 43-1 and ILAC G132007

• Added requirements that some tasks
• shall not be subcontracted (5.5.2)
• Planning PT scheme (4.4.1.2)
• Evaluating performance (4.7.2.1)
• Authorizing final reports (4.8.1)
• Add requirements for equipment similar to
  requirements in 17025
• Relate criteria for homogeneity and stability to
  the effect on evaluation of performance

18
Questions?

• History
• WG28 or next steps
• General format changes

19
ISO/IEC 17043 Title

• Conformity assessment General requirements for
  proficiency testing

20
ISO/IEC 17043 Title

• Conformity assessment General requirements for
  proficiency testing
• PT is conformity assessment activity, so PT
  providers are CABs per ISO 17011
• Title does not say providers

21
ISO/IEC 17043 Scope

• This International Standard specifies general
  requirements for the competence of providers of
  proficiency testing schemes and for the
  development and operation of proficiency testing
  schemes. These requirements are intended to be
  general for all types of proficiency testing
  schemes, and they can be used as a basis for
  specific technical requirements for particular
  fields of application.

22
Introduction Types of ILCs - PT

• evaluation of the performance of laboratories and
  monitoring laboratories continuing performance
• identification of problems in laboratories and
  initiation of actions for improvement
• establishment of the effectiveness and
  comparability of test or measurement methods
• provision of additional confidence to laboratory
  customers
• identification of interlaboratory differences
• education of participating laboratories based on
  the outcomes of such comparisons
• validation of uncertainty claims

23
Introduction Types of ILCs not PT

• evaluation of the performance characteristics of
  a method
• assignment of values to reference materials and
  assessment of their suitability for use in
  specific test or measurement procedures and
• support for statements of the equivalence of
  measurements of National Metrology Institutes
  through "key comparisons" and supplementary
  comparisons conducted on behalf of the
  International Bureau of Weights and Measurement
  (BIPM) and associated regional metrology
  organizations.

24
Normative References

• Documents indispensible for the application of
  ISO/IEC 17043
• ISO/IEC 170002004, Conformity assessment 
  Vocabulary and general principles
• ISO/IEC Guide 992007, International vocabulary
  of metrology Basic and general concepts and
  associated terms (VIM)
• indispensible for application does not mean it
  is followed in all cases not all PT is
  measurement

25
Informative References

• Documents considered as Normative but are
  Informational
• ISO/IEC 17011
• ISO/IEC 17025
• ISO 15189
• ISO Guide 34
• ISO/IEC Guide 98 (GUM)

26
Definitions what it is

• proficiency testing evaluation of participant
  performance against pre-established criteria by
  means of interlaboratory comparisons
• Quantitative qualitative
• Sequential simultaneous
• Single occasion continuous
• Inspection Sampling
• Data transformation and interpretation

27
Definitions what is provided

• proficiency testing scheme - proficiency testing
  designed and operated in one or more rounds for a
  specified area of testing, measurement,
  calibration or inspection
• proficiency test item - sample, product,
  artefact, reference material, piece of equipment,
  measurement standard, data set or other
  information used for proficiency testing

28
Definitions who is involved

• proficiency testing provider organization which
  takes responsibility for all tasks in the
  development and operation of a proficiency
  testing scheme
• participant laboratory, organization or
  individual, that receives proficiency test items
  and submits results for review by the proficiency
  testing provider
• NOTE In some cases the participant may be an
  inspection body

29
Definitions other parties

• coordinator - one or more individuals with
  responsibility for organizing and managing all of
  the activities involved in the operation of a
  proficiency testing scheme
• customer - organization or individual for which a
  proficiency testing scheme is provided through a
  contractual arrangement
• subcontractor organization or individual
  engaged by the proficiency testing provider to
  perform activities specified in this
  International Standard and that affects the
  quality of a proficiency testing scheme

30
Definitions statistical terms

• assigned value value attributed to a particular
  property of a proficiency test item.
• outlier - observation in a set of data that
  appears to be inconsistent with the remainder of
  that set
• NOTE - An outlier can originate from a different
  population or be the result of incorrect
  recording or other gross error.
• this makes it inappropriate to use the term
  outlier for a result outside the acceptance
  limit (e.g., z score gt3) we eliminated the
  term extreme result (Guide 43)

31
Definitions from VIM, with Notes

• metrological traceability - property of a
  measurement result whereby the result can be
  related to a reference through a documented
  unbroken chain of calibrations, each contributing
  to the measurement uncertainty (8 Notes)
• measurement uncertainty - non-negative parameter
  characterizing the dispersion of the quantity
  values being attributed to a measurand, based on
  the information used (4 Notes)

32
4.1 General

• The development and operation of proficiency
  testing schemes shall be undertaken by
  proficiency testing providers having competence
  to conduct interlaboratory comparisons and access
  to expertise with the particular type of
  proficiency test items. Proficiency testing
  providers or their subcontractors shall also have
  competence in the measurement of the properties
  being determined.
• NOTE ISO/IEC 17025 or ISO 15189 can be used to
  demonstrate the competence of a proficiency
  testing provider's laboratory, or the laboratory
  subcontracted to perform tests or measurements
  related to the proficiency testing schemes. ISO
  Guide 34 can be used to demonstrate the
  competence of producers of reference materials
  that provide proficiency test items.

33
4.2 Personnel

• Generally similar to ISO/IEC 17025
• Provider shall authorize specific personnel to
• conduct statistical analysis
• evaluate the performance of proficiency testing
  participants
• give opinions and interpretations and
• authorize the issue of proficiency testing
  reports.

34
4.3 Equipment, accommodation and environment

• Similar to ISO/IEC 17025 and ILAC G13
• Substantial rewording for clarity, no new
  requirements

35
4.4 Design of proficiency testing schemes

• 4.4.1 Planning
• 4.4.2 Preparation of PT Items
• 4.4.3 Homogeneity and Stability
• 4.4.4 Statistical Design
• 4.4.5 Assigned Value

36
4.4.1 Planning

• PT provider shall not subcontract planning
• The Plan shall include (a-u) (like G13)
• Reasonable precautions to prevent collusion
• the origin, traceability and uncertainty of
  assigned values
• Considerations for advisory group and technical
  expertise

37
4.4.3 Homogeneity and Stability

• Links criteria to effect on evaluation of
  performance
• No new requirements, lots of Notes
• References ISO Guide 34, Guide 35, 13528
• 4.3.6 In circumstances where homogeneity and
  stability testing is not feasible the provider
  shall demonstrate that the procedures used to
  collect, produce, package and distribute the
  proficiency test items are sufficient for the
  purpose of the proficiency testing.

38
4.4.4 Statistical Design

• No new requirements, new Notes
• Rewording for clarity
• More tomorrow

39
4.4.5 Assigned Value

• New section added at meeting 3 importance
• 4.4.5.1 The proficiency testing provider shall
  document the procedure for determining the
  assigned values for the measurands or
  characteristics in a particular proficiency
  testing scheme.
• This procedure shall take into account the
  metrological traceability and measurement
  uncertainty required to demonstrate that the
  proficiency testing scheme is fit for its
  purpose.

40
4.4.5 Assigned Value

• Note to 4.4.5.1 added, meeting 5
• NOTE Metrological traceability is not always
  possible or appropriate.

41
4.4.5 new requirement (calibration)

• 4.4.5.2 Proficiency testing schemes in the area
  of calibration shall have assigned values with
  metrological traceability, including measurement
  uncertainty.

42
4.4.5 New requirement (testing)

• 4.4.5.3 For proficiency testing schemes in areas
  other than calibration, the relevance, needs and
  feasibility for metrological traceability and
  associated measurement uncertainty of the
  assigned value shall be determined by taking into
  account specified requirements of participants or
  other interested parties, or by the design of the
  proficiency testing scheme.

43
4.4.5 New requirement (consensus values)

• 4.4.5.4 When a consensus value is used as the
  assigned value, the proficiency testing provider
  shall document the reason for that selection and
  shall estimate the uncertainty of the assigned
  value as described in the plan for the
  proficiency testing scheme.

44
4.5 Choice of method or procedure

• 4.5.2 Where participants are permitted to use a
  method of their choice, the proficiency testing
  provider shall
• have a policy and follow a procedure regarding
  comparison of results obtained by different test
  or measurement methods.
• be aware of which different test or measurement
  methods for any measurand are technically
  equivalent, and take steps to assess
  participants results using these methods
  accordingly.

45
4.6 Operation of PT Schemes

• No (few?) new requirements
• Much reorganization and rewording from G13
• 4.6.1 Instructions
• 4.6.2 Handling and storage
• 4.6.3 Packaging, labeling, distribution

46
4.7 Data analysis and evaluation of results

• Much reorganization and rewording from G13
• 4.7.1 Data analysis and records
• Stronger recommendation of robust methods rather
  than outlier detection
• 4.7.2 Evaluation of performance
• Shall not be subcontracted
• Shall include expert technical commentary (where
  appropriate ) bullet list a)-h)

47
4.8 Reports

• Notes and rewording for Clarity
• 4.8.1 Authorization of the final report shall
  not be subcontracted.
• 4.8.5 Bullet list for amended reports

48
4.8 Reports

• 4.8.2 Reports shall include the following unless
  it is not applicable or the provider has valid
  reasons for not doing so
• m) details of the metrological traceability
  and measurement uncertainty of any assigned
  value
• o) assigned values and summary statistics for
  methods/procedures used by each group of
  participants
• p) comments on participants performance
• t) comments or recommendations, based upon the
  outcomes of the proficiency testing round

49
4.9 Communications

• 4.9.1 The proficiency testing provider shall make
  detailed information available about the
  proficiency testing scheme. This shall include
• relevant details of the scope of the proficiency
  testing scheme
• any fees for participation
• documented eligibility criteria for
  participation
• confidentiality arrangements and
• how to apply.

50
4.9 Communications

• Other rewording and consolidation from other
  requirements
• 4.9.5. If the proficiency testing provider issues
  statements of participation or performance, they
  shall contain sufficient information to not be
  misleading.

51
4.10 Confidentiality

• Rewording and consolidation from other
  requirements
• No new requirements.

52
Traceability and Uncertainty

• 4.4.1.3 Planning the origin, metrological
  traceability and measurement uncertainty of any
  assigned values
• 4.4.5.1 Determining the assigned valuesshall
  take into account the metrological traceability
  and measurement uncertainty required to
  demonstrate that the PT scheme is fit for its
  purpose.
• 4.4.5.2 Calibration
• 4.4.5.3 Testing
• 4.7.2.2 Expert commentary on the performance with
  regard overall performance against prior
  expectations taking measurement uncertainties
  into account
• 4.8.2 Reports shall include details of the
  metrological traceability and measurement
  uncertainty of any assigned value

53
Metrological Traceability

• 3.15 metrological traceability
• property of a measurement result whereby the
  result can be related to a reference through a
  documented unbroken chain of calibrations, each
  contributing to the measurement uncertainty
• NOTE 1 For this definition, a reference can be
  a definition of a measurement unit through its
  practical realization, or a measurement procedure
  including the measurement unit for a non-ordinal
  quantity, or a measurement standard.

54
Metrological Traceability

• NOTE 7 The ILAC considers the elements for
  confirming metrological traceability to be an
  unbroken metrological traceability chain to an
  international measurement standard or a national
  measurement standard, a documented measurement
  uncertainty, a documented measurement procedure,
  accredited technical competence, metrological
  traceability to the SI, and calibration intervals
  (see ILAC P-102002).

55
Questions?

• Technical Requirements
• Assigned Value Requirements
• Metrological Traceability

56
Management Systems

• Generally consistent with ISO/IEC 17025
• Slight modifications for ISO 90012008
• Slight modifications for clarity
• Rejected other comments for consistency with ISO
  9001

57
5.5 Subcontracting

• 5.5 Subcontracting Services
• 5.5.1 When a proficiency testing provider
  subcontracts work, the proficiency testing
  provider shall demonstrate that the
  subcontractors' experience and technical
  competence are sufficient for their assigned
  tasks and that they comply with the relevant
  clauses of this International Standard and other
  appropriate standards.

58
5.5 Subcontracting

• 5.5 Subcontracting Services
• 5.5.2 The proficiency testing provider shall not
  subcontract the planning of the proficiency test
  scheme, the evaluation of performance, or the
  authorization of the final report.
• NOTE This does not preclude the proficiency
  testing provider utilising advice or assistance
  from any advisors, experts or steering group

59
5.8 Complaints and appeals

• 5.8 Complaints and appeals
• The proficiency testing provider shall have a
  policy and follow a procedure for the resolution
  of complaints and appeals received from
  participants, customers or other parties. Records
  shall be maintained of all complaints, appeals,
  investigations and corrective actions taken by
  the proficiency testing provider.

60
5.13.2 Technical Records

• 5.13.2.1 The proficiency testing provider shall
  retain records of all technical data relating to
  each proficiency testing round for a defined
  period these shall include, but may not be
  limited to
• results of homogeneity and stability testing
• instructions to participants
• participants original responses
• collated data for statistical analysis
• information required for reports and
• final reports (summary and/or individual).

61
5.13.2.1 Technical Records

• NOTE 1 It is recommended to retain sufficient
  information to establish an audit trail for the
  processing of results from proficiency testing
  rounds.
• NOTE 2 Technical records are accumulations of
  data and information which result from carrying
  out all proficiency testing activities. They can
  include forms, contracts, work sheets, work
  books, check sheets, work notes, sub-contractor
  reports, participant feedback, etc.

62
Annex A Types of proficiency testing

• Revised from introductory language in ISO/IEC
  Guide 43-1
• Discusses different types of laboratory
  examinations
• Quantitative
• Qualitative (including categorical and ordinal)
• Interpretive (datasets and information)

63
Annex A Sequential schemes

• Common in calibration and medical
• Uses reference result as assigned value.
• Caution about stability of PT item across
  different participants
• Includes measurement audits (independent
  evaluations).

64
Annex A Simultaneous schemes

• Most common, ongoing schemes
• Many ways to get assigned value
• consensus (determined after results are returned
  to PT provider)
• known value (determined before results are
  returned to PT provider)
• Reference values
• Blind (or double blind).

65
Annex A Simultaneous schemes

• Most common, ongoing schemes
• Provides further explanation of NOTES in the
  definition of PT
• Sequential schemes
• Simultaneous schemes
• Bulk interlaboratory
• Split level
• Split sample
• Partial process
• EQA

66
Annex B Statistical methods for proficiency
testing

• Reference ISO 13528 and 2006 IUPAC Harmonized
  Protocol
• Provide further discussion of statistical methods
  for qualitative and ordinal results
• Simple listing of categorical responses
• Dont take average of numeric ordinal data
• D, D, Z score, En, zeta
• ISO TC69/SC6 consulted

67
Annex C Selection and Use of Proficiency Testing

• Based on Guide 43-2.
• Separate sections for laboratories and for
  interested parties
• Accreditation bodies
• Regulatory agencies
• Laboratory Customers
• Very general guidance on recognition of
  competence and use of results
• Considerations for accreditation bodies to be
  addressed by revised ILAC P9

68
Questions?

• Management System Requirements
• Annexes
• Anything else

69

• Thank you