Meeting CLIA requirements for competency assessment using training software

1
Meeting CLIA requirements for competency
assessment using training software

• Michael Astion, MD, PhD, HTBE
• University of Washington
• Department of Laboratory Medicine

2
University of Washington
3
Overview

• CLIA and Competency
• Trends in Competency Assessment
• Using computer-based exams as an objective method
  for competency assessment
• Computer-based training
• Recent survey of how labs use training software
• Implementing training software a case-based
  approach

4
CLIA and competency

• Lab directors and technical supervisors must
  ensure labs have adequate systems for
• initial instruction
• continuing education
• competency assessment
• Grace period is over regarding CLIA enforcement

5
Competency Assessment under CLIA

• Employees assessed 1 or 2 times per year.
• Employees assessed for all significant work tasks
  
• Assessment methods and the definition of
  competence are determined by each lab.
• Competency assessment IS NOT proficiency testing!

6
Proficiency Testing and Competency Assessment are
not the same thing.

• Proficiency testing asks Is a laboratory as a
  whole competent to perform a test?
• Competency assessment asks is each individual
  that performs a particular test or task competent
  to perform it?

7
Methods of competency assessment and frequency of
use by clinical labs (Howanitz et al., Arch Path
Lab Med. 2000, 124195-202)

• Direct observation (88)
• Review of test or quality control results (77)
• Review of instrument maintenance (60)
• Written testing (52) - a more objective measure
• Other methods (21 includes computer-based exams)

8
Trends in Competency Assessment (1)

• Assess all testing phases, not just the analytic
  phase.
• Preanalytic. Examples
• Assess the ability to process a specimen.
• Assess the ability to handle common problems
  (e.g. mislabeled blood tube, mislabeled CSF
  specimen).
• Postanalytic. Example
• Assess whether staff are following read back
  procedures

9
Trends in Competency Assessment (2)

• Assess knowledge that applies throughout the
  testing process
• Personal safety
• Patient safety
• Compliance / Privacy

10
Trends in Competency Assessment (3)

• Moving beyond appearances.
• When regulations are passed (e.g. CLIA), the
  first 5 years are spent appearing to meet the
  regulation.
• After 5 years, organizations start to meet the
  spirit of the regulation.

11
Trends in Competency Assessment (4)

• Use of Computer-based exams or written exams
• Advantage An objective method
• Disadvantage It is a supplementary method for
  competency assessment (I.e., it is not a complete
  method)

12
Approaches to objective exams

• Computer-based exams
• Use the exams contained in computer tutorials
• Advantage Many products available
• But, documentation / statistics lacking
• Use products designed for competency assessment
• Good printouts, stats, but not many titles
  available
• Written Exams
• Purchase books of questions, and select questions
  relevant to your workplace
• Make up your own questions

13
The UW experience with computer-based competency
assessment

• 1996-98
• Disk-based program for microscopic urinalysis
• Nobody used it but us, but it gave useful results
  
• 1999- 2000
• Web-based UA competency assessment
• A few other labs also used it
• Web improves access and inter-lab comparison

14
Sample question (continued)
15
The answer to the question
16
Results of 1st study of UA competency (8 exams
between 1996-98 average of 58 techs per exam)
Astion et al., Clin Chem. 1999 45 757-770
17
Results of preliminary study of web-based
urinalysis competency assessment (Dec 1999)
18
Document showing individual performance on 1 exam
19
Current UW system for competency (2001 - present)

• www.medtraining.org
• Covers 23 areas of clinical lab services
• Active users
• gt 250 labs in the U.S
• gt 30,000 individual exams taken each semester
• Provides benchmarks used in QI projects see Kim
  et al., Clin Chem. 2004 50 753-755

20
There are 23 competency assessment topics covered
by the UW system

• Phlebotomy
• Spec Processing
• Spec Transportation
• Patient Safety
• Worker Safety
• General Chemistry
• Protein Electrophoresis
• Urinalysis
• Toxicology
• Autoantibody testing

• Hematology
• Body Fluids (CSF/Serous Coagulation
• General micro
• Gram stains
• Hepatitis, HIV
• Transfusion theory
• Transfusion therapy
• PPMP
• Others

21
Look and feel of the current competency
assessment software
A sample question from a specimen processing exam
22
Estimates of National CompetencyAve scores on 9
online exams from Semester 1, 2004.
23
Inter-lab comparison Mean score of 4 peer labs
on the Jan 2003 Safety exam (N gt 55 users per
lab)
Exam Score ()
Lab 1a
Lab 1b
Lab 5a
Lab 5b
Kim et al., Clin Chem. 2004 50 753-755
24
Summary of UW experience with computer- based
competency assessment We failed to get national
implementation for 10 years but have now
succeeded because

• Enforcement of CLIA regulations re competency
• Web makes testing and data collection easy
• Easy testing leads to many users
• Easy data collection leads to the ability to
  generate competency benchmarks

25
(No Transcript)
26
(No Transcript)
27
Abbott Instrument Specific Competencies

• Available at Launch
• ARCHITECT family (c8000, i2000sr, ci8200)
• AxSYM Plus
• CELL-DYN 3000 series (CD3700, CD3500, CD3200)
• CELL-DYN 1800
• IMx
• Aeroset
• Commander
• Coming Soon
• ABBOTT PRISM
• CELL-DYN Sapphire
• Checklists for all of our instruments

In development, not available in the U.S.
28
Computer-based training

• The traditional methods of instruction
• Supervised instruction
• Textbooks.
• Computer-based training is becoming more popular
  because of drawbacks to the traditional methods.

29
Disadvantage of traditional teaching methods

• Supervised Instruction
• Labs dont have the time!
• Difficult to obtain image library especially for
  perishable specimens.
• Teacher-to-Teacher variability
• IFA images fade
• Textbooks dull, poor image quality
• Neither method produces documentation of training

30
Characteristics of useful training software

• Covers the right material
• Concise (Less is More!)
• Includes exams
• Produces documentation of initial training,
  continuing education, or competency assessment.

31
Characteristics of useful training software
(continued)

• Accredited for continuing education credits
• not required but often an advantage
• Vendor allows a 15 - 30 day free trial to
  determine if the software is a good fit for your
  lab.
• Vendor supports the software
• beware Free software

32
Where to find computer tutorials for the
laboratory?

• Publishing companies
• Traditional (e.g. AACC Press, Lippincott)
• Internet-based (e.g. Medical Training Solutions)
• Universities
• IVD companies
• A search of the internet using common-sense key
  words will yield fruitful results.

33

• Univ. Washington educational software
• 2004 Patient Safety, Work Safety
• 2003 Intro to Lab, Specimen Processing
• 2002 Pediatric Phlebotomy, Advanced Venipuncture
  
• 2001 Body fluids (includes synovial fluid,
  amniotic fluid, semen, CSF, and serous fluids)
• 2001, 1994 ElectrophoresisTUTOR
• 2001, 1995 UrinalysisTUTOR
• 1999, 1994 GramStainTUTOR
• 1998 PhlebotomyTUTOR, MycologyTUTOR
• 1997 ParasiteTUTOR, MicroscopyTUTOR
• 1995 PeripheralBloodTUTOR

34
For labs that have purchased training software
how are they using it? Astion et al.
Characteristics of educational software use in
106 clinical laboratories. 2002, Am J. Clin Path.
118 494-500.
35
Methods

• 32-question survey to determine how labs have
  implemented Univ. Washington (UW) software
• Survey completed by 106 clinical labs (response
  rate 60) that purchased UW software
• 89 (84) of labs reported using the software
• The following results are from these 89 labs

36
Breakdown of the 89 labs that are using training
software
37
In the 89 labs that use the training software,
what personnel use the software?
38
In the 89 labs that use the training software,
how are lab personnel using the software?
39
In the 89 labs that use the training software,
what are the locations for software use?
40
A summary of the behavior of 89 labs that
actively use training software

• Labs of all sizes use training software.
• Labs use the software predominantly for lab
  employees and clinical lab science students.
  
• Lab employees use the software for a variety of
  reasons including initial instruction, continuing
  education and competency assessment.
• Many of the labs (46) have a dedicated training
  area for using the software.

41
Case Study of training software implementation
A large hospital lab that is reorganizing by
creating a core lab

• Core lab will consist of heme and chem.
• Dayshift chemistry staff will be decreased.
• Hematology will do all microscopy work.
• Lab will need to cross train 1 chem technologist
  and 3 heme technologists

42
Case Study (contd) Issues regarding
retraining the 4 technologists

• Chemistry tech with 20 years experience is
  rotating to nights
• Will be required to read Gram stains and STAT
  peripheral blood smears
• Has not done a Gram stain or blood smear in 20
  years!
• 3 hematology techs with 5-10 years experience
• Now being asked to perform urinalysis testing for
  the first time in many years

43
Educational software solutions exist for the
training problems in Case 1.

• A variety of publishers have quality offerings in
  these areas.

44
Case 1 illustrates some advantages of
educational software

• Minimizes the embarrassment of 1-on-1 training
• Allows the techs to learn at their own pace.
• On average, the software takes 90 min to use, but
  some techs will take less and some more.
• Decreases supervised training time!
• Supervisor will be used for higher level
  teaching. The software provides the basics.
• Provides a consistent teaching method to multiple
  technologists.

45
Other cases where software has been useful

• Lab director driven insane by repetitive teaching
  of protein electrophoresis interpretation regains
  sanity by using software for initial instruction
  of new techs.
• Using CE-accredited training software to provide
  CE to techs in a California lab system.
• Teaching basic phlebotomy to new trainees with no
  medical background
• Giving An Intro to the Clinical Lab to
  non-technical employees (e.g. billing) with no
  medical background

46
Summary

• Computer programs are used by many labs as
  supplements to traditional methods for training
  and competency assessment.
• Advantages of computer programs
• Reduce supervised training time
• Effective for the basics
• Consistent presentation of material
• Exams are more objective
• Produces useful documentation
• Facilitates inter-laboratory comparisons