Assessing the Teaching of Procedural Skills: Can Cognitive-Task-Analysis Add to our Traditional Teaching Methods?

1
Assessing the Teaching of Procedural Skills Can
Cognitive-Task-Analysis Add to our Traditional
Teaching Methods?

• Maura Sullivan, PhD, Adrian Ortega, MD,
• Nir Wasserberg, MD, Howard Kaufman, MD,
• Julie Nyquist, PhD, Richard Clark, EdD
• University of Southern California

2
Cognitive-Task-Analysis (CTA)
Extends traditional task analysis to capture
information about both the overt observable
behavior and the covert cognitive functions
behind it to form an integrated
whole. Schraagen, Chipman Shalin, 2000
3
Acquiring Expertise
Declarative vs. Procedural Knowledge
Cognitive Associative Autonomous
4
Expertise

• Expertise is highly automated
• Ericsson, et al. 2006 Gagne et al. 1993 Clark
  Estes, 1996
• Automaticity interferes with articulation
• Clark et al., 2007, Feldon, 2004, Crandall
  Getchell-Reitter, 1993,
• Chao Salvendy, 1994
• Learners need to be taught decision making
  strategies
• Ericsson et al., 2006 Clark et al. 2007

5
Expertise cont.

• Experts underestimate novice difficulty
• Hinds, 1999
• Experts not fully aware of 70 of own decisions
• Clark Elen, 2006 Feldon Clark, 2006
• Experts decisions can be put into training
  materials
• Merrill, 2002 Schaafstal et al. 2000

6
CTA Gains

• Overt observable behavior
• Deconstruct automated knowledge into concrete
  steps
• Covert cognitive functions
• Identification of decision points throughout
  procedure
• Options related to each decision point
• Ability to gain consensus amongst experts
• Provide residents with an advanced organizer

7
Purpose

• To determine if surgeons omit relevant steps and
  decision points when teaching a colonoscopy
• To determine if CTA can augment our traditional
  teaching methods

8
Methods

• Three experts videotaped teaching a colonoscopy
  followed by free recall
• Participated in CTA
• Procedural Checklist and Cognitive Demands
  Checklist created
• Videotape transcriptions transposed and omitted
  steps identified

9
Results
26-step Procedure 14 Decision Points
Surgeon A 61 43
Surgeon B 46 25
Surgeon C 50 38

• What to do

10
Results cont.
26-step Procedure 14 Decision Points
Surgeon A 61 (50) 43
Surgeon B 46 (30) 25
Surgeon C 50 (26) 38

• What to do
• How to do it

11
Conclusions

• Experts did not articulate all relevant steps
  while teaching a colonoscopy or during free
  recall
• CTA provides a means to capture omitted steps and
  decision points during traditional teaching

12
Limitations

• Small N
• of experts
• of cases observed
• No established Gold Standard
• No inter-coder reliability

13
Future Studies

• Quantify educational effectiveness
• Determine number of experts needed
• Determine number of cases required
• Establish reliability
• Accelerate the learning curve

14
Implications

• CTA can be used to develop more comprehensive
  curricula for teaching procedural skills
• Assures exposure to all steps/decisions involved
  in a procedure
• More complete training of residents

15

• mesulliv_at_usc.edu
• [phone]