Rafael Blanco

1
Evaluation of learning
Lewisham University Hospital London

• Rafael Blanco

2
How do we grade our trainees?
Advance

• Beginners

Intermediate
3
1. scope and duration of training 2.
institutional organization 3. program director
and faculty 4. facilities and resources 5.
educational program 6. scholarly activity 7.
consultant skills 8. evaluation 9. board
certification.

• Current training is not sufficient to allow
  proficiency by the end of residency for a given
  block.

Chelly et al.
Demonstrated a lack of confidence in PNBs by
residents.
4
To reach proficiencywhen? 90?

• Rosenblatt et al.
• Only 50 of the residents were able to perform
  the interscalene block without supervision after
  seven to nine blocks
• with more than 15 attempts, 87.5 of residents
  were autonomously successful

5
The relation between theoretical and practical
knowledge

• Cognitive elements
• Psychomotor elements
• Affective elements

explicit formal knowledge
practical aspects
personal skills
Interpersonal skills
Communication
6

Learning Manual Skills in Anaesthesiology Is
There a Recommended Number of Cases or
Anaesthetic Procedures?
Anaesthesia Analgesia
Volume 86(3), March 1998, pp 635-639Conrad,
Christoph Schupfer, Guido Wietlisbach,
MarkusGerber, Helmut
7
Tracking the early acquisition of skills
by trainees Anaesthesia (C) 2001
Association of Anaesthetists of Great Britain
IrelandVolume 56(10), October 2001, pp
995-998Harrison, M. J.
Period 1 year 100 medical students and 19
trainees No previous experience intubating
8
Tracking the early acquisition of skills by
trainees

• Advantages
• No requirement to perform an x number of
  procedures
• No requirement for a particular level of success
  
• Disadvantages
• Trainees collected the data
• 4 Different institutions
• Students were attached in 6 batched over the
  year

9
What are the markers of expertise?

• Flexibility
• Confidence

10
How do we assess trainees?

• Continuous assessment

Logbooks
Cusum
11
CUSUM
12
If success move along x axis If failure move
along y axis
20/3/07
27/3/07
2/3/07
20/3/07
29/3/07
6/3/07
20/3/07
29/3/07
12/3/07
16/3/07
22/3/07
17/3/07
22/3/07
18/3/07
22/3/07
22/3/07
18/3/07
22/3/07
18/3/07
27/3/07
18/3/07
18/3/07
27/3/07
27/3/07
18/3/07
13
Cusum

• is a target value set for the level
• of performance when the block is successful

1 success 0 failure
Three successes, follow by a failure and 2 more
successes
14
The Construction of Learning Curves for Basic
Skills in Anesthetic Procedures
Anesthesia Analgesia
Economics,Education and Health Systems
Research Volume 95(2), August 2002, pp 411-416
de Oliveira Filho, Getulio Rodrigues MD
There was a wide inter individual variability in
the number of procedures before attaining
acceptable failure rates. This suggests that
performance should be followed on at individual
bases
15
The Construction of Learning Curves for Basic
Skills in Anaesthetic Procedures (Spinal)
Succesful identification SAE first attempt in 569
procedures(82.70)
A second interspace used in 119 procedures
(17.29)
16
The Construction of Learning Curves for Basic
Skills in Anaesthetic Procedures (Epidural)
Successful identification epidural space at
interspace first chosen occurred in 275 patients
(79.94)
A second interspace was used in the remaining 69
cases ( 20.05)
17
Cusum analysis is a useful tool to assess
resident proficiency at insertion of labour
epidurals
Canadian Journal of Anaesthesia
2003 / 50 7 / pp 694-698

• Viren N. Naik , Isabella Devito,  Stephen H.
  Halpern

18
But

• No data

19
CUSUM
Define success
Axillary block with nerve stimulator
20
Designing a CUSUM charting

• The statistical way

21
Upward CUSUM

• At 0 procedures the value of CUSUM is 0

At the nth procedure Cn max (0, Cn-1 Xn- k )
Cngth implies unacceptable performance
22
Values Cn max (0, Cn-1 Xn- k )

• Xn is 0 or 1 for a binary procedure
• 1 failure
• 0 success
• K is the reference value
• H is the decision interval

23
IC and OC

• h is determined by specifying the in-control (IC)
  average run of length (ARL)of a CUSUM chart

Type I error (alpha)
IC-ARL is the average number of consecutive
procedures required for a CUSUM chart to cross a
decision interval despite performing at
acceptable level
1-Type II error (beta) or 1-false negative error
OC-ARL is the average number of procedures
performed before the CUSUM chart signals when an
individual is performing at unacceptable level
24
(No Transcript)
25
Lets reviseHow to calculate the values?

• CUSUM starts at 0
• Define acceptable and unacceptable failure rates
• Desire magnitude of alfa and beta errors ( Type I
  and Type 2)
• Calculate ho and h1

26
(No Transcript)
27
(No Transcript)
28
(No Transcript)
29
Advantages of CUSUM charts

• It works for individual and group observations
• Design to detect small shifts in performance
• We specify the target value
• It makes explicit the trade-off between
  sensitivity and false alarm
• It is objective and with visual appeal

30
(No Transcript)
31
(No Transcript)
32
In summary

• A structured regional anesthesia rotation
• A complete re-evaluation of the anesthetic
  training( compulsory R.A. blocks)
• Guidelines
• A training curriculum
• A group of proper trainers in R.A.
• Numbers
• Determination

33
Thanks