Evaluating Educational Interventions

1
Evaluating Educational Interventions

• Paul Stevenson

2
Background

• Local evaluation of EBP training showed
• Skills Knowledge improved .
• ..but limited change in behaviour.

• Miller, G E The assessment of clinical
  skills/performance
• Academic medicine, 1990, 65, S63-7

3

• How can I make the training more effective?
• I had used the Kirkpartick Model for Evaluation

4
Kirkpatrick Model Learner Focused
Kirkpatrick Model

• Level 1 Learner reaction
• Level 2 Modification of learner attitudes
  perceptions,
• and learner acquisition of
  knowledge skills
• Level 3 - Change in learner behaviour
• Level 4 - Benefits to the organisation/patient
  resulting from learner performance

Level 1 Learner reaction Level 2
Modification of learner attitudes perceptions,
and learner acquisition of
knowledge skills Level 3 - Change in learner
behaviour Level 4 - Benefits to the
organisation/patient resulting from
learner performance
Kirkpatricks four levels of evaluation
(1967,1994), adapted by Barr et al (2000)
5
Kirkpatrick Model

• Tendency for declarative outcome measures
• Shows if the intervention has been effective
• but provides little information about why
• Learner Focused
• Fails to consider contextual/environmental
  variables
• View that training is solely responsible for any
  outcome
• One Directional
• Tendency to focus on the impact of the learner
  on their environment

6

• What alternative methods could be used?

7
Transfer of Learning

• Baldwin Ford (1988) transfer of training into
  practice.
• Three areas of influence on transfer
• Aspects of the course
• Characteristics of the learner
• Features of the work environment

8
Common Barriers

• The most commonly mentioned barriers to EBP are

• Learner
• Personal lack of interest
• Lack of skills
• Low self-efficacy
• Previous experience

• Training
• Appropriate content
• Delivery method

• Environment
• Organisational culture
• Insufficient time
• Lack of resources
• Lack of support from colleagues and management

Salbach et al, 2007 Pollock Legg, 2000 Jette
et al, 2003 McColl Smith, 1998 Freeman
Sweeny,2001 Retsas, 2000 Cranney Warren,
2001 Metcalf et al, 2001
9
Diffusion of Innovation
Diffusion of innovation model, Rogers (1995)
10
LET evaluation

• Lewins (1951) equation Bf (P,E)
• Behavior is a function of the person and the
  environment
• Adapting this to evaluate the impact of an
  educational intervention we get BC f (L,E,T)
  
• Behavioral change is a function of
• L Learner E Environment T- Training

Rather than focus solely on the impact of the
learner on the environment, we also measure the
impact of the environment on the learner.
11
Systems Approach

• System Integrated whole composed of several
  parts
• Look at the educational intervention holistically
• Look for relationships with other parts of the
  organisation (LD, HR, Clinical Audit)

12
Example Measures

• Learner
• Pre-post course evaluation form
• Test exercise/paper to show learning has
  occurred
• Likert/VAS measure of self-efficacy, desire for
  change and other psychological aspects of the
  learner.
• Post-course follow up has learning been
  retained, examples of how training has impacted
  on work, reasons/barriers if new learning has not
  been used.
• Monitor usage statistics such as Athens Usage

13
Example Measures

• Environment
• Floor Walking observe the working environment
  is there access to appropriate resources,
  space/time for staff to implement EBP.
• Dialog with Line Management Creates a two way
  exchange of information
• Integrate Library Services with the wider
  organisation If you are working in the same
  environment as your learners you will experience
  the culture/environment first-hand.

14
Example Measures

• Training
• Pre-post course evaluation form
• Test exercise/paper to show learning has
  occurred
• Post-course follow up has learning been
  retained, examples of how training has impacted
  on work.
• Peer-observation

15
Advantages

• Truly understand the wider impact of the
  training
• Identify and have a better understanding of the
  barriers which exist and can start to resolve
  them.
• Increases profile and importance of library
  services within organisation / departments

16
Further Information

• paul.stevenson_at_bradford.nhs.uk