Information literacy training: How can we know we are not wasting our time

1
Information literacy training How can we know we
are not wasting our time?

• Andrew Booth, Reader in Evidence Based
  Information Practice, ScHARR, University of
  Sheffield

2
A Valentines Day Present!
What line of argument/ piece of evidence would
you use to justify continuation of this service?"
Allan Sack-arine
The Chief Executive has just proposed that all
information skills training in your organisation
be scrapped and staff spend their time on more
productive activities instead.
3
Information skills training (IST) systematic
review (Brettle, 2003)

• RESULTS
• Wide range of study designs including randomised
  controlled trials. Most took place in US medical
  schools. Wide variations in course
  content/training methods. 8 used objective
  testing, 2 compared methods and 2 examined
  effects on patient care.
• Limited evidence to show training improves
  skills.
• Insufficient evidence to determine most effective
  methods of training
• Limited evidence to show training improves
  patient care.
• Further research is needed.

4
15
1 Million
14
500,000
13
250,000
12
125,000
11
64,000
10
32,000
9
16,000
What do we typically evaluate after an
Information Skills Training Session?
8
8,000
7
4,000
6
2,000
5
1,000
4
500
3
300
2
200
5050
1
100
A Whether chocolate biscuits were OK?
B Whether training was pleasurable?
D Whether it was better than Blood, faeces,
urine, gangrene, death?
C Whether heating/ lighting was OK?
5
Evaluation bypass

• We interpret mentions of biscuits or heat of the
  room as at least this means that there is
  nothing else more important to worry about
  (false positives).
• ? ? ?
• Could mean that this is what they consider a
  successful outcome from the session OR
• We have designed a good instrument for assessing
  their physical comfort!

6
Why evaluate your training?

• To help decide whether techniques and methods
  used are worthwhile and making a difference
  (evidence based library information practice)
• To help understand whether we are making best use
  of our resources
• To demonstrate need for further funding or a
  redistribution of funding
• To help us improve our service
• To help redesign materials and methods
• Towards assessing library performance

NLH Librarians DL-Net
7
How to evaluate?

• We should evaluate
• The right population
• The right intervention
• The right comparison
• The right outcomes using
• The right outcome measurement tool at
• The right measurement interval

8
The right Population?
9
15
1 Million
14
500,000
13
250,000
12
125,000
11
64,000
10
32,000
9
16,000
Who are the right population when evaluating an
Information Skills Training Session?
8
8,000
7
4,000
6
2,000
5
1,000
4
500
3
300
2
200
5050
1
100
B Those required to attend?
A Students?
C Volunteers?
D Guinea Pigs?
10
And what about non-users how do we reach them?
11
15
1 Million
14
500,000
13
250,000
12
125,000
11
64,000
10
32,000
9
16,000
Who determines the design of our Information
Skills Training Sessions?
8
8,000
7
4,000
6
2,000
5
1,000
4
500
3
300
2
200
5050
1
100
A Our most able users?
B Our least able users?
C We decide what is best for our users?
D Our average users?
12
One size does not fit all

• Training courses designed around average user
  (the one with 2.4 children!)
• At any point in time almost half our users are
  getting more than they want and other (almost)
  half are getting stuff they already know
• They either decide what they need (and they are
  notorious for not being able to self-assess their
  competency e.g. beginners, intermediate,
  advanced) OR we decide what is best for them

13
Not having information skills training can be..

• good because you have to learn the hard way.
  You tend to have better skills if you have had to
  find your own way around - but it would've been
  nice to have been eased into it, in a bit more
  structured way as well
• Goodall, D. L. and P. Brophy (1997). A comparable
  experience? library support for franchised
  courses in higher education. British Library
  Research and Innovation Report 33. Preston,
  CERLIM, University of Central Lancashire viii,
  233p.

14
The right Intervention

• Most training is stand-alone and/or opportunistic
• Often at beginning of courses or included in
  inductions
• We have no idea what is right dosage!
• Usually in classrooms/laboratories away from
  workplace
• Tension between real-life examples and examples
  that work

15
If information skills training was a drug it
would never get licensed!

• But at least we arent doing any harm..

16
Can we be sure?

• Concept of opportunity cost what could we be
  doing instead? What could they be doing instead?
• If added value of our training is close to zero
  is it worth doing? Is it ethical to keep on
  doing it if we dont know that it works?
• Are librarians partly/totally responsible for
  anxieties regarding information
  overload/information explosion?
• Might our teaching them the proper way to
  search actually impair their effectiveness if
  all roads lead to Rome (or Google)?

17
Effective Methods for Teaching Information
Literacy Skills to Undergraduate Students
(Koufogiannakis Wiebe, 2006)

• METHODS
• To assess which library instruction methods are
  most effective at undergraduate level
• From 257 full articles of 4356 citations
  retrieved , 122 unique studies underwent data
  extraction and critical appraisal. 55 met quality
  criteria. 16 provided sufficient information for
  meta-analysis.79 studies (65) used
  experimental/quasi-experimental methods. Most
  focused on traditional teaching, followed by
  computer assisted instruction (CAI), and
  self-directed independent learning. Outcomes
  correlated with Blooms lower levels of learning
  (Remember, Understand, Apply).

18
Effective Methods for Teaching Information
Literacy Skills to Undergraduate Students
(Koufogiannakis Wiebe, 2006)

• RESULTS
• Traditional vs no instruction (12/16 found
  positive outcome). Meta-analysis of 4 studies
  favoured traditional instruction. 14 studies
  compared CAI vs traditional instruction with
  neutral result confirmed by meta-analysis. 6
  compared self-directed independent learning with
  no instruction, with positive result confirmed by
  meta-analysis.
• CAI is as effective as traditional instruction.
• Traditional instruction and self-directed
  independent instruction are more effective than
  no instruction.
• Future research needs to compare active learning,
  computer assisted instruction, and self-directed
  independent learning.

19
15
1 Million
14
500,000
13
250,000
12
125,000
11
64,000
10
32,000
9
16,000
What is the best comparison for one of our
Information Skills Training Sessions?
8
8,000
7
4,000
6
2,000
5
1,000
4
500
3
300
2
200
5050
1
100
A A placebo?
B Doing nothing?
C Other forms of education?
D Doing their searches for them?
20
The right Comparison

• What is right comparison for our information
  literacy training?
• Do we just assume that the importance of
  information skills training is self-evident?
• Is this why we now call it information literacy?

21
Information skills training versus mediated
searching

• To compare effectiveness and costs of mediated
  searches and information-skills training.
• Using satisfaction and use of skills both
  mediated searches and information skills training
  are effective. Cost-effectiveness depends on
  whether costs are viewed from a library or trust
  point of view. Providing information skills
  training does not reduce volume of
  mediated-search requests.
• Neither method more effective/cost-effective than
  other. Decisions cannot be made on effectiveness
  or costs alone views of library staff and
  professionals should also be taken into account.
  A proactive approach and targeting training
  towards those most likely to benefit may be
  appropriate.

Brettle A et al (2006) HILJ 23 (4), 239247
22
The right Outcomes
More meaningful measures
Attitudes
Outcomes (Health)
Knowledge
Behaviour
More difficult to establish cause and effect
Requires more robust designs
23
Information literacy outcomes - questionnaires

• Knowledge Which of the following best describes
  the Cochrane Library?
• Attitudes Rank the following databases in the
  order you are most likely to use them
• Behaviour How many times have you used PsycLit
  since the training session?
• Outcomes Please give an instance where your use
  of the Cochrane Library has impacted on patient
  care

24
Information literacy outcomes other methods

• Knowledge Objective Structured Clinical
  Examination (OSCE)
• Attitudes Likert scales
• Behaviour Observation, Transaction Logging
  Software
• Outcomes Observation, Patient Notes, Hospital
  Records, Critical incident technique

25
From output to outcome to impact
Training received
Changes to clinical decisions
Patients getting better
26
What is impact?

• any effect of a service, product or other event
  on individual or group. It
• may be positive or negative
• may be what was intended or something entirely
  different
• may result in changed
• attitudes
• behaviours
• outputs (i.e. what an individual or group
  produces during or after interaction with the
  service)
• may be short or long term
• may be critical or trivial.

Brophy, 2005
27
We need evidence on Impact!

• Research that can provide rigorous evidence of
  outcomes is needed for managers to make decisions
  that will maximise the impact of library and
  information servicesThe Evidence Based
  Librarianship movement proposes new standards for
  research that can be applied to outcomes research
  and also to the extensive work being done on
  service quality and satisfaction
• Source Cullen, 2001

28
The right measurement tool

• There is a shortage of validated measures
  available. Research could be undertaken to
  develop and validate measures to enable library
  and information professionals to evaluate the
  effects of their training more easily
• (Brettle, 2003)
• Further studies utilizing appropriate
  methodologies and validated research tools would
  enrich the evidence base.
• (Koufogiannakis Wiebe, 2006)

29
15
1 Million
14
500,000
13
250,000
12
125,000
11
64,000
10
32,000
9
16,000
What is the right measurement period for
evaluating Information Skills Training?
8
8,000
7
4,000
6
2,000
5
1,000
4
500
3
300
2
200
5050
1
100
A Straight after Training Session?
B During Training Session?
C 12 months after Training Session?
D 3 months after Training Session?
30
The training half-life

• time taken for half of knowledge acquired
  through training to undergo decay
• ... at the very moment when a learning period is
  finished, the brain has not had enough time to
  integrate the new information it has
  assimilatedIt needs a few minutes to complete
  and link firmly all the interconnections within
  the new material to let it 'sink in'.
• The decline that takes place after the small rise
  is a steep one within 24 hours of a one-hour
  learning period at least 80 per cent of detailed
  information is lost.
• (Buzan, 2005)

31
Evidence for the training half-life
32
An external example

• Participants in sales training forget half of
  what they learn within five weeks - survey of
  more than 6,000 sales professionals
• "Without regular reinforcement, sales training's
  half life is a median of just 5.1 weeks, which is
  even shorter than we suspected. Indeed, for 44
  of participants in the study the half life is
  less than a month." (American Salesman,  January,
  2004)

33
Discussion Point

• What strategies could we put in place to aid
  recall and to alleviate onset of learning
  half-life?

34
Some suggested strategies

• Note-taking
• Mnemonic techniques
• Review
• Repetition and reinforcement
• Interaction
• Problem-based learning
• Integrated into Context

35
Standalone versus integrated
Coomarasamy Khan BMJ 2004329 1017-9.
36
90/20/8 rule

• Rule based on Using Both Sides of Your Brain.
  Estimates average adult can listen with
  understanding for 90 min, but with retention for
  only 20 min.
• The 90/20/8 rule states no module should run
  more than 90 min the pace of instruction should
  change every 20 min, and we should involve
  participants in content application every 8 min.
• Applying this rule to teaching and training can
  move us from a passive learning environment to an
  interactive one.

37
In Summary

• Given the above discussionIs Information
  literacy skills training like a dog's walking on
  his hind legs. It is not done well but you are
  surprised to find it done at all?

38
The Way Forward We need better evidence

• Comparative
• Prospective/Longitudinal Study Design
• Clearly Described Intervention
• Specific and Measured Outcome
• In a relevant/comparable Study Population
• It all comes down to PICOS!

39
Conclusions

• Simply believing that we are doing a good job
  is not sufficient
• Need to start with very clear idea of what we
  want to achieve (Learning Objectives)
• Need to measure what we have achieved
• Need to use reliable instruments
• Above all, need to continually reflect on what we
  are doing and why before Allan Sack-arine does!

40
References - 1

• Booth, A (2006). Counting what counts
  performance measurement and evidence-based
  practice. Performance Measurement and Metrics,
  7(2) 63-74
• Booth, A (2007) In search of the information
  literacy training half-life. Health
  Information and Libraries Journal, 24 (2)
  145-149
• Brettle A. (2003) Information skills training a
  systematic review of the literature. Health Info
  Libr J. 2003 Jun20 Suppl 13-9.
• Brophy, P (2005) The development of a model for
  assessing the level of impact of information and
  library services  Library and information
  research, 29 (93), Winter, 43-49

41
References - 2

• Cullen, R. (2001). Setting standards for library
  and information service outcomes, and service
  quality. 4th International Conference on
  Performance Measurement.
• Koufogiannakis, D, and Wiebe, N (2006). Effective
  Methods for Teaching Information Literacy Skills
  to Undergraduate Students A Systematic Review
  and Meta-Analysis. Evidence Based Library and
  Information Practice 1(3) 3-43.