Does your information literacy work make a difference How can you tell

1
Does your information literacy work make a
difference? How can you tell?

• Sharon Markless
• Kings College London and IMA

2
What do we mean by IL work making a difference?

• Getting people to
• Use appropriate search strategies rigorously
  evaluate e-resources manage research information
  effectively
• Value publications/research datasets/ICT
  resources/ IL interventions
• Be more confident in using the full range of
  information/databases
• Access a different range of resources assess
  students IL collaborate/alter pedagogy
• Know more about where to find appropriate
  resources, which portals to use, where to
  publish

3
What longer term/higher order impact might IL
work achieve?

• Support widening participation in HE
  (recruitment, graduation rates)
• Influence patient outcomes e.g. choice of drugs,
  appropriate investigations
• Save clinical time
• Increase in publications
• More/larger research grants
• Increased professional or academic success????

4
But beware of taking a step too far

• Whose impact? Identifying the significant
  contribution of the library/ information service?
  (attribution studies factorial design,
  controlling variables - scale required e.g.
  Colorado etc)
• Important not to make library a hostage to
  fortune What can you really affect?
• Use the research when considering your possible
  value and impact

5
Research evidence shows health libraries can
influence patient outcomes by supporting

• rapid diagnosis
• appropriate investigation
• choice of drugs
• reduced length of stay in hospitals
• avoiding hospital-acquired infection
• avoiding additional outpatient visits
• Weightman and Williamson (2005)
• reduced waiting list due to new treatment
• Single site case study (2007)

6
Research evidence shows academic library impact

• Local level (case studies)
• can enhance student retention rates
• help academic research publications output
• may meet specific student and researcher IL
  training needs
• changes in assessment practices
• high ROI in electronic content, in grant money
• National level
• enhance knowledge and use of e-resources
• enhance information literacy

7
Step one in evaluating impact focus

• Make decisions about your focus
• areas of impact for your IL work your impact
  objectives (what changes are you looking for?)
  All linked to the purposes and aspirations of the
  organisation
• Precision matters impact objectives drive the
  whole process of evaluation
• Be realistic with available resources 3-5 years

8
Examples of impact objectives from real libraries

• To secure the library as a main source of
  information in conducting research in the field
• To enable students/doctors/nurses to operate
  effectively in an e-environment
• To develop clinicians ability to find and use
  appropriate information
• Targeted staff making more effective use of
  evidence-based information in clinical practice
• Competent independent use of electronic resources
  
• Healthcare staff find the evidence to develop
  local care pathways.

9
Challenges librarians encountered when
formulating impact objectives

• using support and provide in objectives was
  automatic but they will draw you down the
  process road!
• Often unclear where the impact/process line is
  drawn (integrate IL into the curriculum lecturer
  attitudes)
• Its hard work and means reviewing plans,
  policies, strategy but no-one else can do it for
  you!

10
Step two in evaluating impact impact indicators

• Indicator a statement around which you can
  collect evidence on a regular basis to show a
  trend
• Strong surrogates for impact
• Choose what matters
• Limit numbers
• Direct link to relevant impact objective

11
Examples of impact indicators from real libraries

• Levels of remote use of resources
• Proportion of targeted staff using the evidence
  base
• Do students cite eresources in bibliographies?
• Proportion of users needing intermediary
  assistance for directional and basic search
  enquiries
• Nurses competence in appraising evidence
• Percentage of searches focused in the wrong area
• Do clinicians follow up and ask for articles?
• Percentage of clinical time spent on searching

12
Step three in evaluating impact collecting
evidence

• What counts as evidence?
• Statistics (relationships between)
• Case studies, stories, critical incidents
• Analysis of student bibliographies and
  assignments/projects
• Results of self-assessment or diagnostic tests
• Document analysis
• Systematic observation data
• Systematic, rigorous, mainly qualitative

13
How can we collect evidence?

• Observation
• Asking questions
• Inferring
• Time consuming
• New skills for librarians?
• Appropriateness different tools for different
  groups and contexts

14
Rich mixture of methods adopted by teams in
LIRG/SCONUL initiative and health libraries work

• Diagnostic tests (especially for distance
  learners)
• Questionnaires (ideal type) embedded questions
• Focus groups (structured)
• Interviews during searches
• Student progress files/reflective diaries
• Analysis of assignments and bibliographies
• Teaching audit
• Critical incident interviews

15
Some issues in gathering evidence reported by
librarians

• What standard of evidence do you need?
• Necessity of collecting baseline data
• Short-, medium- or long-term? Impact may take
  time, not fit annual planning cycles
• Low response rates
• Capturing evidence in busy libraries

16
Some issues about evaluating impact reported by
librarians

• Collaboration/cooperation is critical,
  particularly in data collection
• Important to focus on one aspect of provision in
  depth
• Sustaining the work what might be needed for
  institutionalisation?
• Challenging and stressful nature of engaging with
  impact

17
Some issues about evaluating impact reported by
librarians

• Importance of a coherent framework and structure
  to support the process
• Use of an impact model systematic approach
• 3 x workshops take detailed work back to own
  library troubleshooting
• Materials, especially examples of indicators etc,
  approaches to data collection, research tools
• Teams within each participating library

18
Some final words from librarians who have been
doing it

• Keep a balance you also need to maintain the
  infrastructure
• Choose what matters
• Keep it manageable
• Influencing academics and getting change at
  Academic Boards was harder to do than the
  evaluation.

19

• Weightman, A. L. and Williamson, J. (2005) The
  value and impact of information provided through
  library services for patient care a systematic
  review in Health Information and Libraries
  Journal 22 (1) march 4-25
• For research tools and approaches used in
  evaluating impact http//vamp.diglib.shrivenham.c
  ranfield.ac.uk/impact/impact-initiative
• Return On Investment case study Elsevier
  website Library Connect