Activities at the University of Worcester

1
Activities at the University of Worcester
Judith Keene Information and Learning Services,
University of Worcester
2
Evaluation areas

• Information Resources use
• Citations analysis
• Questionnaires
• Impact of training
• Mixture of qualitative and quantitative techniques

3
Evaluating the use of e-journals

• Introduction of e-journals in 2002 in
  Business and IT
• Project aims
• To promote the new resources
• To assess their affect on student learning and
  teaching

4
What did we do?

• Developed a measure of student skills and
  searching behaviour (LT quotient)
• Measured the quotient
• Gave an e-journal induction
• Reinforced use of e-journals
• Re-measured the quotient
• Collected other information

5
The quotient

• A measure of a students ability to carry out
  appropriate research to support their Learning
  and Teaching
• Questionnaire
• Which resources they use
• How resources were located and selected
• Electronic search techniques used

6
How was it calculated?

• Answers were scored against benchmarks we derived
  from SCONUL ILit skills
• E.g. high scores for
• Always using journals
• Always discarding many of the web resources found
• Often refining searches using e.g. Boolean
• Appropriate skills were covered in the induction
  session

7
Results

• Average quotient was significantly higher at the
  end of the module
• Students claimed they were
• Making greater use of using journals
• Making more use of advanced search techniques

8
Further evidence

• Bibliographies contained significantly more
  journals references (citations analysis again!)
• Focus groups
• Students were attempting to use e-journals
  packages effectively
• Still a means to an end (assignments) rather than
  a tool to be used holistically in their learning

9
Was it an effective means of evaluation?

• Resource use
• Format of the training
• Effectiveness of training in terms of outcomes
• Impact on Learning and Teaching

• ?
• ?
• ?
• ?

10
Impact of Information Literacy teaching

• Rationale for evaluation
• To improve future programmes
• Student experience
• Set baseline
• To investigate whether resources are being used
  effectively
• Academic credibility

11
Methods used at UoW

• Questionnaires
• Skills checklists
• Post-its stop, start, continue
• Recent development Pre- and post-testing

12
Why use pre- and post- tests? Theoretical reasons

• Objective measurements
• Measured difference in knowledge and skill should
  be due to the intervention
• Can employ statistical techniques to measure any
  change
• Success of techniques in e-journals evaluation

13
Why use Pre- and post- tests practical reasons

• Easy to replicate
• Easy to use with different groups
• Quick to administer
• Large response rate if used in classroom
  situation
• (Salisbury and Ellis, 2003)

14
Level 1 ILit module

• ILit and IT skills over 12 weeks
• 71 enrolled, nearly all first years
• Following routes from across the modular scheme
• Students tested in week 2 and at end using a
  WebCT based quiz

15
DipHE Nursing information literacy programme

• Embedded across three years of course
• 2 intakes a year
• 3 key sessions in first semester resource
  intensive
• Students were tested at beginning of second and
  end of third sessions
• Self assessment of confidence

16
The ILit module group

• Only 12 students completed quiz twice
• Increase in score statistically significant
• Paired samples test p?0.05
• No correlation between test score and attendance
  or assignment mark
• Useful for baseline evidence and showing up
  possible problem areas
• Less useful for student self-monitoring
• WebCT format had some disadvantages

17
The nursing students

• Change in group is statistically significant
  plt0.05 (Wilcoxon Signed Ranks Test)
• Tests provided evidence that sessions should be
  made mandatory
• Tests highlighted areas of difficulty for whole
  group inform planning

18
Disadvantages

• Doesnt measure long-term behaviour changes
• Can these tests measure deep learning?
• No rich data why are particular wrong answers
  chosen?
• Time consuming to construct
• Writing questions is hard! (Brown and Knight,
  1994)
• Need to pilot
• Guessing

19
References

• Brown, S. and Knight, P. (1994). Assessing
  learners in higher education. London Kogan Page
• Colvin, J. and Keene, J. (2004). Supporting
  undergraduate learning through the collaborative
  promotion of e-journals. Information Research,
  9(2) paper 173 available at http//InformationR.n
  et/ir/9-2/paper173.html
• Keene, J. 2005 Pre-and post-testing of
  Information Literacy skills a preliminary
  assessment of its effectiveness Relay

20

• Kirkpatrick, D.L. (1998). Evaluating training
  programs the four levels. San Francisco
  Berrett-Koelhler,
• Salisbury, F. and Ellis, J. (2003). Online and
  face-to-face evaluating methods for teaching
  information literacy skills to undergraduate arts
  students. Library Review. 52(5), 209-217