Evaluating the Impact of a VLE on Learning and Teaching

1
Evaluating the Impact of a VLE on Learning and
Teaching?

• Dr Barbara Newland
• Learning Design Studio
• Academic Services
• Bournemouth University
• http//www.bournemouth.ac.uk/lds/
• bnewland_at_bournemouth.ac.uk

2
Overview

• Aims of evaluation
• Durham context
• Methodology
• Results
• Conclusions

3
Aims of evaluation

• International widespread adoption of VLEs
• Longitudinal study of impact of a VLE on learning
  and teaching
• Qualitative and quantitative methods

4
University of Durham

• 670 academic and 280 academic-related staff
• 9400 undergraduates and 2400 postgraduates
• Founded in 1832
• Split campus
• Research led
• MIS outsourced to Unisys
• Learning Technologies Team (LTT)

5
Images of Durham
6
Learning Technologies Team

• aims to encourage, support and develop elearning
  throughout the University by
• providing advice, information and training
• supporting the University's learning environment
  - duo (Durham University Online)
• developing interactive materials
• 4 members of staff Juliette Pavey, Kate
  Boardman, Malcolm Murray, Victoria Boyd

7
Duo

• Durham University Online
• http//duo.dur.ac.uk

8
Uptake of duo
9
Implementation

• Varied patterns of departmental implementation
• All courses
• All first year courses
• Individual academics
• Administration courses
• Duo is now embedded into learning and teaching

10
Methodology for duo surveys

• Quantitative and qualitative
• Online surveys
• Analyse factors such as gender, year of study and
  faculty
• 3 faculties Arts and Humanities, Science,
  Social Science and Health
• Relate to CIT skills in learning and teaching
• Data collected from students and staff in all
  faculties and both campuses

11
Response rate

• All students and staff are in duo so hard to know
  the response rate in relation to active users
  which has increased over the years
• Staff April 01 (n 91)
• May 02 (n 153)
• May 03 (n 283, c. 40)
• Students May 01 (n 793)
• May 02 (n 2435)
• May 03 (n 3668, c. 37)

12
Survey questions

• Similar questions for staff and students in all 3
  years to enable comparisons
• between years
• between staff and students
• Software
• ease of use
• levels of confidence
• features used

13
Survey questions

• Frequency of use and access during term time and
  vacations
• Learning and teaching
• contribution
• effect

14
Student responses support, frequency and access

• Help documents
• 19, 15, 10
• Small and decreasing use
• Accessed duo at least once a week
• 70, 71, 71
• Consistent pattern but increasing numbers over
  time
• Accessed duo out of term time
• 60, 61, 64
• Consistent pattern but increasing numbers over
  time

15
Students Overall, how do you rate the
contribution duo has made to your learning?
16
Staff Overall, how do you rate the effectiveness
of duo as a learning resource?
17
Staff Has duo affected your approach to teaching?
18
Students If duo is not currently being used for
all your modules, would you recommend that it
should be?
19
Staff use of duo features - content

• 62, 49 staff have used more features
• 79, 76 staff have developed content since the
  previous year
• Course information 74, 84, 92
• Course documents 77, 89, 95
• External links 42, 56, 55
• Bibliography 35, 53, 43 (plus 17, 21, 21
  link to OPAC)
• Staff information 49, 63, 72

20
Staff use of duo features - communication

• Group work 25, 23, 26
• Email 72, 77, 80
• Announcements 75, 84, 86
• Discussion board 16, 32, 31
• Chat 10, 4, 6

21
Staff use of duo features - assessment

• Assignments 53, 63, 67
• Quizzes 11, 20, 26
• Surveys - 15, 13, 21
• Digital drop box 10, 6, 15

22
Reflect current ways of teaching

• Context relevant, flexible (start from where
  they are at)
• Collis also found at the University of Twente
  that instructors are most likely to begin by
  choosing aspects of a system that reflect their
  current ways of teaching, and then gradually move
  to new instructional approaches and new
  features. (Collis and Messing, 2001)

23
Lecture notes - staff

• Is it good practice to put lecture notes online?
• Depends on what the lecturer does with them
• 2/3 put lecture summaries, handouts, full notes
  and/or Powerpoint slides after the lecture

24
Lecture notes - students

• Depends how students use them
• 73 listen more
• 81 clarify information
• 72 improve lectures
• 93 study for exams
• 2/3 (64, 64) students note-taking had stayed
  the same but for ¼ (27, 26) it had decreased
• Only 1, 1 stated no longer attended lectures

25
Gender

• No statistical significance between gender and
• Staff with effectiveness as learning resource
• Students with contribution to learning
• Both with levels of confidence
• Higher percentage of females think duo helps them
  to manage their learning

26
Staff by faculty skills

• More Science faculty found it very easy/easy to
  use compared to Arts
• Some Arts staff took longer to gain confidence
  (6-10 visits)
• Fewer Science staff chose to have training
• Reflects the levels of staff confidence in CIT
  skills found in Audit June 2000

27
Staff by faculty teaching

• 2002 - no significant difference
• 2003
• Basic approach is changing
• Science and social science 12 compared to 3
  arts
• Basic approach has not changed but duo helps me
  do certain things better
• Science 79, social science 76 compared to 98
  arts

28
Students by faculty

• Contribution that duo has made to their learning
  overall
• 2002 - no significant difference
• 2003 higher percentage of science (69) and
  social science (52) rate good or excellent
  compared to 42 arts students
• Higher percentage of social science students
  think duo helps them to manage their learning
  greatly or quite a lot compared to science
  students

29
Students by year group

• Contribution that duo has made to their learning
  overall
• 2001 and 2002 - 2nd and 3rd year students rates
  more higher than 1st years (1st years have always
  had duo)
• 2003 1st year students rate slightly higher
  than 2nd and 3rd years

30
CIT skills survey of new undergraduates 2002

• n 2656
• 82 of students perceive themselves to be
  confident/very confident computer users
• 60 have brought a computer with them to
  University
• 69 have been using a computer for 5 years
  (including 21 for over 10 years) and only 2
  have been using a computer for lt2 years

31
CIT skills survey of new undergraduates - 2002

• 79 perceive themselves to have good/advanced
  email skills
• 65 perceive themselves to have good/advanced web
  browser skills
• 70 stated a preference for a combination of web
  and paper based for learning support materials

32
CIT skills survey of new undergraduates - 2002

• 95 own a mobile phone
• 5 have a PDA
• 38 take part in online discussions
• 59 take part in online chat
• 49 have a qualification in IT
• 54 regularly switch between several applications
  when using a computer

33
CIT skills survey of academic staff - 2000

• n 576 (53)
• 90 perceive themselves to have good/advanced
  email skills
• 67 perceive themselves to have good/advanced web
  browser skills
• only 17 of staff had taken part in online
  discussions
• only 12 of staff had taken part in online chat

34
Catalyst

• Introduction encouraged continual reflection on
  teaching
• Postgraduate Certificate in HE compulsory for new
  academics from Jan 2001
• Academics working together within courses

35
Factors influencing enhancement

• How used
• How integrated into curriculum
• Quality of the parts eg content, assessment,
  communication (earlier research)
• Enhance learning if the parts enhance learning
• Flexibility allows lecturers to choose parts

36
Is the whole greater than the sum of its parts?

• But is a VLE more than the pieces put together?

37
Evaluating the impact of a VLE on learning and
teaching

• Incredible uptake of duo which has become
  embedded into the University
• Staff and students believe duo has enhanced
  teaching and learning
• Academics are gradually using more features so
  this may result in more interactive and
  innovative teaching
• Students expectations are increasing as the basic
  use of a VLE is assumed
• Further analysis is being undertaken

38
References

• Collis, B. and Messing, J. (2001) Usage,
  attitudes and workload implications for a
  web-based learning environment, Association of
  Learning Technology Journal 9 (1)
• Collis, B. and Moonen, J. (2001) Flexible
  Learning in a Digital World, London, Kogan Page  
• Laurillard, D. (1993). Rethinking University
  Teaching a Framework for the Effective Use of
  Educational Technology. London Routledge.
• Richardson, J., Turner, A. (2000). A
  large-scale local evaluation of students
  learning experiences using virtual learning
  environments. Educational Technology and Society,
  3 (4).
• Salmon, G. (2000). E-moderating the Key to
  Teaching and Learning Online. London Kogan Page.