A Practical Corporate E-Learning Effectiveness Model

1
A Practical Corporate E-Learning Effectiveness
Model

• Lim Kin Chew
• SIM University
• 23 Feb 2012
• 8th International Conference on E-Learning for a
  Knowledge-Based Society
• Bangkok, Thailand

2
Quotation - 1

• Although recent attention has increased
• e-learning evaluation, the current research
• base for evaluating e-learning
• is inadequate Due to the initial cost
• of implementing e-learning programs,
• it is important to conduct evaluation
• studies.
• (American Society for Training and
• Development, 2001).

3
Quotation - 2

• Despite the ample support and enthusiasm for
  virtual models of learning from both the
  corporate and educational arenas, the
  effectiveness of e-Learning is by no means a sure
  thing. Successful e-Learning programmes require
  specific conditions and constant monitoring and a
  lot more research has yet to be done before an
  acceptable universal model can be developed.
  (Kar-tin Lee, Hong Kong Institute of Education,
  Malaysian Online Journal of Instructional
  Technology, August 2005, ISSN 1823-1144, Vol. 2,
  No.2, pp 61-71)

4
Purpose of this Paper

• Share experiences on how an actual evaluation of
  a large-scale e-learning project was carried out
• Add to the research base on e-learning
  effectiveness

5
Agenda

• Why do we need a practical corporate e-learning
  effectiveness model?
• Different approaches to evaluation models
• Case study on an e-learning implementation
• QPICC model
• Lessons learnt
• Conclusion

6
Different Approaches to E-learning Evaluation
(Source Graham Attwell (ed.). (2006).
Evaluating E-learning - A Guide to the Evaluation
of E-learning. Evaluate Europe Handbook Series
Volume 2. ISSN 1610-0875.)
7
Why do we need a practical e-learning
effectiveness model?

• E-learning effectiveness is difficult to carry
  out.
• Even if it is carried out, it is usually done in
  a rather ad hoc manner.
• E-learning evaluation takes up time, resources,
  manpower, cost and it may end up inconclusive
• E-learning evaluation should be done by
  experienced e-learning practitioners
• E-learning evaluation depends very much on what
  the user client wants to evaluate or measure.
  The consultant guides him along.

8
Purpose of Project

• To evaluate the effectiveness of the e-learning
  efforts in facilitating the training functions of
  ABC
• Based on the evaluation outcomes, the client
  wants appropriate recommendations on e-learning
  capabilities so as to further enhance
  instructional delivery in the ABC training.

9
Case Study on E-learning Evaluation

• ABC (name changed because of confidentiality) is
  a government body in charge of training personnel
  for home security.
• They have been using e-learning since 2006.
• Most of the training has been supplementary, i.e.
  used to support the classroom face-to-face
  sessions.
• Investments have been made to set up three LMSs
• Management now wanted to find out how effective
  has the e-learning been in training the personnel
• Client has several departments. Only three
  departments were involved in the evaluation
  project.

10
Objectives Methodologies - 2

• To evaluate the effectiveness of ABC e-learning
  training platform in the following areas
•  
• qualify for the pedagogical underpinnings
  considered for e-learning modules
• measure the quality of e-learning course wares
• measure the performance output by benchmarking
  the e-learning platform against a set of
  pre-defined criteria
• determine the adequacy of existing e-learning
  infrastructure and to validate congruency in
  these capabilities with their instructional
  intent
• identify trainer and training designers
  competency gaps in e-learning facilitation and
  design based on the evaluation outcome.

11
Proposed E-learning Quality Effectiveness Model
for ABC
12
How the Project was organized
Team Leader
Team UniSIM
Team ABC

• Participative approach
• Expertise transfer
• Joint review of results, observations and
  conclusions
• PLAN DO CHECK ACT (PDCA)

13
S/No. Module Specific Evaluation Tools Frequency of Use Involvement of ABCs e-Learning Champions
1 Quality Criteria of E-Learning Courses Quality Criteria Checklist Once for every E-Learning course Some involvement
2 Performance Measurements of E-Learning Courses Satisfaction survey (for participating e-learners) Pre-Test Results (if available) Post-Test Results (if available) Questionnaire survey for supervisors / facilitators To do all for one time for one e-learner in the sample Considerable involvement as we will be checking their performance results
3 E-Learning Infrastructure Inventory report of existing E-Learning infrastructure Inventory report of proposed E-Learning infrastructure To do a one-time compilation report Some Involvement especially from management support staff
4 Staff Competencies Questionnaire survey for each member of the sample One-time survey Considerable involvement
5 Organizational Culture Questionnaire survey for each member of the sample Feedback from focus group One-time survey Considerable involvement
Evaluation tools used in Project
14
S/No. Module Specific Evaluation Tools Maximum Sample Size
1 Quality Criteria of E-Learning Courses ECC Quality Criteria Checklist 20
2 Performance Measurements of E-Learning Courses Satisfaction survey Pre-Test Results (if available) Post-Test Results (if available) Questionnaire survey for supervisors / facilitators 40
3 E-Learning Infrastructure Inventory report of existing E-Learning infrastructure Inventory report of proposed E-Learning infrastructure 20
4 Staff Competencies Questionnaire survey for each member of the sample 20
5 Organizational Culture Questionnaire survey for each member of the sample Feedback from focus group 20
Sample size of Trainees
15
Project Schedule on E-learning Evaluation for ABC
16
E-learning Development Cycle
14 competencies defined
9 competencies defined
2 competencies defined
4 competencies defined
3 competencies defined
17
Effectiveness of E-learning
How effective has the e-learning been in the ABC
training?
18
Overall Key Findings of the Evaluation - 1

• Management Level
• Lack of a short-term or long-term plan for
  e-learning (vision, goals, targets, timeline,
  etc)
• Unsure of the position of e-learning in the
  overall training curriculum
• Pedagogical area
• Lack of a consistent explanation or definition of
  e-learning as used in the context of the ABC
  environment
• Have a narrow interpretation that e-learning
  means delivering courseware or content-based
  learning

19
Overall Key Findings of the Evaluation - 2

• E-learning is used in a reinforcement mode
• All the courseware titles (except the
  scenario-based video courses) that have been
  evaluated are based on passive learning. Other
  types like e-Test system are used in Dept A and
  other Departments have been mentioned but we were
  not shown.
• Need to deploy appropriate instructional
  strategies (e.g. discussion, lecture,
  self-directed learning, mentorship, small group
  work, project, collaborative learning, case
  study, forum) for effective training using
  e-learning. Passive learning is based very much
  on memory work. Need to be aware of paradigms
  like constructivism and social constructivism to
  engage the learners.
• Trainers need to know how to track trainees
  e-learning progress
• Need for a proper assessment of the performance
  of the trainees in e-learning and then to measure
  the transfer to his job performance

20
Overall Key Findings of the Evaluation - 3

• Organizational Culture towards E-learning
• Need sharing of best practices amongst staff
  members
• Need to build up e-learning training resources
  for e-learning training staff
• Need to have an E-learning Outsourcing
  Development Methodology
• Awareness and competency levels of e-learning
  stakeholders
• Need to raise awareness on the following areas
• What is e-learning?
• What e-learning can do and cannot do?
• Why ABC is pushing for e-learning?
• Need to move to a more competency-based model to
  train staff and assign tasks and responsibilities
  

21
Overall Findings of the Evaluation - 4

• Overall
• Need to have an ABC centre which coordinates the
  following
• Monitor and raise the e-learning competencies of
  the e-learning personnel.
• Maintain the quality of the courseware or other
  e-learning activities
• ABC can also position itself as the central body
  for the following activities
• e-learning awareness and promotional activities
• Identify foundational e-learning competencies for
  e-learning personnel in the ABC Training schools
  
• work with external organizations to get the best
  deals in any tender i.e. courses, infrastructure
  requirements etc.

22
Scoring Scheme
Totally ineffective 1
Ineffective 2
Satisfactory 3
Effective 4
Very effective 5
23
Overall Effectiveness Model
S/No. Criterion Score
1 Quality of Courseware 2 / 5
2 Performance of Ex-trainees 1.5 / 5
3 E-learning Infrastructure 2 / 5
4 Key Competencies of staff 1.5 / 5
5 Organizational Culture towards E-learning 1.5 / 5
Overall score 8.5 / 25 or 34 Overall score 8.5 / 25 or 34 Overall score 8.5 / 25 or 34
24
of E-learning in Training Curriculum
S/No. Dept. Total no. of training hours (approved timetable) Total no. of e-learning hours (approved timetable) of E-learning in Training Curriculum
1. Dept 1 1130 16 1.4
2. Dept 2 1039 12.5 1.2
3. Dept 3 724 23 3.2
(Note the low percentages of e-learning component
in the training curriculum.)
25
Lessons Learnt - 1

• E-learning evaluation is not straightforward. No
  set formula or set template or set model to use.
• Based very much on what the client wants to
  measure.
• The percentages of e-learning in the overall
  training curriculum in the three departments are
  1.4, 1.2 and 3.2 respectively. Is it worth your
  while to do the evaluation?
• Most of the feedback are feedback gathered from
  the trainees, supervisors, focus group
  discussions. These are all very qualitative.
• Job performance is the hardest area to determine.

26
Lessons Learnt - 2

• Plan to measure and measure what you have
  planned. E.g. Quality of courseware, job
  performance.
• Difficult to measure what you have not planned to
  measure.
• Senior Management may have other views about
  e-learning. Good to have their buy-in.

27
What is E-learning?

• E-learning is the use of Internet and digital
  technologies to create experiences that educate
  our fellow human beings
• E-technologies do not change how human beings
  learn but remove constraints
• E-learning is not just Web-casting lecture, and
  not training materials dumped online
• Professor Tapio Varis
• Unesco Chair in global e-Learning
• University of Tampere

28
E-Learning
Self-directed learning Collaborative
learning Peer learning Passive learning Instructor
-led
Pedagogy
LMS Internet Intranet Mobile CD-ROMs
Highly structured Open Contents Test-based Knowled
ge-based Learning objects Rich multimedia
Technology
Contents
29
End of Presentation
Lim Kin Chew kclim_at_unisim.edu.sg