Evaluating California

1
Evaluating Californias Web-based Training for
Medical Assistants

• Leslie Heyden, MA
• California Department of Public Health
• Immunization Branch
• Information and Education Section
• February 19, 2009

2
Overview

• The purpose of the CDPH IZ Branch evaluation is
  to assess the use and effectiveness of the first
  two immunization skills elearning lessons
• Preparing Vaccines
• Administering Vaccines
• The elearning is custom developed, interactive
  web-based training.
• The primary target audience is medical assistants
  in California VFC provider practices.
• The lessons are available via eziz.org.

2
3
eziz.org

• The new California VFC program website
• A one-stop shop for California VFC providers and
  their staffs
• A repository of a wide variety of immunization
  skills training and VFC program resources,
  including
• VFC forms (e.g., vaccine order and
  recertification forms, vaccine temperature and
  usage logs, patient eligibility screening forms)
• Job aids, guidelines, posters, and
  up-to-the-minute news about vaccine availability

3
4
eziz.orghome page
4
5
Webinar learning objectives

• By the end of this webinar you will be able to
• List the objectives of the evaluation plan
• Explain why CDPH IZ branch selected the EZ-IZ
  elearning lessons to evaluate
• Describe the methodology CDPH developed to
  evaluate the training
• Indicators
• Targets
• Methods, including data sources
• Describe lessons learned

5
6
Why evaluate EZ-IZ elearning?

• Is IZ Branchs first elearning project
• Has potential for high impact re ability to
  reach unlimited number of learners with limited
  resources
• Primary tool needed to conduct evaluation (LMS)
  was already being created
• IZ Branch wanted to compare the reach of the
  elearning to existing classroom training

6
7
Goal of activity to be evaluated

• To educate MAs in California VFC provider
  practices about best practices for vaccine
  preparation and administration

7
8
Evaluation objectives

• By September 5, 2008 the first two elearning
  lessons of the EZ-IZ curriculum (Preparing
  Vaccines and Administering Vaccines) will launch,
  and the learning management system will begin
  tracking usage and learner performance.
• By March 31, 2009, at least one person from 5
  (approximately 180) of Californias active VFC
  provider sites (approximately 3,600) will have
  completed one of the elearning lessons (either
  Preparing Vaccines or Administering Vaccines) and
  have passed its posttest.

8
9
Evaluation objectives (contd)

• By September 5, 2009, at least one person from
  20 (approximately 720) of active California VFC
  provider sites will have completed the Preparing
  Vaccines and Administering Vaccines elearning
  lessons and passed the posttest for each.

9
10
Evaluation objectives (contd)

• By November 30, 2008 all user feedback about the
  EZ-IZ website will have been gathered and
  analyzed, and recommendations made.
• Recommendations concerning website usability will
  be forwarded to and resolved by the web developer
  and/or the web editor.
• Recommendations concerning the EZ-IZ Curriculum
  and the lessons in it will be forwarded to and
  resolved by the instructional designer.
• Recommendations about programming and usability
  issues within the elearning lessons will be
  forwarded to and resolved by the elearning
  vendor.
• Recommendations concerning the learning
  management system (LMS) will be forwarded to and
  resolved by the LMS vendor.

10
11
Stakeholders

• Management
• John Talarico, Interim CDPH IZ Branch Chief
• Tiffany Sutter, Section Chief Information
  Education
• Natalie Nakahara, Project Manager and Creative
  Director
• California VFC Program
• Claudia Aguiluz, VFC Program Coordinator
• VFC field staff and field services reps
• VFC provider staff (particularly MAs who work in
  private provider practices)

11
12
Stakeholders (contd)

• Immunization (federal, state, and local)
• CDC (NCIRD)
• Steve Nickell, Research Coordinator, IZ Branch
• Sandra Jo Hammer, Nurse Consultant, IZ Branch
• CDPH IZ Branch Evaluation team (Leslie Heyden,
  Natalie Nakahara, and Steve Nickell)
• IZ Coordinators in LHDs
• Others
• California Medical Assistants Association (CMAA)
  and California medical assistant training
  programs

12
13
Table 3 Program Component/Activity Description
OutcomesShort term Long term
Inputs
ActivitiesInitial Subsequent
Outputs
Funding   Staff   Time
Review available materials choose
content Design training   Employ vendor to
program elearning and create learning management
system   Make training available via
website   Promote training
Review of content by content experts   Beta test
on target audience   Collect assemble
feedback   Decide on changes   Revise training
  Create a one-stop shop learning and resource
portal for CA VFC providers, immunizers and other
staff with clinical or administrative
immunization-related responsibilities
(Live) online elearning lessons, with learning
management system tracking
VFC providers will adapt to elearning as a
training delivery method   Increased skill and
knowledge (pre- posttest comparison) about
vaccination among medical assistants other
office staff   Steady increase in the number of
MAs who take the elearning lessons
Fewer real-time person hours required to train
MAs   Fewer vaccine administration
errors   Higher coverage rates   Fewer missed
opportunities   Fewer VPDs   Increased provider/
user satisfaction
14
Primary data sourceEZ-IZ LMS

• An LMS (Learning Management System) is software
  that automates the administration of training.
• The LMS registers users, tracks courses in a
  curriculum, and records data from learners
• The LMS provides reports to management.

14
15
Evaluation question 1

• Q Did the Preparing Vaccines and Administering
  Vaccines elearning lessons launch by September 5,
  2008 (target)?
• Indicator and method
• The Preparing Vaccines and Administering Vaccines
  elearning lessons launch request IZ branch staff
  to access lessons and report any problems
  accessing the lessons
• Data source feedback (verbal and email) from IZ
  Branch staff (and confirmation from LMS vendor)

15
16
Evaluation question 2

• Q Did the learning management system (LMS) begin
  tracking usage and learner performance data by
  September 5, 2008 (target)?
• Indicators and methods
• LMS is operational verify with vendor that LMS
  is operational
• LMS is gathering usage data obtain registration
  and login data from LMS report
• LMS is gathering learner performance data obtain
  pretest, lesson start, lesson completion, and
  posttest data from LMS report

16
17
Data sourceregistrationdata
17
18
Learning History Page
18
19
Data sourceLMS
Pre-test
Lesson
Post-test
19
20
Evaluation question 3

• Q How many provider sites have at least one
  person who has completed one lesson and passed
  the posttest?
• Target and indicator
• 5 of provider sites have at least one person who
  has completed one lesson and passed its posttest
  by March 31, 2009
• Method
• Generate LMS report of provider sites that shows
  lessons completed and posttest scores by provider
  site and user
• Data source LMS

20
21
Data sourcepre- and posttests
21
22
Evaluation question 4

• Q How many provider sites have at least one
  person who has completed one lesson and passed
  the posttest?
• Target and indicator
• 20 of provider sites have at least one person
  who has completed one lesson and passed its
  posttest by September 5, 2009 (one year after
  launch)
• Method
• Generate LMS report of provider sites that shows
  lessons completed and posttest scores by provider
  site and user
• Data source LMS

22
23
Evaluation question 5

• Q What feedback has been received from EZ-IZ
  users that could indicate potential obstacles to
  accessing and completing the elearning lessons?
• Targets and indicators
• 100 of emails and 100 of all other feedback
  that could indicate technical problems with the
  website
• 100 of emails and 100 of all other feedback
  that could indicate technical or any other
  problems with the lessons
• Method
• Emails about technical problems and overall
  feedback about the EZIZ website will be
  automatically sent to a specific (ezig.org) MS
  Outlook folder phone calls will be logged and
  categorized by November 30, 2008
• Data source emails submitted by users via links
  on eziz.org

23
24
Data sourcefeedback from users
24
25
Development of evaluation planChallenges

• Very few challenges. IZ Branch Research
  Coordinator collaborated on and provided very
  valuable input into plan.
• Due to limited resources, not able to build as
  robust an LMS as originally planned.
• Proposed revisions to methodology during plan
  development required consulting with LMS vendor
  who then had to modify LMS database or do
  additional programming.

25
26
Lessons learned

• Did not add question about MAs experience could
  not separate primary target audience from
  secondary target audience.
• Learned that more data was needed than provided
  by LMS. Added question to QAR for field reps to
  administer.
• The project team learned a lot during this entire
  pilot effort (including the evaluation plan) and
  is applying lessons learned during the current
  development of the next two lessons.

26
27
Added question to QAR
27
28
Questions?
28
29
Leslie Heyden

• Health Educator Instructional
  DesignerLeslie.Heyden_at_cdph.ca.gov(510)
  620-3754www.eziz.org

29
30
This document

• This document can be found on the CDC website at
• http//www.cdc.gov/vaccines/programs/progeval/down
  loads/cali-eval-02-09.ppt