Evaluating Intended Continuing Education Outcomes

1
Evaluating Intended Continuing Education Outcomes

• Joshua D. Southwick, MRC, CRC
• David Vandergoot, PhD

2
Outline

• Why Continuing Education?
• Intended Outcomes of Continuing Education
• Actual Results of Continuing Education
• Evaluating Training
• Why Evaluate?
• How to Evaluate?
• Approaches to Evaluation
• Guiding Principles
• Recommendations
• Examples
• Practice Share (if time)

3
Why Continuing Education? Conferences? In-Service
Training?

4
Why Continuing Education? Conferences? In-Service
Training?

• We Need Qualified Rehabilitation Counselors
• The Rehabilitation Act of 1973, as amended,
  requires qualified vocational rehabilitation
  counselors to provide services
• Ethically relevant
• CRCs practice only within the boundaries of
  their competence (CRCC, p. 11)
• CRCs recognize the need for continuing education
  . . . to maintain competence in the skills they
  use (p. 11)

5
What are the Intended Outcomes of Continuing
Education? Conferences? In-service Training?

6
What are the Intended Outcomes of Continuing
Education? Conferences? In-service Training?

• Continuing Professional Development (CPD)
  involves the continuous acquisition of new
  knowledge, skills, and attitudes to enable
  competent practice.1
• Well-trained employees may feel less frustration,
  more job satisfaction and more job commitment2
• After pre-service, graduates should expect to
  learn new skills or hone existing skills3
• Gaining specialty-specific expertise
• Understanding ever-changing challenges arising
  within the field
• Becoming familiar with promising and
  evidence-based practices emerging from new
  empirical research

1. Peck, McCall, McLaren, Rotem, 2000, p. 432
2. Allen van der Velden, 2001
3. Leahy et al., 2009

7
Intended Outcomes (Continued)

• Certification or Licensure Maintenance
• Team Building
• Networking
• Increase Organizational Effectiveness and
  Efficiency
• Increased capacity to serve individuals
• Better services for persons with disabilities
• Greater consumer satisfaction

8
Mixed Results forTraining Effectiveness

• Training has often been less effective than
  expected
• Managerial Training people are learning but not
  applying1
• Medical Professionals2, 3
• Training has been effective
• medium to large effect sizes for training outcome
  criteria related to learning (e.g., knowledge
  d0.63), behavior (e.g., job-related behavior
  changes d0.62), and results (e.g.,
  productivity d0.62)4

1. Powell Yalcin (2010) 2. Davis, OBrien,
Freemantle, Wolf, Mazmanian, Taylor-Vaisey
(1999) 3. Green Seifert (2005) 4. Arthur,
Bennett, Edens, Bell (2003)
9
Training Evaluation Studies in Rehabilitation

• New Zealand study, Flett, Biggs, Alpass (1994)
• Finding professional training decreased
  occupational stress, thereby increasing the
  rehabilitation practitioners ability to work
  effectively
• Christensen, Boisse, Sanchez, Friedmann (2004)
• Finding a one-day training workshop impacted VR
  counselors knowledge and reported practice in
  substance abuse screening (never to rarely)

10
Black Hole?

• Despite the intended outcomes of continuing
  education, the return on this training investment
  remains, to a great extent, unmeasured and
  unknown.
• Most training programs are evaluated only for the
  participants reactions (i.e., satisfaction
  Alliger Janak, 1989 Van Buren Erskine, 2002)

11
Why Evaluate?

12
Why Evaluate?

• It is important to evaluate continuing education
  in order to validate and improve such training
  efforts
• When budgets are tight, it may be necessary to
  justify training expenses

13
Evaluating Training

• Comic showing 2 men at a chalkboard. The
  chalkboard has a complicated formula on it. In
  step 2 of the formula are the words "then a
  miracle occurs." One man says "I think you
  should be more explicit here in step two."

Better Outcomes for Persons with Disabilities
Continuing Education Training
14
Approaches to Training Evaluation

• Kirkpatricks Four Levels
• Logic Model

15
Kirkpatricks Four Levels
The Four Levels The Four Levels
Level 1 Reaction To what degree participants react favorably to the learning event.
Level 2 Learning To what degree participants acquire the intended knowledge, skills, and attitudes based on their participation in the learning event.
Level 3 Behavior To what degree participants apply what they learned during training when they are back on the job.
Level 4 Results To what degree targeted outcomes occur, as a result of the learning event(s) and subsequent reinforcement.

• The Four Levels
• Level 1 Reaction To what degree participants
  react favorably to the learning event.
• Level 2 Learning To what degree participants
  acquire the intended knowledge, skills, and
  attitudes based on their participation in the
  learning event.
• Level 3 Behavior To what degree participants
  apply what they learned during training when they
  are back on the job.
• Level 4 Results - To what degree targeted
  outcomes occur, as a result of the learning
  event(s) and subsequent reinforcement.

From Kirkpatrick Kirkpatrick (2010)
16
Percentage of Training Evaluated at each Level

• Reaction 78
• Learning 32
• Behavior 19
• Results 7
• (Reported across multiple disciplines)

Reported across multiple disciplines
Morin, L., Renaud, S. (2004). Participation in
corporate university training Its effect on
individual job performance. Canadian Journal of
Administrative Sciences, 21(4), 295-306.
17
Evaluation through Logic Models

• A logic model shows the rationale or program
  theory for how program planners believe that the
  resources and activities invested in a program
  will produce the expected outcomes.
• Used to
• Visually display the components of a program
• Identify measures that will be useful in
  evaluating the program outcomes

18
Components of a Logic Model

• Inputs
• Resources (human, financial, organizational,
  community)
• Activities
• Implementation how resources are used (projects,
  events, actions)
• Outputs
• Participation
• Direct products (deliverables)
• Outcomes
• Impact (expected changes or benefits)
• Short-term learning
• Medium-term action or behavior
• Long-term conditions
• Adapted from University of Wisconsin-Extension-Coo
  perative Extension, 2003 W. K. Kellogg
  Foundation, 2004

Adapted from University of Wisconsin-Extension-Co
operative Extension, 2003 W. K. Kellogg
Foundation, 2004
19
Logic Model Example

• Inputs
• Activities
• Outputs
• Outcomes

Inputs
Activities
Outputs
Outcomes
20
A Possible Logic Model for Continuing Education

• Inputs
• program planners
• instructor preparation
• training materials
• money
• facilities
• technology
• Activities
• Using resources to implement a
• continuing education program
• conference
• Workshop
• webinar
• Outputs
• verification of attendance at the training
• CEU credits earned
• satisfaction scores
• Outcomes
• Short-term increases in participants knowledge,
  skills, and confidence

21
How to Evaluate Intended Outcomes

• Which criteria should be measured in order to
  most accurately assess the outcomes of continuing
  education?
• What types of measures can act as indicators that
  professionals are developing?

22
Guiding Principles

• Knowledge Translation (KT)
• Organization Development (OD)

23
Knowledge Translation (KT)

• A move beyond the simple dissemination of
  knowledge into actual use of knowledge1
• Barriers to KT / research utilization2
• Environmental organizational factors (culture,
  leadership)
• Individual factors (age, years of service)
• Difficulty accessing research (database access,
  time)
• Difficulty determining the relevance of research

1. Straus, S. E., Tetroe, J., Graham, I. (2009).
   Defining knowledge translation. Canadian Medical
   Association Journal, 181(3-4), 165-168.
2. Johnson, K., Brown, P., Harniss, M., Schomer,
   K. (2010). Knowledge translation in
   rehabilitation counseling. Rehabilitation
   Education, 24(3-4), 239-250.

24
KT what happens after the knowledge is in our
heads?
25
Organization Development (OD)

• Organizational Development A method for
  designing, implementing, and reinforcing
  intentional organizational changes1
• Key characteristic of OD
• The action taken is deliberately and consciously
  designed to bring about change over a specified
  time period, and there must be some way to
  demonstrate and/or measure the degree to which
  the change occurred2
• 1. Cummings Worley, 2009
• 2. Worley Feyerherm, 2003

26
Guiding Principles for the Evaluation of
Continuing Education
KT2

• Notes
• 1. Kirkpatricks levels
• 2. Knowledge Translation principles
• 3. Organization Development principles

27
How it is done now

• Requirements for pre-approval of CRC continuing
  education credits
• gt60 minutes
• Focus is to increase knowledge of or skills in
  rehabilitation counseling
• Clearly defined learning objectives or expected
  outcomes
• Participants complete an evaluation of the
  programs value (not an evaluation of learning)
• Accessible, barrier free location
• For CE through written means, multiple choice
  questions are required.
• CRCC (2011)

28
Example of How Evaluation could be done

• VR agency does an in-service training
• They identify learners needs/wants Involve
  learners in training planning process (Adult
  learning theory OD)
• Set objectives for learning (Use these in the
  evaluation questions indicators)
• Hold training
• Evaluate
• Reaction (satisfaction survey)
• Learning (pre-post quiz)
• Behavior (2-3 month follow up survey on
  Objectives) (Knowledge Translation)
• Results (3-12 month follow up on agency
  indicators specifically related to Objectives)
  (Organization Development)

29
(No Transcript)
30
Evaluation Recommendations

• Timing
• 2-12 months post training? Or the amount of time
  estimated for participants to implement new
  skills.
• Measurements not too distal from training
  objectives (if too distal, you wont see the
  impact)
• May also want to assess organizational culture
  (did it support or hinder implementation)

31
Additional Recommendations for Distance Blended
Training

• Include planned assessments in the course outline
• Build assessments so that they are fully
  integrated into the course
• Ensure participants that their satisfaction
  scores will remain anonymous (i.e., they will not
  be tracked by IP address)
• Provide immediate feedback when feasible

32
Daves Example
33
Context

• Online training provided to 42 counselors
• Trained to implement a case management model in
  10 sites
• Trainees were administered a knowledge check as a
  post-training assessment
• Ongoing training provided using Case Reviews
• Performance evaluated using benchmarks of key
  model indicators aggregated by site

34
Evaluation Strategy

• Conduct training and evaluate the extent of
  content learned using a knowledge check
• Assess interim performance by conducting case
  reviews using a protocol reflective of model
  processes and providing one-on-one instruction as
  needed
• Evaluate relationship of performance on case
  review protocol with model performance indicators
  (interim assessment reported here)
• Eventually relate performance on model indicators
  with employment outcomes (will not be available
  for several years)

35
Ongoing Evaluation Model

• Provide online training
• Assess Knowledge
• Conduct quarterly case reviews
• Provide TA as needed
• Monitor site performance monthly
• Analyze individual and site data
• (back to) Conduct quarterly case reviews

Conduct quarterly case reviews
Provide online training
Assess Knowledge
Provide TA as needed
Monitor site performance monthly
Analyze individual and site data
36
First Question

• Did those who completed all the online courses do
  better than those who did not complete the
  courses on their first Case Review?

37
Answer to First Question

• Score on First Case Review
• 8 did not complete all the courses
• 34 did complete them
• Means
• Completers averaged 76
• Non Completers averaged 86
• This is not a significant difference
• Implication more assistance needed

38
Second Question

• Did those who completed all the online courses do
  better than those who did not complete the
  courses as averaged over all their Case Reviews?

39
Answer to Second Question

• The average score of all case reviews
• Means
• Completers averaged 80
• Non Completers averaged 82
• This is not a significant difference
• Providing training in and of itself may not be
  sufficient to achieve desired performance

40
Third Question and Answer

• What is the relationship between the average
  course grade and Case Review scores?
• First Case Review score Correlation .09
• Averaged Case Review scores Correlation .12
• These are both significant at the .01 level
• Implication Although these results are in the
  desired direction, they are weak and reinforce
  the need for ongoing technical assistance

41
Fourth Question and Answer

• What is the degree of improvement in Case Review
  Ratings over time?
• Mean Case Review Ratings Quarter 1 77.9
• Quarter 2 70.7
• Quarter 3 81.9
• Quarter 4 85.3
• Quarter 5 89.0
• Implication providing ongoing technical
  assistance indicates improved conformance to
  model expectations

Quarter Mean Case Review Rating
1 77.9
2 70.7
3 81.9
4 85.3
5 89.0
42
Fifth Question and Answer

• This analysis was based on aggregated data by
  site
• How do Case Review scores, as measured by
  aggregating most recent two reviews, relate to
  the most recent performance indicators?
• Correlation with
• Key indicators most reflective of course content
  .59
• Total benchmark score .65
• Ranking of site performance .71

43
Overall Implications

• Training with follow up technical assistance
  leads to desired performance. Simply providing
  training may not lead to success.
• Looking at data using the individual as unit of
  analysis and also using aggregated site data as
  unit of analysis leads to enhanced understanding
  of the impact of training and technical assistance

44
Joshs Example
45
Context

• Professional Conference
• 2 follow-up surveys were sent
• 1st survey immediately following the conference
• 3 month follow-up survey after the conference
• Sought to identify factors that facilitated or
  hindered Knowledge Translation
• Sought to identify the types of collaboration
  that resulted from networking at the conference

46
First Survey Results

• N 98
• Reported KT 76
• Facilitators of KT
• Personal Interest 72 (55)
• Opportunity in current situation 61 (46)
• Belief that applying the knowledge/skills will
  make a positive difference 45 (34)
• High Self-efficacy 39 (30)
• Supportive policies and/or superiors 34 (26)
• Peer Interest 33 (25)
• New collaborations as a result of the conference
• About 500 new collaborative projects (not unique)
  reported among 76 responders

47
First Survey Results (cont.)

• Reported no KT 22
• Barriers to KT
• Lack of personal interest 0
• Lack of peer interest 1
• Lack of opportunity 8
• Lack of supportive policies and/or superiors 0
• Low Self-efficacy 0
• Belief that applying the knowledge/skills will
  not make a difference 1
• Lack of Time 6

48
Reported Facilitators of KT
49
3 month Follow-up Survey Results
50
Advantages of Logic Model Evaluation

• Validation of the effectiveness of continuing
  education
• Timely feedback to continuing education providers
• Evaluate at levels beyond just Reaction
  Learning, Behavior, Results
• The very act of evaluating (and planning to
  evaluate) can positively impact
• Organization of the continuing education activity
• Engagement of the participants during the
  continuing education activity
• Transfer of knowledge and skills to workplace
  behaviors
• Results for the entire organization

51
Implications for research and practice

• Research
• A potential framework for gathering data on the
  effectiveness of continuing education programs
  (learning, behavior, results, impact).
• Practice
• A tool in planning, evaluating, validating, and
  improving ongoing counselor training efforts.
• Better ensure that continuing education increases
  the competence of practitioners and thereby
  improves services and outcomes for service
  recipients.

52
Questions? Comments?