Measuring and Evaluating Training

1
Measuring and Evaluating Training
2
Session Topics

• Purpose and Benefits of Evaluating Training
• Measurement Process
• Issues in Assessment Development
• Formative vs. Summative Evaluation

3
Session Topics

• Kirkpatricks Four Levels
• ROI Methodology and Calculation
• Three Measures of Central Tendency
• Uses of Analysis

4
Purpose of Evaluating Training

• What is the Purpose of Evaluating Training?
• Determining business impact
• Improving the design of the learning experience
• Determining the contents adequacy

5
Benefits of Evaluating Training

• What are some benefits of evaluating training?

• Secure client support and build relationships
  w/client
• Allows practitioner to see whether the results of
  learning are consistent w/business analysis and
  needs assessment
• Helps focus the training

6
Benefits of Evaluating Training

• What are some benefits of evaluating training?

• Validates performance gaps and learner needs
• Determines where training is the solution to a gap

7
QUESTIONS?
8
Measurement Process

• List the basic steps of evaluation.
• Identify evaluation goals
• Develop an evaluation design and strategy
• Select and construct measurement tools
• Analyze data
• Report data

9
QUESTIONS?
10
Issues in Assessment Development

• Validity vs. Reliability
• Validity measuring what you intended to
  measure
• Reliability ability of the same
  measurement to produce consistent results
  over time

11
Issues in Assessment Development

• How do Validity and Reliability Relate to Each
  Other?

• Validity concerns what is being measured
• Reliability concerns how consistently the
  measure is measuring what it is intended to
  measure

12
Issues in Assessment Development

• What are the five types of validity?
• Content validity
• Construct validity
• Concurrent validity
• Criterion validity
• Predictive validity

13
Issues in Assessment Development

• What is content validity?
• Extent to which the items on the evaluation
  reflect a representative sample of that which is
  to be measured
• Do the items on the evaluation encompass what was
  taught?

14
Issues in Assessment Development

• What is construct validity?
• Extent to which the evaluation we implemented
  contained the content we intended it to and the
  outcome we measured was what we intended to
  measure
• Do the items on the evaluation test for what we
  wanted to test for?

15
Issues in Assessment Development

• What is concurrent validity?
• Extent to which an assessments results agree
  with the results of other assessments
  administered at approximately the same time to
  measure the same characteristics
• Does the assessment bear out or conflict with
  known and accepted standard measures or criteria?

16
Issues in Assessment Development

• What is criterion validity?
• Extent to which a test predicts some criterion
  (measure of performance), usually in the future
• Does doing well on this assessment predict that
  the person will do well elsewhere later in life?

17
Issues in Assessment Development

• What is predictive validity?
• Extent to which the results obtained by the
  evaluation agree with results obtained from more
  direct and objective measurements taken later on
• Does doing well on this assessment predict that
  the person will do well when tested on the same
  material later on?

18
Issues in Assessment Development

• If you dont have a statistician at your
  disposal, what other source do you have for
  validating your evaluation?

• SME

19
QUESTIONS?
20
Formative vs. Summative Evaluation

• What is Formative Evaluation?
• Assessment of the effectiveness of the training
  program while the program is still under
  development.
• What is Summative Evaluation?
• Assessment of the effectiveness of the training
  program after the program has been delivered.

21
QUESTIONS?
22
Kirkpatricks Four Levels

• What are Kirkpatricks 4 Levels of evaluation?

• Reaction
• Learning
• Behavior
• Results

23
Kirkpatricks Four Levels

• List some of the tools you could use to conduct a
  Level 1 evaluation.

• Reaction sheets
• Word-of-mouth feedback

24
Kirkpatricks Four Levels

• Which level assesses the overall impact of the
  program on the organization?

• Level 4

25
QUESTIONS?
26
ROI Methodology and Calculation

• What are some ways to assign monetary value to
  program improvement?
• Benchmarking
• Expert input
• Estimates
• Control groups
• Customer input

27
ROI Methodology and Calculation

• What is the formula for deriving ROI?
• ROI Total Benefits Program Costs X 100

Program Costs
28
ROI Calculation Exercise
29
QUESTIONS?
30
Three Measures of Central Tendency

• What is another word for measures of
  central tendency?

• Averages

• What are the three measures of central
  tendency we are concerned with today?

• Mean
• Median
• Mode

31
Three Measures of Central Tendency

• Define Mean
• The sum of all numbers divided by the number of
  values that make up the sum
• 64 73 82 98 317 (sum of values in list
  aka Sx)
• 317 4 79.25 (mean Sx n, where n is the
  number of values in the list)

32
Three Measures of Central Tendency

• Define Median
• The number separating the higher half of a sample
  from the lower half. The median of a finite list
  of numbers can be found by arranging all the
  observations from lowest value to highest value
  and picking the middle one. If there is an even
  number of observations, the median is not unique,
  so take the average of the two middle values.

33
Three Measures of Central Tendency

• Define Median
• Less sensitive to extreme scores than the mean
  is, so the median is a better measure than the
  mean for highly skewed distributions.

34
Three Measures of Central Tendency

• Define Median
• 64, 73, 82, 98 (scores from previous slide)
• 73 82 155 (even number of scores, so
  add the two in the middle
• and
• 155 2 77.5 divide the result in half to
  get the median)

35
Three Measures of Central Tendency

• Define Mode
• The most frequently occurring score in a
  distribution.
• Greatly affected by sample fluctuations, so not
  recommended for use as the only measure of
  central tendency.
• Distributions can be multi-modal contain more
  than one mode.

36
Three Measures of Central Tendency

• Define Mode
• 3 8 5 3 9 2 6 3

37
Three Measures of Central Tendency

• Define Mode
• 3 8 5 3 9 2 6 3
• Mode 3

38
Central Tendency Exercise
39
QUESTIONS?
40
Uses of Analysis

• What are some of the uses for ROI evaluation,
  cost-benefits, utilities, and forecasting?
• To measure a programs economic contribution
• To measure the degree of change in retention
• To isolate the effects of training

41
Uses of Analysis

• Case Problem Activity
• An organization has been running the same new
  hire orientation program for over a year.
• The HR director wants to determine the programs
  effectiveness, with a desire to improve it. Also,
  HR staff want to know why they seem to be
  answering the same questions repeatedly from new
  hires.

42
Uses of Analysis

• Case Problem Activity
• Your boss asks you, the primary trainer, to
  identify the strengths and weaknesses of the
  program.
• Company management has indicated they would like
  to offer the new hire orientation program online,
  in connection with links to the other helpful
  information that HR provides.

43
Uses of Analysis

• Program Evaluation Activity
• As a program evaluation consultant, you have been
  hired to evaluate a training program at Horizons,
  Inc.
• Complete the handout to perform your evaluation.

44
QUESTIONS?
45
Session Review

• Purpose and Benefits of Evaluating Training
• Measurement Process
• Issues in Assessment Development
• Formative vs. Summative Evaluation

46
Session Review

• Kirkpatricks Four Levels
• ROI Methodology and Calculation
• Three Measures of Central Tendency
• Uses of Analysis

47
Next Session

• Selecting and Planning Your Work Product

48
Questions?