Training 2003 Presentation: Wednesday, February 26

1
Training 2003 Presentation Wednesday, February
26
Conducting Effective Assessment, Measurement
Evaluation of Training
Roger Anderson Senior Consultant Tel (Brussels)
32 2 424 3423
2
Introduction to the Session

3
Assessment, Measurement and Evaluation Cycle
Organizational Opportunity/Problem
Evaluation Making a decision on basis of the
measurement, regarding the effect of the
intervention on opportunity/problem.
Assessment Determining the correct thing to do in
order to address the opportunity/problem.
Performance Measures Efficiency Effectiveness Qua
lity
Measurement Quantify the outcome of what is done
vis-à-vis the opportunity/problem
4
Performance Consulting System Model
Identify Client, Sponsor, and Stakeholder
Question Basic Assumptions
Assessment
Define the Needed Intervention
Plan Evaluation
Determine Metrics for Measuring Impact
Select Best Intervention
Measurement
Begin Collecting Data
Implement Intervention
Evaluation
Analyze the Data
Evaluate the Impact
5
Unit 1 Assessment Tools

6
Objectives

• By the end of this unit, you will
• Identify stakeholders, clients, and sponsors of
  the projected intervention
• Accurately identify the problem or opportunity by
  questioning basic assumptions
• Define needs in terms of business, performance,
  and/or training as determined through the problem
  identification process
• Identify operational indicators that are linked
  to the defined needs and establishes measures for
  these indicators
• Plan an evaluation strategy so that resources are
  properly assigned

7
Strategic Context of HR Interventions
The Cultural Context
Strategic Vision/ Purpose
Individual and Team Competencies
Work Flow/ Process
Learning
Support Structures/ Systems
8
Identifying the Client, Sponsor, and Stakeholders

• Stakeholdersthose whose work will be directly
  impacted by the changed produced by the
  intervention
• Clientthe person who will own the program the
  person most directly affected by the success of
  the intervention
• Sponsorthe highest-level advocate of the
  intervention the person who provides the power
  to implement the changes required

9
Identifying the Expressed Level of Need
Training Need
Performance Need
Business Need
Work Environment Need
10
Identifying the Expressed Level of Need

• Training Need A specific knowledge, skill, or
  process that a participant must learn to perform
  successfully.
• Performance Need Behaviors that must be
  performed by participants in order to achieve
  business needs.
• Business Need Strategic goals of an
  organization, business unit, or department that
  are expressed in operational terms.
• Work Environment Need A process, system, or
  condition that must be changed in order to help
  participants perform successfully.

11
Metrics for Measuring Impact

• Business What will attainment of business goals
  look like?
• Performance What behaviors are being performed
  effectively?
• Training What knowledge, skill, or process has
  an employee successfully learned?
• Work Environment What has been influenced as a
  result of environmental redesign?

12
Types of Evaluation

• Formative Evaluation
• Needs Analysis
• Stakeholder Analysis
• Objective Identification
• Program Design
• Pilot Testing
• Summative Evaluation

13
Unit 2 Linking HR, Organizational Strategy, and
Individual Performance

14
Objectives

• By the end of this unit, you will
• Differentiate between strategic and program
  evaluation
• Identify the components of a balanced scorecard
• Identify organizational measures of high
  performance work systems

15
The Impact of High Performance Work
Systems on Organizational Performance

• It is increasingly important today to build and
  maintain a skilled and motivated workforce in
  order to achieve strategic goals.
• Organizations face changing demands based on
  shifts in the marketplace.
• Organizational structures are changing to include
  broader spans of control.
• With a shift away from command and control
  culture, comes the need for employee to have
  increasing amounts of knowledge and to be able to
  apply it in a way which adds value to the
  organization.
• Employees are a competitive advantage, not a cost

16
The Balanced Scorecard Links Performance Measures
17
Example Balanced Scorecard Hi-Tech Manufacturer
18
Telling a Story with a Balanced Scorecard
19
Unit 3 Measurement Metrics and Operational
Indicators

20
Objectives

• By the end of this unit, you will
• Understand issues related to measurement of
  employee output
• Understand the fundamentals of survey design and
  Level Three (on-the-job application) evaluation

21
The Ultimate Measure

• The total value added to the organization by an
  employee the sum of all relevant performance.

Ultimate Measure
22
The Linkage Competency Paradigm
PERFORMANCE/RESULTS OUTCOMES
BEHAVIOR
SKILL
KNOWLEDGE
BELIEFS
VALUES
TRAITS
MOTIVES
Inferred
Observed
COMPETENCIES
23
Begin with the End in Mind

• What does your client value?
• What sorts of decisions need to be made?
• How will you summarize the results?
• Is qualitative or quantitative data needed?

24
Types of Data

• Qualitative
• Data collected via open-ended questions
• Subjective
• Quantitative
• Data collected via close-ended questions
• More objective

25
(No Transcript)
26
Qualitative Data

• Advantages
• Enables the participants to express their
  evaluation of the intervention in their own
  terms.
• Provides greater depth of critique data
• Disadvantages
• Data is more open to interpretation
• Can be difficult to collect compare like to
  like.
• Individual responses cannot be statistically
  compared one to one.

27
Quantitative Data

• Advantages
• Data can be statically compared across modules
  and courses
• Data is more efficiently collected and analyzed
• Disadvantages
• Expertise is needed to ensure results are not
  biased.
• Participants are forced to respond in a
  predetermined way.

28
Summarizing Results - Counts/Frequency
29
Summarizing Results - Average (Mean)
30
What to Measure?

• Whole job vs. part of job
• One measure vs. several measures
• Individual employees vs. group of employees
• Reactions, learning, application, impact
• Criteria
• Important to your client
• Linked to project objectives and scope
• Subject to relatively short-term change

31
Kirkpatrick Model
Level One Reaction
Level Two Learning
Level Three Application
Level Four Impact
32
Level One Reaction

• Purpose
• To gain feedback for course development and/or
  improvement by measuring participants reactions
  to the intervention.
• Format
• Questionnaire/Survey
• Immediate Feedback
• Data Collected Regarding
• Course content/concepts
• Instructor style
• Applicability of material to current job
• Course materials
• Facilities
• Work environment (transfer facilitators/inhibitors
  )

33
Level One Reaction

• Step One Determine specific information you wish
  to gather.
• Step Two Determine the specific questions to
  get the data.
• Step Three Determine the scale.
• Step Four Determine the scale descriptors.

34
Level Two Learning

• Purpose
• To determine participants level of mastery of
  course objectives.
• Format
• Performance Simulations
• Written Tests
• Data Collected Regarding
• Participants knowledge of course materials
• Participants ability to demonstrate the skills
• Participants fluency (speed) at using the skills
• Accuracy or quality of participants output
• Procedural automaticity

35
Level Two Learning

• Step One Design learning measures.
• Step Two Consider test use.
• Step Three Develop means of increasing validity
  or reliability. (Optional)

36
Level Three Application

• Purpose
• To determine the extent to which skills/behaviors
  developed during the intervention are being used
  on the job.
• Format
• Questionnaire/Survey
• Behavioral Checklist
• Frequency Checklist
• 360-Degree Survey
• Managerial Appraisal
• Data Collected Regarding
• Participants on-the-job application
• Factors encouraging application
• Factors hindering application
• Follow-up support required to assist in continued
  development

37
Level Three Application

• Step One Determine basis for evaluating
  on-the-job application.
• Step Two Decide how to collect information,
  and when to collect the information
• Step Three Design the evaluation instrument.
• Step Four Pilot the evaluation instrument.
• Step Five Develop means of increasing
  objectivity of data.(Optional)
• 1. Use multiple observers
• 2. Train observers
• 3. Use 360-degree methodology

38
Level Four Impact

• Purpose
• To determine the impact on the organization of
  participants application of training on-the-job.
• Format
• Review of Records
• Observation
• Efficiency Ratios
• Data Collected Regarding
• Change in individual/organizational output
• Change in economic performance

39
Level Four Impact

• Step One Develop a system for measuring impact.
• 1.What are the business needs/objectives of the
  course?
• 2. What operational indicators reflect these
  needs/objectives?
• 3. What specific knowledge and skills being
  developed in the intervention are linked to the
  indicators being tracked?
• 4. What indicators currently exist do you want
  to develop others?
• 5. What time intervals will you use to determine
  that the impacts are taking place?
• Step Two Develop a means for collecting data.
• 1. Measure current performance for each
  operational indicator.
• 2. Involve line personnel in the collection of
  Level Four data.

40
Unit 4 Evaluation Strategy

41
Objectives

• By the end of this unit, you will
• Understand how to develop evaluation strategies
  that will allow you to discern the effectiveness
  of the intervention by
• Separating the impact of your intervention from
  other organizational forces
• Identifying the factors that can destroy your
  evaluation efforts
• Ensuring that your sample matches the situation
  that you are making decisions about

42
Separating Effects of Training from Other
Influences

• Proving Causes
• versus
• Providing Evidence

43
Separating Effects of Training from Other
Influences

• Know the business you are consulting for.
• Reach agreement with your client as to what the
  most appropriate measures are get your clients
  to own the measures.
• Obtain pre-measures.
• Use control groups.
• Use multiple sources of information
• Identify level of performance required for
  business need/opportunity.
• Plan timing of data collection.

44
Control Groups
Example 1 Other Organizational Influences
45
Control Groups
Example 2 Remedial Training
46
Control Groups
Example 3 Alternative Interventions
47
Control Groups
Example 4 Predetermined Level of Need
48
Designing Control Groups

• Control groups should be as similar as possible
  in terms of relevant attributes and performance.
• Matched Control Groups
• Random Control Groups
• Convenience Control Groups

49
Multiple Sources of Information in Evaluation
Reaction
Knowledge
Application
Impact
50
Maintenance of Trainings Effect (Transfer)
51
What Supports/Inhibits Transfer?

• Organization supports/inhibitors
• Manager supports/inhibitors
• Trainer supports/inhibitors
• Participant supports/inhibitors

52
Steps for Evaluation Strategy

• Step One Define the problem or opportunity.
• Step Two Identify the population affected.
• Step Three Identify the performance factors and
  metrics associated with the problem/opportunity.
• Step Four Establish a baseline for current
  performance.
• Step Five Determine the most effective form of
  intervention.
• Step Six Crosswalk the objectives.
• Step Seven Implement the intervention.
• Step Eight Evaluate effectiveness/impact.
• Step Nine Assess impact on problem or
  opportunity.

53
Framing Meeting Topics

• Problem/opportunity to address
• Client, sponsor, stakeholders
• Operational indicators and metrics
• Time frame in which measures will be taken
• Communication agreements and plans
• Roles and accountabilities
• Assessment
• Design of evaluation
• Design of intervention
• Implementation of intervention
• Data collection
• Data analysis

54
Methods for Sampling

• Step One Begin with complete list of people
  involved in the intervention.
• Step Two Identify relevant demographics for
  examining differences between groups.
• Step Three Select a sampling strategy
• Simple random sampling
• Systematic sampling
• Stratified sampling
• Convenience sampling
• Matched sampling

55
Determining Sample Size

• Focus on what your evaluation goals and strategy
  are.
• Dont get bogged down with exact numbers.
• Assume low return rate for surveys.
• Consider
• What resources are available?
• How strong an impact do you anticipate from the
  intervention?
• How many variables are you using in stratifying
  the sample?

56
Methods of Increasing Return Rate
57
Unit 5 Appendix Costs of HR Interventions

58
Objectives

• By the end of this unit, you will
• Understand the process for calculating costs
  associated with the development and
  implementation of an HR intervention
• Recognize that there are many assumptions being
  made when conducting a cost-benefit analysis
  (i.e., this is not an exact science)
• Determine the method for computing the benefit or
  impact of the intervention

59
Glossary of Terms

• Cost Benefit Analysis - An analysis, which
  compares the costs and benefits of the program.
• Direct (out-of-pocket) Costs - Expenses you
  actually pay out money for in delivering a
  service.
• Fully Loaded Labor Costs - The dollar value of
  the time of all people directly involved in
  delivering a service, including fringe benefits
  and overhead costs.
• Labor Multiplier - A constant for a particular
  position that is used to calculate the fully
  loaded cost of a persons time. Direct Labor x
  Labor Multiplier Fully Loaded Labor Cost
• Opportunity Cost - Money the organization loses
  by using people and resources one way rather than
  another, e.g. sales or production lost when
  people attend a training program instead of
  performing their normal job functions.

60
Measurement Tools and Action Steps

• Step One Establish metrics/measures.
• Step Two Estimate costs for proposed/establishe
  d intervention.
• Step Three Estimate benefits of
  proposed/established intervention.
• Step Four Calculate the resulting cost benefit
  ratio for program

Cost benefit ratio Gain return
Costs investment Return on investment
Gain return - Cost investments
Costs investment
61
Cost of Delivery

• Direct (out-of-pocket) costs
• Fully loaded labor costs
• Opportunity costs

62
Benefits

• Benefits can be obtained in two ways
• Increase in production
• Decrease in operating costs
• Determining potential benefits is a crucial part
  of the performance consulting process.
• Your client provides you information on
• What are the important operational indicators?
• What are the expected goals of the intervention?
• What are the dollar figures attached to the
  operational indicators?
• Your role as a consultant is to
• Ask the right questions that will give you
  answers to the questions above.
• Help your client identify what changes might be
  expected from the intervention.

63
Realizing the Benefits of Training
Costs
Benefits
Options Reduce Operating Costs Increase Benefits
(Improve Productivity)
64
Key Terms

• Direct Costs (out of pocket)
• Expenses you pay out
• Meals
• Travel
• Materials
• Fully Loaded Labor Costs
• value of the time people spend delivering a
  service
• Payroll
• Benefits (25 of payroll)
• Overhead (120 of payroll and benefits) (labor
  multiplier)
• Opportunity Costs
• the organization loses by using people and
  resources one way rather than another
• Sales lost when sales people attend training
• Production lost when operations manager attend
  training

65
Cost is Only Meaningful in Light of Benefits...
Begin Project
Complete Project
February
September
Conduct Proj. Mgt. Training
Begin Project
Complete Project
Saved
January
August
February
66
The Cost of a Learning Curve
Existing Employee
Cost of Learning
Close the gap with training
New Hire
Direct Labor Cost/Day
of Days
Below Effectiveness
Labor Multiplier
X
X
X
67
Measure Estimated Benefit Turnover 6 Months
to 2 Years of Salary Absenteeism Cost of
Replacement/Temps Scrap Cost per Unit of
Scrap Project Completion Time Labor Costs
68
Biography Roger Anderson, Senior Consultant,
Linkage, Inc.

• Roger Anderson has over twenty-five years of
  experience in training and organizational
  development. He is currently a member of the
  Leadership and Strategy Group at Linkage and his
  practice is focused on leadership development and
  executive coaching. His recent coaching and
  leadership development clients include The United
  Nations, Adgas, Pfizer Consumer Health Care, the
  Belgian Armed Forces, monster.com and the
  Washington Hospital Center.
• Roger has provided his clients with the
  assessment of their leadership needs, designed
  and implemented leadership development systems at
  the organization level, provided one-on-one
  executive coaching, and designed and implemented
  programs to develop the capability of leadership
  teams. He has designed and developed action
  learning programs for his clients which have
  enabled the organizations leaders to develop
  their leadership activities while accomplishing
  critical work outcomes.
• Roger has developed competency models and
  competency-based leadership development systems,
  performance management systems, selection
  systems, and succession management systems in a
  variety of industries including financial
  services, government, technology, and health
  care.
• Rogers other clients at Linkage have included
  Federal Express, Brigham Womens Hospital,
  ATT, LEGO Systems, the International
  Organization for Migration in Geneva,
  Switzerland, Kodak, Centura Health Systems, and
  the Childrens Television Network, the U.S.
  Department of Labor, and the Virginia Department
  of Transportation.
• As an undergraduate, Roger was involved in voter
  registration drives in the Watts area of Los
  Angeles. He spent two years as a Peace Corps
  Volunteer in Ahvaz, Iran. He received his Master
  of Human Resource Education Degree from Boston
  University where his work focused on the
  development of a model for determining the
  value-added benefit of training and development
  programs.

69
About Linkage, Inc.

• Linkage, Inc. is an internationally recognized
  leadership and organizational development
  consulting firm
• Founded in 1988, with offices in Atlanta,
  Minneapolis, San Francisco, Brussels, and London,
  and headquarters in Lexington, Massachusetts.
• Dedication to creating successful programs and
  customized materials/tools that address specific
  development needs, combining assessments,
  strategic education, skill-building, and global
  networks.
• Leading provider of competency-based training on
  leadership and management development.
• Designing and facilitating training programs that
  are highly interactive, practical, memorable,
  compelling, and offering skills and tools for
  on-the-job use.
• Instructors and designers who are Principal and
  Senior Consultants with extensive experience
  developing and delivering development for a wide
  variety of leading organizations.
• Commitment to a comprehensive solution that
  combines theoretical expertise and practical
  experience into a clear sequence of developmental
  opportunities with measurable impact.
• Creator of the High Performance Leadership Model
  (in conjunction with Warren Bennis).
• Assessment and evaluation expertise and resources
  to measure the business impact of development
  programs, as well as to evaluate the strengths
  and developmental opportunities of individuals
  and the organization.

70
Training 2003 Presentation Conducting Effective
Assessment, Measurement Evaluation of Training
To contact presenter, Roger Anderson Tel
(Brussels) 32 2 424 3423 For more information
about Linkage, Inc. products and services Tel
[phone] E-mail Info_at_Linkage-Inc.com Web
www.LinkageInc.com