Mr. Shaw & Shearer SafeMark Demo to DNFSB

1
Leading Indicators for ESH Performance
Measuring How Well Our Management System Is
Performing
ISM Best Practices Workshop September 12, 2006
Steven R. Woodbury DOE Office of Nuclear Safety
and Environmental Policy
2
Overview

• The mix of ESH performance measures changes as
  an ESH program matures.
• DOE currently uses many lagging measures.
• Our next level of performance measurement
  should focus on how well key elements of our
  ESH management system are performing.
• There are instructive examples in the literature
  of best practices on which we can build.
• But nobody says this is easy.

3
Three Levels of Performance Measurement

• Grace Wever identifies three types of
  performance measures that organizations may
  employ
• 1. Operational indicators (lagging)
• For example
• waste generation
• accident/injury rates
• regulatory violations
• 2
• 2. Identifying underlying management system
  deficiencies (leading) For example, the quality
  of
• training
• audits
• communications
• documentation
• review processes

4
Three Levels of Performance Measurement
(continued)

• 3. Identifying gaps in internal competencies
  and external relationships (leading)For
  example,
• Organization begins to measure
• Impacts of products, operations, or services on
  public health and the environment
• EHS research and development
• Capital investments
• Stakeholder and customer partnerships
• Organization strives to link financial, human
  resource, operation, EHS, and other goals and
  measures
• The mix of an organization's performance
  indicators tends to reflect the maturity of its
  EHS program.
• Grace Wever, Strategic
  Environmental Management, pp 158-9

5
Importance of Leading Indicators

• The importance of developing good leading
  indicators goes beyond Its hard to steer by
  looking in the rearview mirror.
• The importance goes beyond continuous improvement
  We should do what leading organizations are
  doing.
• We need to be concerned because even good
  performance, on good lagging indicators, cant
  assure us that risks are really being controlled.

6
Outcome Measures are Not Sufficient

• When safety is good and injury and loss rates
  are low, then outcome measurements are not
  sufficient to provide adequate feedback for
  managing safety.
• For operations where there may be potential
  for severe accidents, the likelihood of such
  events must be extremely low. This means that
  the absence of very unlikely events is not, of
  itself, a sufficient indicator of good safety
  management.
• --European Process Safety Center, 1996, p. 3.

7
Outcome Measures are Not Sufficient

• Health and safety differs from many areas
  measured by managers because success results in
  the absence of an outcome (injuries or ill
  health) rather than a presence. But a low
  injury or ill-health rate, even over a period of
  years, is no guarantee that risks are being
  controlled and will not lead to injuries or ill
  health in the future. This is particularly true
  in organizations where there is a low probability
  of accidents but where major hazards are
  present.
• A Guide to
  Measuring Health Safety Performance Health
  Safety Executive (UK), p 5

8
Some Best Practices in Next Level Performance
Measurement

• Guide to Measuring Health Safety Performance
  (British Health and Safety Executive)
• Total Quality Environmental Management Matrix
  (Grace Wever, Council of Great Lakes Industries)
• Positive Performance Indicators (Australia,
  Department of Employment and Workplace
  Relations)
• General Electric Company
• ProSmart software (Chemical Process Safety
  Council)

9
Health Safety Executive (UK)

• Guide discusses performance measurement in the
  context of a health and safety management system
• Discusses the limitations of traditional
  lagging measures
• Provides a systematic approach to deriving
  measures which link to the HS management system

10
Why Measure Performance?

• The primary purpose of measuring HS
  performance is to provide information on the
  progress and current status of the strategies,
  processes and activities used by an
  organization to control risks to health and
  safety
• A Guide to Measuring Health Safety
  Performance Health Safety Executive (UK), p 7

11
HSE Guide a Balanced Approach

• Performance measurement should be based on a
  balanced approach, which addresses
• Input
• Monitoring the scale, nature and
  distribution of hazards created by the
  organizations activities
• Process
• Active monitoring of the adequacy,
  development, implementation and deployment of the
  HS management system
• Outcomes
• Reactive monitoring of adverse outcomes
  resulting in injuries, ill health, loss and
  accidents with the potential to cause injuries
  ill health or loss (measures of failures)

12
HSE Guide Measure the Management System

• The HS management system is the process which
  turns uncontrolled hazards to controlled risks
• The key elements of
• Policy
• Organizing
• Planning and implementation
• Measuring performance
• Audit and review
• all need to be in place to control risks
  effectively
• The performance measurement system must cover
  each element of the HS management system

13
HSE Guide What and How to Measure

• Measuring elements of the HS management system
  should cover three aspects
• Capability
• Implementation
• Deployment
• The measurement process can gather information
  through
• Direct observation of conditions and peoples
  behavior
• Talking to people to elicit facts and their
  experiences as well as gauging their views and
  opinions
• Examining written reports, documents and records

14
HSE Guide Observations

• Performance measurement is approached in the
  context of the health and safety management
  system
• Guide acknowledges the need to measure outcomes
  but this is not sufficient
• measures of failure
• doesnt guarantee that risks are really
  controlled
• Approach is not limited to Health and Safety
  completely applicable to management across ESH
• Focus is on the performance of key management
  system elements
• Measures are largely qualitative
• Provides a clear framework for thinking about
  ESH management and performance measurement

15
Total Quality Environmental Management Matrix
Seven Categories

• TQEM identifies 7 performance categories, based
  on Baldrige Criteria most relevant to EHS
  management, as well as the ISO 14001 standard
• Identifies performance levels (110) (based on
  descriptive text)
• Detailed self-assessment questionnaire is
  provided to support evaluation.
• Within a category, one cannot score at a higher
  level unless performance is fully satisfactory at
  all lower levels
• Developed in the 1990s by Grace Wever, and the
  Council of Great Lakes Industries

Ten Performance Levels
16
Total Quality Environmental Management Matrix
Categories

• Seven categories (different weights)
• Leadership (15)
• Information and Analysis (7.5)
• Strategic Planning (7.5)
• Human Resource Development (10)
• Process Management (15)
• Environment, Health Safety Results (30)
• Customer/Stakeholder Satisfaction (15)

17
TQEM Performance Levels Environmental Health
and Safety Results

• Example ESH Results Category
• 10. Benchmarking shows unit is best-in-class
• 7. EHS improvements contribute to financial and
  business improvements
• 5. Positive improvement trends in units key
  EHS measures
• 4. EHS measures reviewed and improved at least
  annually
• 3. EHS results compared with objectives and
  targets used to improve effectiveness of
  management systems and performance
• 1. EHS performance measures identified baseline
  data and trends collected

EHS Results
18
TQEM Performance Levels Strategic Planning

• Example Strategic Planning Category
• 9. Information on competitors EHS strategies
  used to improve units strategies and performance
• 8. EHS integrated into long and short-term
  business plans for all of units products,
  processes and services.
• 6. EHS plans and deployment consistently aligned
  at all levels of the unit
• 4. Long and short-term plans that include EHS
  objectives are reviewed and improved at least
  annually. Key measures include EHS measures
• 1. Long-term and short-term planning process used
  that addresses EHS needs annual operating plan
  includes EHS management needs

Strategic Planning
19
TQEM Performance Levels Human Resource
Development

• Example HR Development Category
• 9. EHS needs fully integrated into units HR
  development plan. Training/education for EHS
  staff include key business knowledge
• 7. Employees proactively initiate activities to
  improve EHS performance
• 5. Measures of EHS training effectiveness in
  place employees with potential to impact EHS are
  competent to perform EHS responsibilities
• 3. All employees have received appropriate EHS
  training employees aware of EHS compliance
  requirements.
• 1. EHS training needs identified, resources
  committed.

Human Resource Development
20
TQEM Matrix Observations

• Not limited to environment completely
  applicable to management across ESH
• System is based on explicit, but qualitative,
  criteria for ESH management
• Focus is on qualitative measures of performance
  in key management categories
• Published matrix is fully defined (not
  open-ended)
• Matrix was developed more than 10 years ago it
  should be updated on the basis of additional
  experience
• Its importance is as a conceptual framework
  the specific content can be adapted

21
Positive PerformanceIndicators

• Positive Performance Indicators (PPIs) focus on
  assessing how successfully an organization is
  performing by monitoring the processes that
  provide good OHS outcomes.
• PPIs are to be developed in addition to
  traditional outcome indicators.
• PPI program was developed by the Australian
  Governments Department of Employment and
  Workplace Relations

22
Positive Performance Indicators Based on
Quality Management

• PPI is based on a Quality Management approach.
  PPIs may be
• Inputs (Key Activities)
• Measures of actions or initiatives undertaken to
  improve OHS
• Provide indicators of commitment and effort, but
  not of effectiveness
• Can provide useful information on participation,
  leadership and communication
• Processes (Monitoring Key Risks)
• Identify key risks
• Identify key contributors to outcomes of concern
• Develop measures to monitor behaviors and
  practices
• Outputs (Milestones)
• Achievement of objectives
• Progress toward achievement of higher-level OHS
  goals and targets

23
Positive Performance Indicators Part of a
Management System

• PPIs are only useful if an organization has a
  systematic approach to management of OHS
• Many variations exist, but all have the following
  principles
• Commitment and policy
• Planning
• Implementation
• Measurement and evaluation
• Review and improvement

24
Positive Performance Indicators Categories of
PPIs

• Commitment and policy
• Measures demonstrated commitment to improve OHS
  performance
• Planning
• Measures what procedures are in place to
  eliminate workplace injury and disease
• Implementation
• Measures the capability and support mechanisms
  that are necessary to achieve OHS objectives and
  targets
• Measurement and Evaluation
• Measures the extent to which workplace health and
  safety is monitored and evaluated so that issues
  can be identified and corrective action taken
• Review and improvement
• Measures the effectiveness of the OHS management
  system, and its continuing suitability

25
Examples of PPIs Commitment and Policy
26
Examples of PPIs Implementation
27
Positive PerformanceIndicators Observations

• PPIs are not limited to Occupational Health and
  Safety completely applicable to management
  across ESH
• PPIs focus on identifying key elements of the
  ESH management system, and measuring how well
  these key elements are functioning
• PPIs are non-prescriptive and open-ended
  approach provides great flexibility at the
  facility and site level
• Most PPIs are qualitative, based on assessments,
  record reviews, surveys, or audits
• The Guidance document assumes a fairly basic
  management system. DOE sites generally have more
  sophisticated ESH management systems in place,
  with the opportunity to develop more
  sophisticated measures.

28
General Electric Health and Safety Framework

• At the April ISM meeting in Albuquerque, Kurt
  Krueger described GEs corporate-wide Health and
  Safety program
• One standard, one program, one set of metrics
  for every GE facility around the world No
  exceptions!
• GEs Health and Safety Framework includes a mix
  of 21
• Management system elements For example
• Training
• Accident investigation and follow-up
• Subject matter areas For example
• Industrial hygiene
• Chemical management
• Motor vehicle safety

29
General Electric (continued)

• Detailed questions
• GE-specific guidance expectations

30
General Electric (continued)

• Scorecard ratings are based on
• Plant self-assessment (540 questions, twice a
  year)
• Periodic corporate audits
• Tied to monthly site operational metrics and
  supervisor scorecards,
• tailored to site operations
• designed to drive supervisor behaviors that will
  find and fix HS issues before an accident finds
  them
• GE has a parallel system for assessing
  environmental performance
• GE also continues to track trailing metrics
  For example
• Recordable injury and illness rates
• Lost time injury and illness rates
• Permit exceedances
• Notices of noncompliance

31
General Electric Observations

• Integrated into GEs broader management system
• Reflects a balance between frequent site
  self-assessment and reporting, and periodic
  headquarters audits
• GE chose to implement a uniform corporate system.
  While this might not be DOEs choice, there is
  still a lot to learn from the overall framework,
  and from the measures GEs chose to include

32
ProSmart Center for Chemical Process Safety

• In 1993, CCPS began a project to measure the
  effectiveness of process safety management
  systems
• Their objective wasto provide management with
  the tools for assessing the health of process
  safety management systems on a real-time basis
• Based around 12 management system elements
• Extensive set of computer-based questions, using
  qualitative and quantitative measures
• Rolls up responses into a single score or index

33
CCPS ProSmart Based on 12 Management System
Elements

• CCPS identified 12 key management system
  elements, such as
• Accountability
• Process knowledge and documentation
• Capital project hazard review
• Training and performance
• Management of change
• Incident investigation
• Audits and corrective action
• ProSmart generates a score, which is continually
  updated

34
CCPS ProSmart Based on 12 Management System
Elements
35
CCPS ProSmart Drills Down to Detailed
Evaluation Questions
36
ProSmart Observations

• Derived from a quality management approach
• Focus is on specific management system elements
• Established for chemical process safety, but in
  principle could encompass environment,safety and
  health
• Like TQEM and GE, ProSmart provides a set of
  detailed assessment questions, for qualitative
  assessment against established criteria

37
Key Aspects of Next Level Performance Measures

• Next Level performance measures identify and
  measure how well key elements of the ESH
  management system are functioning
• Measures tend to involve
• Precursor analysis and trending
• Observations, assessments and audits
• Measures tend to be qualitative rather than just
  counting events
• (This doesnt mean they cant be rigorous
  and replicable)

38
DOE Has Many Pieces in Place

• Analysis and trending
• Data collection
• Trending and analysis
• Root cause analysis
• Observations, assessments, audits
• Contractor self-assessment processes
• DOE oversight activities
• Annual ISM assessments should systematicallyaddre
  ss management system elements
• Inspections by the Office of ESH Inspections
  (HS-64, formerly SP-44) address ESH management
  system functioning at a high level (ISM core
  functions)
• Individual IG audits address some management
  system elements

39
Audits and Inspections
40
ESH Inspections

• (colors added)
• Define Scope of Work
• Analyze the Hazards
• Identify and Implement Controls
• Perform Work Within Controls
• Feedback and Continuous Improvement
• (contractor) (DOE)
• Essential System Functionality

41
ESH Evaluations of 17 DOE Sites 2002-2004
42
Summary

• There is a common thread of best practice, across
  many different organizations to move beyond
  outcome (lagging) indicators, we need to start
  systematically measuring how well key elements
  of our management system are performing.
• This is not as easy as counting occurrences
  (injuries, enforcement actions) it must involve
  walk-throughs, audits, surveys, and other varied
  approaches.
• This next level of leading performance
  measurement complements, and does not replace,
  outcome (lagging) measures.
• Approaches developed for process safety, worker
  safety, environment, can be applied across ESH.
  We need to be eclectic and ecumenical in our
  thinking.
• Nobody has a turnkey system thats right for DOE.
  We need to identify and build on the best of
  these various approaches.
• Nobody says it will be easy.

43
Next Steps

• Identify key ESH management system elements
• We currently categorized some information (e.g.,
  ORPS reports Independent Oversight Assessments)
  around ISMs five broad core functions
• Within these core functions, we can identify more
  specific management system elements, including
• Identification of requirements - Corrective
  action management
• Hazard identification - Lessons
  Learned sharing
• Communication -
  Performance Measurement
• Training -
  Worker involvement
• Documentation - Audits
  and assessments
• Periodic top management review

44
Next Steps (continued)

• Identify the various things we are already doing
  to assess how well these elements are functioning
• Identify and fill gaps as appropriate
• Organize and present the information in a way
  which provides current feedback to management on
  how effectively key management system elements
  are functioning at each site and each facility
• In Sum Use the health of these key management
  system elements as the conceptual and organizing
  framework to strengthen our ability to know how
  well we are managing risks

45
In Closing

• Far better an approximate answer to the right
  question which may be difficult to frame,
• than an exact answer to the wrong
  questionwhich is always easy to ask.
• statistician John
  W. Tukey

46
References

• Grace Wever, 1996, Strategic Environmental
  Management Using TQEM and ISO 14000 for
  Competitive Advantage, John Wiley and Sons, NY
• European Process Safety Centre (Jacques van
  Steen, ed.), 1996, Safety Performance
  Measurement, Institution of Chemical Engineers,
  Rugby (UK) (distr by Gulf publishing Co., PO Box
  2608, Houston, TX 77252-2608)
• Health and Safety Executive (United Kingdom),
  2001, A Guide to Measuring Health Safety
  Performance http//www.hse.gov.uk/opsunit/perfmea
  s.pdf
• Commonwealth of Australia (Department of
  Employment and Workplace Relations), 2005,
  Guidance on the Use of Positive Performance
  Indicators to Improve Workplace Health and
  Safety http//www.nohsc.gov.au/PDF/Standards/Guida
  nceNotes/ASCCPPIGuidanceBooklet.pdf
• Kurt Krueger, Creating a Culture of Safety
  Excellence The Journey and the Prize, talk at
  DOE ISM Champions Workshop, Albuquerque NM, April
  2006 http//www.eh.doe.gov/ism/workshops/Creating_
  a_Culture_of_Safety_Excellence.pdf
• George Eckes, The Six Sigma Revolution How GE
  and Others Turned Process into Profits, New
  York John Wiley and Sons, 2000.
• Center for Chemical Process Safety, ProSmart
  software http//www.aiche.org/CCPS/Publications/So
  ftware/ProSmart/index.aspx

47
Contact Information

• Steven R. Woodbury
• DOE Office of Nuclear Safety and Environmental
  Policy
• 202-586-4371
• steven.woodbury_at_eh.doe.gov