Mr. Shaw & Shearer SafeMark Demo to DNFSB - PowerPoint PPT Presentation


PPT – Mr. Shaw & Shearer SafeMark Demo to DNFSB PowerPoint presentation | free to view - id: 3c3af6-OWZhM


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation

Mr. Shaw & Shearer SafeMark Demo to DNFSB


Leading Indicators for ES&H Performance: Measuring How Well Our Management System Is Performing ISM Best Practices Workshop September 12, 2006 Steven R. Woodbury – PowerPoint PPT presentation

Number of Views:69
Avg rating:3.0/5.0
Slides: 48
Provided by: hssDoeGov


Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Mr. Shaw & Shearer SafeMark Demo to DNFSB

Leading Indicators for ESH Performance
Measuring How Well Our Management System Is
ISM Best Practices Workshop September 12, 2006
Steven R. Woodbury DOE Office of Nuclear Safety
and Environmental Policy
  • The mix of ESH performance measures changes as
    an ESH program matures.
  • DOE currently uses many lagging measures.
  • Our next level of performance measurement
    should focus on how well key elements of our
    ESH management system are performing.
  • There are instructive examples in the literature
    of best practices on which we can build.
  • But nobody says this is easy.

Three Levels of Performance Measurement
  • Grace Wever identifies three types of
    performance measures that organizations may
  • 1. Operational indicators (lagging)
  • For example
  • waste generation
  • accident/injury rates
  • regulatory violations
  • 2
  • 2. Identifying underlying management system
    deficiencies (leading) For example, the quality
  • training
  • audits
  • communications
  • documentation
  • review processes

Three Levels of Performance Measurement
  • 3. Identifying gaps in internal competencies
    and external relationships (leading)For
  • Organization begins to measure
  • Impacts of products, operations, or services on
    public health and the environment
  • EHS research and development
  • Capital investments
  • Stakeholder and customer partnerships
  • Organization strives to link financial, human
    resource, operation, EHS, and other goals and
  • The mix of an organization's performance
    indicators tends to reflect the maturity of its
    EHS program.
  • Grace Wever, Strategic
    Environmental Management, pp 158-9

Importance of Leading Indicators
  • The importance of developing good leading
    indicators goes beyond Its hard to steer by
    looking in the rearview mirror.
  • The importance goes beyond continuous improvement
    We should do what leading organizations are
  • We need to be concerned because even good
    performance, on good lagging indicators, cant
    assure us that risks are really being controlled.

Outcome Measures are Not Sufficient
  • When safety is good and injury and loss rates
    are low, then outcome measurements are not
    sufficient to provide adequate feedback for
    managing safety.
  • For operations where there may be potential
    for severe accidents, the likelihood of such
    events must be extremely low. This means that
    the absence of very unlikely events is not, of
    itself, a sufficient indicator of good safety
  • --European Process Safety Center, 1996, p. 3.

Outcome Measures are Not Sufficient
  • Health and safety differs from many areas
    measured by managers because success results in
    the absence of an outcome (injuries or ill
    health) rather than a presence. But a low
    injury or ill-health rate, even over a period of
    years, is no guarantee that risks are being
    controlled and will not lead to injuries or ill
    health in the future. This is particularly true
    in organizations where there is a low probability
    of accidents but where major hazards are
  • A Guide to
    Measuring Health Safety Performance Health
    Safety Executive (UK), p 5

Some Best Practices in Next Level Performance
  • Guide to Measuring Health Safety Performance
    (British Health and Safety Executive)
  • Total Quality Environmental Management Matrix
    (Grace Wever, Council of Great Lakes Industries)
  • Positive Performance Indicators (Australia,
    Department of Employment and Workplace
  • General Electric Company
  • ProSmart software (Chemical Process Safety

Health Safety Executive (UK)
  • Guide discusses performance measurement in the
    context of a health and safety management system
  • Discusses the limitations of traditional
    lagging measures
  • Provides a systematic approach to deriving
    measures which link to the HS management system

Why Measure Performance?
  • The primary purpose of measuring HS
    performance is to provide information on the
    progress and current status of the strategies,
    processes and activities used by an
    organization to control risks to health and
  • A Guide to Measuring Health Safety
    Performance Health Safety Executive (UK), p 7

HSE Guide a Balanced Approach
  • Performance measurement should be based on a
    balanced approach, which addresses
  • Input
  • Monitoring the scale, nature and
    distribution of hazards created by the
    organizations activities
  • Process
  • Active monitoring of the adequacy,
    development, implementation and deployment of the
    HS management system
  • Outcomes
  • Reactive monitoring of adverse outcomes
    resulting in injuries, ill health, loss and
    accidents with the potential to cause injuries
    ill health or loss (measures of failures)

HSE Guide Measure the Management System
  • The HS management system is the process which
    turns uncontrolled hazards to controlled risks
  • The key elements of
  • Policy
  • Organizing
  • Planning and implementation
  • Measuring performance
  • Audit and review
  • all need to be in place to control risks
  • The performance measurement system must cover
    each element of the HS management system

HSE Guide What and How to Measure
  • Measuring elements of the HS management system
    should cover three aspects
  • Capability
  • Implementation
  • Deployment
  • The measurement process can gather information
  • Direct observation of conditions and peoples
  • Talking to people to elicit facts and their
    experiences as well as gauging their views and
  • Examining written reports, documents and records

HSE Guide Observations
  • Performance measurement is approached in the
    context of the health and safety management
  • Guide acknowledges the need to measure outcomes
    but this is not sufficient
  • measures of failure
  • doesnt guarantee that risks are really
  • Approach is not limited to Health and Safety
    completely applicable to management across ESH
  • Focus is on the performance of key management
    system elements
  • Measures are largely qualitative
  • Provides a clear framework for thinking about
    ESH management and performance measurement

Total Quality Environmental Management Matrix
Seven Categories
  • TQEM identifies 7 performance categories, based
    on Baldrige Criteria most relevant to EHS
    management, as well as the ISO 14001 standard
  • Identifies performance levels (110) (based on
    descriptive text)
  • Detailed self-assessment questionnaire is
    provided to support evaluation.
  • Within a category, one cannot score at a higher
    level unless performance is fully satisfactory at
    all lower levels
  • Developed in the 1990s by Grace Wever, and the
    Council of Great Lakes Industries

Ten Performance Levels
Total Quality Environmental Management Matrix
  • Seven categories (different weights)
  • Leadership (15)
  • Information and Analysis (7.5)
  • Strategic Planning (7.5)
  • Human Resource Development (10)
  • Process Management (15)
  • Environment, Health Safety Results (30)
  • Customer/Stakeholder Satisfaction (15)

TQEM Performance Levels Environmental Health
and Safety Results
  • Example ESH Results Category
  • 10. Benchmarking shows unit is best-in-class
  • 7. EHS improvements contribute to financial and
    business improvements
  • 5. Positive improvement trends in units key
    EHS measures
  • 4. EHS measures reviewed and improved at least
  • 3. EHS results compared with objectives and
    targets used to improve effectiveness of
    management systems and performance
  • 1. EHS performance measures identified baseline
    data and trends collected

EHS Results
TQEM Performance Levels Strategic Planning
  • Example Strategic Planning Category
  • 9. Information on competitors EHS strategies
    used to improve units strategies and performance
  • 8. EHS integrated into long and short-term
    business plans for all of units products,
    processes and services.
  • 6. EHS plans and deployment consistently aligned
    at all levels of the unit
  • 4. Long and short-term plans that include EHS
    objectives are reviewed and improved at least
    annually. Key measures include EHS measures
  • 1. Long-term and short-term planning process used
    that addresses EHS needs annual operating plan
    includes EHS management needs

Strategic Planning
TQEM Performance Levels Human Resource
  • Example HR Development Category
  • 9. EHS needs fully integrated into units HR
    development plan. Training/education for EHS
    staff include key business knowledge
  • 7. Employees proactively initiate activities to
    improve EHS performance
  • 5. Measures of EHS training effectiveness in
    place employees with potential to impact EHS are
    competent to perform EHS responsibilities
  • 3. All employees have received appropriate EHS
    training employees aware of EHS compliance
  • 1. EHS training needs identified, resources

Human Resource Development
TQEM Matrix Observations
  • Not limited to environment completely
    applicable to management across ESH
  • System is based on explicit, but qualitative,
    criteria for ESH management
  • Focus is on qualitative measures of performance
    in key management categories
  • Published matrix is fully defined (not
  • Matrix was developed more than 10 years ago it
    should be updated on the basis of additional
  • Its importance is as a conceptual framework
    the specific content can be adapted

Positive PerformanceIndicators
  • Positive Performance Indicators (PPIs) focus on
    assessing how successfully an organization is
    performing by monitoring the processes that
    provide good OHS outcomes.
  • PPIs are to be developed in addition to
    traditional outcome indicators.
  • PPI program was developed by the Australian
    Governments Department of Employment and
    Workplace Relations

Positive Performance Indicators Based on
Quality Management
  • PPI is based on a Quality Management approach.
    PPIs may be
  • Inputs (Key Activities)
  • Measures of actions or initiatives undertaken to
    improve OHS
  • Provide indicators of commitment and effort, but
    not of effectiveness
  • Can provide useful information on participation,
    leadership and communication
  • Processes (Monitoring Key Risks)
  • Identify key risks
  • Identify key contributors to outcomes of concern
  • Develop measures to monitor behaviors and
  • Outputs (Milestones)
  • Achievement of objectives
  • Progress toward achievement of higher-level OHS
    goals and targets

Positive Performance Indicators Part of a
Management System
  • PPIs are only useful if an organization has a
    systematic approach to management of OHS
  • Many variations exist, but all have the following
  • Commitment and policy
  • Planning
  • Implementation
  • Measurement and evaluation
  • Review and improvement

Positive Performance Indicators Categories of
  • Commitment and policy
  • Measures demonstrated commitment to improve OHS
  • Planning
  • Measures what procedures are in place to
    eliminate workplace injury and disease
  • Implementation
  • Measures the capability and support mechanisms
    that are necessary to achieve OHS objectives and
  • Measurement and Evaluation
  • Measures the extent to which workplace health and
    safety is monitored and evaluated so that issues
    can be identified and corrective action taken
  • Review and improvement
  • Measures the effectiveness of the OHS management
    system, and its continuing suitability

Examples of PPIs Commitment and Policy
Examples of PPIs Implementation
Positive PerformanceIndicators Observations
  • PPIs are not limited to Occupational Health and
    Safety completely applicable to management
    across ESH
  • PPIs focus on identifying key elements of the
    ESH management system, and measuring how well
    these key elements are functioning
  • PPIs are non-prescriptive and open-ended
    approach provides great flexibility at the
    facility and site level
  • Most PPIs are qualitative, based on assessments,
    record reviews, surveys, or audits
  • The Guidance document assumes a fairly basic
    management system. DOE sites generally have more
    sophisticated ESH management systems in place,
    with the opportunity to develop more
    sophisticated measures.

General Electric Health and Safety Framework
  • At the April ISM meeting in Albuquerque, Kurt
    Krueger described GEs corporate-wide Health and
    Safety program
  • One standard, one program, one set of metrics
    for every GE facility around the world No
  • GEs Health and Safety Framework includes a mix
    of 21
  • Management system elements For example
  • Training
  • Accident investigation and follow-up
  • Subject matter areas For example
  • Industrial hygiene
  • Chemical management
  • Motor vehicle safety

General Electric (continued)
  • Detailed questions
  • GE-specific guidance expectations

General Electric (continued)
  • Scorecard ratings are based on
  • Plant self-assessment (540 questions, twice a
  • Periodic corporate audits
  • Tied to monthly site operational metrics and
    supervisor scorecards,
  • tailored to site operations
  • designed to drive supervisor behaviors that will
    find and fix HS issues before an accident finds
  • GE has a parallel system for assessing
    environmental performance
  • GE also continues to track trailing metrics
    For example
  • Recordable injury and illness rates
  • Lost time injury and illness rates
  • Permit exceedances
  • Notices of noncompliance

General Electric Observations
  • Integrated into GEs broader management system
  • Reflects a balance between frequent site
    self-assessment and reporting, and periodic
    headquarters audits
  • GE chose to implement a uniform corporate system.
    While this might not be DOEs choice, there is
    still a lot to learn from the overall framework,
    and from the measures GEs chose to include

ProSmart Center for Chemical Process Safety
  • In 1993, CCPS began a project to measure the
    effectiveness of process safety management
  • Their objective wasto provide management with
    the tools for assessing the health of process
    safety management systems on a real-time basis
  • Based around 12 management system elements
  • Extensive set of computer-based questions, using
    qualitative and quantitative measures
  • Rolls up responses into a single score or index

CCPS ProSmart Based on 12 Management System
  • CCPS identified 12 key management system
    elements, such as
  • Accountability
  • Process knowledge and documentation
  • Capital project hazard review
  • Training and performance
  • Management of change
  • Incident investigation
  • Audits and corrective action
  • ProSmart generates a score, which is continually

CCPS ProSmart Based on 12 Management System
CCPS ProSmart Drills Down to Detailed
Evaluation Questions
ProSmart Observations
  • Derived from a quality management approach
  • Focus is on specific management system elements
  • Established for chemical process safety, but in
    principle could encompass environment,safety and
  • Like TQEM and GE, ProSmart provides a set of
    detailed assessment questions, for qualitative
    assessment against established criteria

Key Aspects of Next Level Performance Measures
  • Next Level performance measures identify and
    measure how well key elements of the ESH
    management system are functioning
  • Measures tend to involve
  • Precursor analysis and trending
  • Observations, assessments and audits
  • Measures tend to be qualitative rather than just
    counting events
  • (This doesnt mean they cant be rigorous
    and replicable)

DOE Has Many Pieces in Place
  • Analysis and trending
  • Data collection
  • Trending and analysis
  • Root cause analysis
  • Observations, assessments, audits
  • Contractor self-assessment processes
  • DOE oversight activities
  • Annual ISM assessments should systematicallyaddre
    ss management system elements
  • Inspections by the Office of ESH Inspections
    (HS-64, formerly SP-44) address ESH management
    system functioning at a high level (ISM core
  • Individual IG audits address some management
    system elements

Audits and Inspections
ESH Inspections
  • (colors added)
  • Define Scope of Work
  • Analyze the Hazards
  • Identify and Implement Controls
  • Perform Work Within Controls
  • Feedback and Continuous Improvement
  • (contractor) (DOE)
  • Essential System Functionality

ESH Evaluations of 17 DOE Sites 2002-2004
  • There is a common thread of best practice, across
    many different organizations to move beyond
    outcome (lagging) indicators, we need to start
    systematically measuring how well key elements
    of our management system are performing.
  • This is not as easy as counting occurrences
    (injuries, enforcement actions) it must involve
    walk-throughs, audits, surveys, and other varied
  • This next level of leading performance
    measurement complements, and does not replace,
    outcome (lagging) measures.
  • Approaches developed for process safety, worker
    safety, environment, can be applied across ESH.
    We need to be eclectic and ecumenical in our
  • Nobody has a turnkey system thats right for DOE.
    We need to identify and build on the best of
    these various approaches.
  • Nobody says it will be easy.

Next Steps
  • Identify key ESH management system elements
  • We currently categorized some information (e.g.,
    ORPS reports Independent Oversight Assessments)
    around ISMs five broad core functions
  • Within these core functions, we can identify more
    specific management system elements, including
  • Identification of requirements - Corrective
    action management
  • Hazard identification - Lessons
    Learned sharing
  • Communication -
    Performance Measurement
  • Training -
    Worker involvement
  • Documentation - Audits
    and assessments
  • Periodic top management review

Next Steps (continued)
  • Identify the various things we are already doing
    to assess how well these elements are functioning
  • Identify and fill gaps as appropriate
  • Organize and present the information in a way
    which provides current feedback to management on
    how effectively key management system elements
    are functioning at each site and each facility
  • In Sum Use the health of these key management
    system elements as the conceptual and organizing
    framework to strengthen our ability to know how
    well we are managing risks

In Closing
  • Far better an approximate answer to the right
    question which may be difficult to frame,
  • than an exact answer to the wrong
    questionwhich is always easy to ask.
  • statistician John
    W. Tukey

  • Grace Wever, 1996, Strategic Environmental
    Management Using TQEM and ISO 14000 for
    Competitive Advantage, John Wiley and Sons, NY
  • European Process Safety Centre (Jacques van
    Steen, ed.), 1996, Safety Performance
    Measurement, Institution of Chemical Engineers,
    Rugby (UK) (distr by Gulf publishing Co., PO Box
    2608, Houston, TX 77252-2608)
  • Health and Safety Executive (United Kingdom),
    2001, A Guide to Measuring Health Safety
    Performance http//
  • Commonwealth of Australia (Department of
    Employment and Workplace Relations), 2005,
    Guidance on the Use of Positive Performance
    Indicators to Improve Workplace Health and
    Safety http//
  • Kurt Krueger, Creating a Culture of Safety
    Excellence The Journey and the Prize, talk at
    DOE ISM Champions Workshop, Albuquerque NM, April
    2006 http//
  • George Eckes, The Six Sigma Revolution How GE
    and Others Turned Process into Profits, New
    York John Wiley and Sons, 2000.
  • Center for Chemical Process Safety, ProSmart
    software http//

Contact Information
  • Steven R. Woodbury
  • DOE Office of Nuclear Safety and Environmental
  • 202-586-4371