2012 EXAMINER TRAINING Final Workshop November 15, 2012 - PowerPoint PPT Presentation

Loading...

PPT – 2012 EXAMINER TRAINING Final Workshop November 15, 2012 PowerPoint presentation | free to download - id: 6e546f-ODYxN



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

2012 EXAMINER TRAINING Final Workshop November 15, 2012

Description:

2012 EXAMINER TRAINING Final Workshop November 15, 2012 Presented by The Granite State Quality Council and The Northern New England Alliance for Excellence – PowerPoint PPT presentation

Number of Views:60
Avg rating:3.0/5.0
Slides: 96
Provided by: JanetW160
Learn more at: http://gsqc.com
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: 2012 EXAMINER TRAINING Final Workshop November 15, 2012


1
2012EXAMINER TRAINING Final WorkshopNovember
15, 2012
Presented by The Granite State Quality
Council and The Northern New England Alliance for
Excellence
2
Goal of this Training
White River Junction VA Medical Center
  • Provide each participant with a method and the
    tools to develop Baldrige-based feedback for
    Award Applicants in New Hampshire, Vermont,
    Maine, and other State Baldrige-based Programs.
  • Provide each of you with a deeper understanding
    of Baldrige in support of your organization
    career

3
Agenda Examiner Training Final Workshop
  • Introductions
  • Putting faces with voices.
  • Quick Review
  • Independent Review Process
  • The Consensus Process
  • Creating Consensus Comments and Rationale
  • Feedback (Applicant) Ready Comments
  • Practice, practice, practice
  • Examiner Roles, Ethics Conflict of Interest
  • The Organizational Profile Assessment
  • After Consensus
  • Key Themes
  • Introduction to Site Visit

4
The Baldrige Application Evaluation Process -
Stages
Goal Value Added Feedback To the Applicant
Site Visit Comments
Stage 3
Consensus Comments
Stage 2
Independent Review
Stage 1
5
Review The Application Evaluation Process
Independent Review
Evaluate Each Criteria Item
Check for Conflict of Interest
Assemble Materials and Read Application
Key Themes Observations
Draft Initial Key Factors
Complete Checklist and Assemble Scorebook
6
Independent Review Worksheet
A-D-L-I Or Le-T-C-I
Supporting Evidence Examples
Main Idea or Nugget
Strengths / OFIs
Relationship with Key Factors (Importance)
Relationship with Criteria (traceability)
7
Independent Review Worksheets Comments Score
8
Feedback Ready Comments
  • Turns Strength/OFI Evidence into clear and
    actionable feedback for the applicant
  • Polite, Professional, Positive
  • Not Judgmental or Prescriptive
  • Concise one message per comment
  • Indicates
  • Criteria-based strength or potential OFI
  • Potential significance of Strength/OFI

9
Elements of a feedback ready Comment
usingN-E-R-DComments should include the
following
1. Nugget Key point based on Criteria
2. Examples for clarity context
3. Relevance to this applicant
Done! No need to add additional information
what wont add value to the applicant
10
Creating a Feedback Ready Comment 5.1 Strength
2,4,5,6 Systematic integrated approach to employee health safety - Exec Director of each facility risk management/safety officer, serve on system wide Safety Committee - Review all facility/system health safety, security policies procedures - Meet monthly to review key measures APEX Goals - Integrated with daily rounds with Collaborative Care Teams - Use of Data to drive improvements b(1)
b(1) A variety of methods and associated goals
(Figure 5.1-2) help ensure workplace health,
safety, and security. These methods include a
system-wide Safety Committee that meets monthly
to review progress toward goals, unexpected
events, and best practices. Daily rounding by
Collaborative Care Teams helps identify potential
safety hazards.
11
b(1) A variety of methods and associated goals
(Figure 5.1-2) help ensure workplace health,
safety, and security. These methods include a
system-wide Safety Committee that meets monthly
to review progress toward goals, unexpected
events, and best practices. Daily rounding by
Collaborative Care Teams helps identify potential
safety hazards.
Nugget Key point based on Criteria
Polite?
Examples for clarity context
Professional?
Indicates Relevance to this applicant
Positive?
12
Creating a Feedback Ready Comment 5.1 OFI
1,2,3, 4, 6 Approaches to management of workforce capability/capacity to maintain a safe, secure supportive work climate do not address all segments of workforce - Little discussion of physicians, volunteers, students -Unclear if security is an issue at some of the locations (e.g. SLFs) Approach for capacity planning is somewhat unclear they just post the positions and move people around? a,b
a, b It is unclear how the applicant manages
volunteers, physicians, and students to fully
support its work. For example, it is unclear how
volunteers are trained and managed and how
approaches are deployed to precepted students and
credentialed physicians. Approaches for managing
these workforce groups may help the applicant
deliver high quality care and services and
maintain a safe, secure environment.
13
a, b It is unclear how the applicant manages
volunteers, physicians, and students to fully
support its work. For example, it is unclear how
volunteers are trained and managed and how
approaches are deployed to precepted students and
credentialed physicians. Approaches for managing
these workforce groups may help the applicant
deliver high quality care and services and
maintain a safe, secure environment.
Nugget Key point based on Criteria
Polite?
Examples for clarity context
Professional?
Indicates Relevance to this applicant
Positive?
Avoid starting OFI with the While or Although
phrase
14
Agenda Review
  • Quick Review
  • Independent Review Process
  • The Consensus Process
  • Creating Consensus Comments and Rationale
  • Feedback (Applicant) Ready Comments
  • Practice, practice, practice
  • The Organizational Profile Assessment
  • Examiner Roles, Ethics Conflict of Interest
  • After Consensus
  • Key Themes
  • Introduction to Site Visit

15
The Consensus Process
  • Incorporating Team Feedback into Comments

16
(No Transcript)
17
The Consensus Process
  • Creating Consensus Comments Rationale

18
What is Consensus?
  • The Gospel according to Baldrige states
  • Consensus is a form of agreement reached by a
    group of people resulting from discussion and
    consideration of everyones ideas, ultimately
    bringing the group to agreement on a particular
    decision all can live with.

19
Why have Consensus?
  • Resolve diversity in individual evaluation
  • Key Factors
  • Strengths and OFIs
  • Score
  • Key Themes
  • Pooling of information adding value to overall
    knowledge
  • All members exposed to pool of knowledge
  • Achieving basic agreement on Strengths OFIs
  • Weigh overall findings to assign score

20
Why have Consensus? (cont.)
  • Consensus comments used
  • As basis for site visits
  • For feedback report if no site visit
  • Used by Judges in some cases to determine site
    visit or site visit purpose

21
What Changes in Consensus?
  • Item Leads
  • 3 Criteria Items lead by each Examiner
  • Incorporate the input of other team members
  • 4-6 Key Factors
  • Most important ideas/evidence
  • Output
  • Feedback Ready Comments Rationale
  • Score based on Consensus Comments

22
Creating Consensus Feedback
Examiner A Independent Review
Consensus Comments Rationale
Examiner B Independent Review
Examiner C Item Lead
Examiner C Independent Review
Examiner D Independent Review
23
The Consensus Process
Consensus Key Factors
Examiners send Independent Review to Item
Lead
Item Lead Back-up Identified
Item Lead creates V1 draft
Back-up Review Feedback
Item Lead creates V2 Draft
(V3 Draft) Consensus Meeting
Consensus Output Consolidated
Team Review Feedback
Virtual Consensus
24
SN
25
Virtual Consensus ProcessV1 Draft
  • Re-familiarize yourself with Criteria
    Requirements for assigned item
  • Develop consensus 4-6 Key Factors
  • Re-familiarize yourself with Applicant Response
  • Aggregate all team member ideas/evidence
  • Determine most important for feedback
  • Draft 6 Comments for each Item
  • Include Rationale for each comment
  • Propose Score based on Comments

26
Virtual Consensus ProcessV2 V3
  • Receive input from teammates
  • V2 Backup (and Team Leader)
  • V3 Full Team
  • Review Criteria Requirements
  • Verify/update 4-6 Key Factors
  • Review Applicant Response
  • Make Changes, Refine for Clarity
  • Read from the Applicant perspective
  • Update Rationale
  • Verify/update Score

Would you be comfortable giving that feedback to
the applicant?
27
Consensus Meeting
  • Entire Examiner Team participates
  • Discuss all items
  • Focus on topics Virtual Consensus did not close
    out
  • Key Factors
  • Key Themes
  • Strengths OFIs by Item
  • Scores
  • Team Leader sets agenda
  • Item Owner leads discussion

28
Agenda Review
  • Quick Review
  • Independent Review Process
  • The Consensus Process
  • Creating Consensus Comments and Rationale
  • Feedback (Applicant) Ready Comments
  • Practice, practice, practice
  • The Organizational Profile Assessment
  • Examiner Roles, Ethics Conflict of Interest
  • After Consensus
  • Key Themes
  • Introduction to Site Visit

29
(No Transcript)
30
Same content as in Independent Review Item lead
reviews input of all Independent Reviews and
decides on Consensus Key Factors
31
Significant ? Double
Criteria Reference
Feedback Ready Comment
  • Notes
  • Unused Strengths/OFIs Rationale
  • Enough detail to make a Strength/OFI
  • Rationale Why is this a Strength/OFI?
  • Examiners having same/similar Strength/OFI
  • Examiners having conflicting Strength/OFI
  • Indication of ADLI/LeTCI
  • Notes to team members
  • Not provided to Applicant

Capture all ideas from all Independent Reviews
on Consensus worksheets
32
Score Range Which of the 6 Scoring Bands best
reflects feedback?
Rationale Use ADLI or LeTCI to explain Range
selected
Score Value Score within the Band 5
intervals
33
Round 1 Practice Consensus(Items 1.1, 1.2, 7.4)
  • As a TEAM
  • Review Criteria Requirements
  • Verify/update 4-6 Key Factors
  • Review Applicant Response
  • Identify 6 Total Strengths OFIs
  • Use Blank Item Worksheet
  • Develop Rationale for each
  • Verify/update Score
  • Summarize your findings and share with group

34
Results vs. Process Items
  • Process Items Strengths and gaps in How
  • Approach, Deployment, Learning, Integration
  • Results charts, graphs, tables, and text on
    outcomes/outputs of processes
  • Importance to the applicant, alignment with key
    factors
  • Beneficial vs. adverse trends, inconclusive
    pattern
  • Performance relative to comparisons
  • Breadth of results address key customer, market,
    and process requirements address appropriate
    segmentation for the item.
  • Comment on what results tell you that can be of
    benefit to applicants improvement
  • Levels, Trends, Comparisons, Segmentation,
    Importance
  • What is missing??

35
Practice One - ProcessInstructions
  • Using a round robin approach, capture all the
    strengths and opportunities that you noted or
    recall from your reading.
  • Test each observation for its relationship to
    major ideas in the Criteria, the key factors you
    identified, the value added content, and its
    potential impact on the applicants desired
    results.
  • Draft around six strength and OFI nuggets.
  • Prioritize your strengths according to their
    importance to the applicant repeat with your
    opportunities.
  • Remember to take a holistic view of the
    applicants responses.

36
Agenda Review
  • Quick Review
  • Independent Review Process
  • The Consensus Process
  • Creating Consensus Comments and Rationale
  • Feedback (Applicant) Ready Comments
  • Practice, practice, practice
  • The Organizational Profile Assessment
  • Examiner Roles, Ethics Conflict of Interest
  • After Consensus
  • Key Themes
  • Introduction to Site Visit

37
Writing Value-Added Comments
Beyond Criteria
ENVIRONMENT
ORGANIZATION
Parroting
Prescriptive
CRITERIA
Value-Added Comments
38
Comment Fundamentals
  • Comment Purpose Communication
  • Clearly identify most important Strengths and
    Opportunities for Improvement (OFIs)
  • Provide feedback in a non-prescriptive,
    non-judgmental manner
  • Comments are based on what the Criteria is asking
    and what is important to the Applicant (Hint Key
    Factors)
  • Comment Components clearly concisely state
  • What is Strength or Opportunity For Improvement?
  • Why is Strength or OFI important to the Applicant?

39
NEW Comment Components
  • Applicant Requirements
  • Traceability back to Criteria
  • Criteria references now listed in feedback
  • A concise opening sentence
  • Expresses a single thoughtthe nugget (or
    essence, or main point)
  • Some examples
  • The relevance or importance of the nugget to the
    application

No prescribed order
Can Applicant use each part of my comment?
40
Writing a Comment
Application Organizational Profile Item Responses
Criteria Requirements
ADLI LeTCI
your best judgment
Call to Action
Comment
41
Strengths and OFIs defined
  • A strength is a process, approach, or result that
    supports the applicants achievement of its
    desired results.
  • An OFI is a process, approach, or result (or lack
    of one of these) that creates a vulnerability in
    achieving the applicants desired results.

Give the applicant the benefit of the doubt when
appropriate (Your best judgment)
42
Comment ComponentsThe Nugget
Application Organizational Profile Item Responses
Criteria Requirements
your best judgment
The characteristics of a well-written comment
Precise
Concise
Shows insights
43
Sample NuggetsItem 5.2
  • a(3) The alignment of individual performance
    goals with strategic goals contributes to the
    achievement of those goals.
  • b(1) The applicants approach to assessing
    workforce engagement is aligned with performance
    measurement and strategic planning.
  • a, b, c It is unclear how the applicant applies
    workforce performance management, engagement, and
    professional development processes to volunteers,
    students, and credentialed physicians/nurse
    practitioners.
  • c(2) The applicants evaluation of the learning
    and development system, which appears to be
    limited to a review of posteducation performance,
    may not allow the organization to fully leverage
    its investment in learning.

44
Comment ComponentsExamples
Application Organizational Profile Item Responses
Criteria Requirements
your best judgment
45
Sample ExamplesStrengths Item 5.2
  • a(3) The alignment of individual performance
    goals with strategic goals contributes to the
    achievement of those goals. The applicant uses
    resident outcome data to drive high workforce
    performance by linking outcomes to the
    performance evaluation and compensation
    structure, which is unique in the industry.
  • b(1) The applicants approach to assessing
    workforce engagement is aligned with performance
    measurement and strategic planning. APEX
    scorecard results that are considered indicators
    of workforce engagement are reviewed regularly,
    and the organization addresses key workforce
    engagement findings within strategic planning.

46
Sample ExamplesOFIs Item 2.1
  • a, b Tillingate Livings strategic planning and
    objectives do not appear to address all strategic
    challenges or balance all stakeholder needs. For
    example, it is not clear how the strategic
    objectives (Figure 2.1-3) address the challenge
    to integrate existing practices with ACOs or how
    physician partners and suppliers are
    systematically included in the SPP. Such gaps may
    prevent Tillingate Living from being a top choice
    for care.
  • a(1) It is not clear how Tillingate Living
    systematically identifies and refines its core
    competencies. For example, how the core
    competency of designing and delivering
    rehabilitation services was systematically
    determined during step 3 of the SPP is unclear.
    Without a defined approach in this area,
    Tillingate Living may miss the opportunity to
    leverage these strategically important
    capabilities for future success.

47
  • 5.2 Workforce Engagement Tillingate OFIs
  • a, b, c It is unclear how the applicant applies
    workforce performance management, engagement, and
    professional development processes to volunteers,
    students, and credentialed physicians/nurse
    practitioners. Without engaging the entire
    workforce, the applicant may miss opportunities
    to maintain its excellent reputation, especially
    related to the key family requirement of an
    attentive staff.
  • c(2) The applicants evaluation of the learning
    and development system, which appears to be
    limited to a review of posteducation performance,
    may not allow the organization to fully leverage
    its investment in learning. A systematic approach
    in this area may help the applicant maintain its
    core competency of developing clinical and
    service competencies for a caring and exceptional
    staff.

OFI Examples help to clarify exactly what the gap
is
48
Comment ComponentsRelevance/Importance
Application Organizational Profile Item Responses
Criteria Requirements
your best judgment
Comment Relevance demonstrates how well we know
the applicant
49
Sample CommentsItem 2.1
  • a(1) The five-month, 14-step strategic planning
    process (SPP) enables Tillingate Living to learn
    about residents needs and expectations and
    determine its strategic challenges and
    advantages. The SPP considers resident needs and
    minimizes blind spots through an environmental
    scan and SWOT analysis (Figure 2.1-2). The short-
    and near-term horizons allow responses to senior
    living trends and corporate/facility performance.
    Resident and Family Councils now give input to
    the process.

Note ADLI is not called out but evident
Relevance/Importance
50
Sample CommentsItem 2.1
  • a, b Tillingate Livings strategic planning and
    objectives do not appear to address all strategic
    challenges or balance all stakeholder needs. For
    example, it is not clear how the strategic
    objectives (Figure 2.1-3) address the challenge
    to integrate existing practices with ACOs or how
    physician partners and suppliers are
    systematically included in the SPP. Such gaps may
    prevent Tillingate Living from being a top choice
    for care.

How can we make this relevance more motivating to
act on??
51
Comments Other Considerations
your best judgment
Comment Guidelines
Call to Action
Comment
Can Applicant use each part of my comment?
52
Sample CommentsItem 3.1
Note no while or although - consider Can
Applicant use each part of my comment? (is
while or although value added?)
53
Remember Results Items
  • Charts, graphs, tables, and text describing
    outcomes/outputs of process
  • Importance to the applicant, alignment with key
    factors
  • Beneficial vs. adverse trends, inconclusive
    pattern
  • Performance relative to comparisons
  • Breadth of results address key customer, market,
    and process requirements address appropriate
    segmentation for the item.
  • Comment on what results tell you that can be of
    benefit to applicants improvement
  • Levels, Trends, Comparisons, Segmentation,
    Importance
  • What is missing??

54
Item 7.3 Workforce-Focused Outcomes - Strengths
  • a(3) Top-decile performance on employee
    satisfaction and engagement survey measures, as
    well as low vacancy and turnover rates (Figure
    7.3-7), help the organization sustain its
    strategic advantage of high employee retention.
    Overall satisfaction has been at or better than
    the top-decile level since 2008 (Figure 7.3-4).
    Engagement results for "I am proud to work
    here" and "I would recommend applicant to
    family members survey items were at top-decile
    levels in 2010 and 2011 (Figure 7.3-6).
  • a(2) Most of Tillingate Living's reported
    workforce climate results have improved over the
    past five or six years, which helps the
    organization support an engaged workforce.
    Employee back injury results (Figure 7.3-2),
    which approached top-decile levels in 2010 and
    2011, reflect reductions in back injuries and
    workers compensation costs over the past four
    years.

Bolded Means - -
What is the difference from a Process Comment??
55
Item 7.3 Workforce-Focused Outcomes - OFIs
  • a(3) Tillingate Living does not present workforce
    engagement results by service offering, facility,
    and state, and engagement results for volunteers,
    credentialed physicians, and students are
    missing. Without results for all segments of the
    workforce, the organization may be unable to
    improve engagement and achieve its vision to be
    among the top 10 of facilities.
  • a(2) Tillingate Living does not report most
    workforce health, safety, and security results.
    For example, results are missing for tuberculosis
    screenings, injuries unrelated to resident care,
    and injuries from agitated residents (Figure
    5.1-2). Without such results, Tillingate Living
    may limit its ability to ensure a safe and secure
    environment.
  • a(1) Tillingate Living has limited results for
    workforce capability and capacity. For example,
    results on competencies or skill levels are not
    reported. Also, results for capacity are limited
    to ratios of nursing care time to health care
    outcomes. Without measures in this area,
    Tillingate Living may be challenged in achieving
    its strategic objectives and goals.

What is the difference from a Process Comment??
56
Dos and Donts
Comment Guidelines
  • Do
  • Keep the customer in mind.
  • Can the applicant act based on my comment?
  • Ensure relevancy by drawing linkages across items
    and to the applicants Organizational Profile
  • Ensure that the comment does not contradict other
    comments.
  • Address your comments to the basic, overall, or
    multiple requirements that are most important to
    the applicant.
  • Use the evaluation factors (A-D-L-I or Le-T-C-I)
    to articulate the areas of strength or
    opportunities for improvement.
  • Write a unified, coherent, well-developed
    comment. Express the comments single, main point
    in a topic sentence, followed by concisely
    written evidence

.
57
Dos and Donts
Comment Guidelines
  • Do Not
  • Construct an exhaustive list of every method
    described by the applicant that is related to
    your main point.
  • Go beyond the requirements of the Criteria or
    assert your personal opinions.
  • Be prescriptive by using must, should, and
    would.
  • Be judgmental by using terms such as good,
    bad, or inadequate.
  • Comment on the applicants style of writing or
    data presentation.
  • Parrot the Applicant/Application

58
Comment Style Guidelines
  • Do not
  • Use jargon or acronyms unless they are used by
    the applicant.
  • Do
  • Use a polite, professional, and positive tone.
  • Use active voice and present tense
  • completes rather than is completed
  • Use the applicant or the organization
  • Until site visit tell what is missing if
    something is not clear. After site visit
    clarify all not clear statements.

59
Comment Litmus Test
  • Traceable?
  • Single nugget?
  • Examples?
  • Relevance/Importance?
  • Follows Content and Style Guidelines?
  • Call to Action?

Can Applicant use each part of my comment?
Ask yourself, Can the applicant use each part of
my comment? If the answer is no, consider some
deletions or editing.
60
Round 2 Practice Consensus Feedback Ready
Comments (Items 1.1, 1.2, 7.4)
  • Remaining in your Team swap items
  • Agree on most important Strength and OFI
  • Develop a feedback ready comment for each
  • Make any changes to Rationale
  • Identify Item Ref and if or - -
  • Time permitting repeat steps 1 2 with next
    Strength OFI
  • When finished, review comments of other Teams
  • Summarize your findings and share with everyone

61
Agenda Review
  • Quick Review
  • Independent Review Process
  • Consensus
  • Creating Consensus Comments and Rationale
  • Feedback (Applicant) Ready Comments
  • Practice, practice, practice
  • Examiner Roles, Ethics Conflict of Interest
  • The Organizational Profile Assessment
  • After Consensus
  • Key Themes
  • Introduction to Site Visit

62
The Role of a Baldrige Examiner Ethics Code
of Conduct
  • Four Guiding Principles
  • Integrity
  • No conflicts of interest or anything that could
    skew your evaluation
  • Professional Conduct
  • Truth, accuracy, fairness, respect,
    responsibility
  • Maintain confidentiality
  • Anything that could disclose applicant identity
    or information
  • Respect for Intellectual Property

63
Examiner Roles Lead and Back-up
  • Lead Examiner Team Leader
  • Back-up Team Leader
  • Category/Item Leader
  • Team Member Everyone

64
Examiner Roles Lead and Back-up
  • Lead Examiner Team Leader
  • Work with the State/Local Program
  • Develop timeline, site visit plan manage site
    visit
  • Finalize and Deliver feedback to applicant
  • Guides examiner team
  • Establishes category leads and due dates
  • Leads consensus process, maintains focus
  • Liaison with Applicant and Judges
  • Back-up Team Leader
  • Assists with Lead Examiner duties
  • OTJ training to become future lead examiner

65
Examiner Roles Category/Item Lead
  • Typically assigned a Process Category and
    corresponding Results Item
  • Cat 1/Item 7.4 Cat 2/Item 7.5/7.1 Cat 3/Item
    7.2 Cat 4/Item 7.1 Cat 5/Item 7.3 Cat
    6/Item 7.1
  • Team expert for assigned Items
  • Collects team member IR for Virtual Consensus
  • Drafts and finalizes comments
  • Drafts leads investigation of Site Visit Items
  • Acts as back-up for additional category
  • Typical pairings Cats 1 2 3 5 4 6

66
Examiner Roles Team member
  • Review all assigned material
  • Verify no conflict of interest availability
  • Complete individual review
  • Serves as Category Lead Backup
  • Provides input and feedback
  • Site visit plan
  • Development and closure of all SVIs
  • Conducts site visit
  • Everyone is a team member

67
Agenda Review
  • Quick Review
  • Independent Review Process
  • Consensus
  • Creating Consensus Comments and Rationale
  • Feedback (Applicant) Ready Comments
  • Practice, practice, practice
  • Examiner Roles, Ethics Conflict of Interest
  • The Organizational Profile Assessment
  • After Consensus
  • Key Themes
  • Introduction to Site Visit

68
The Profile Level Recognition Process
Application Submission Baldrige Organizational
Profile plus 2-page Action Plan
  • Phase 1 Analysis 2-3 Weeks
  • Examiners Independently Develop Clarifying
    Questions
  • Consensus meeting generates request to Applicant

Phase 1
Applicant provides requested information to
support Examiner evaluation (2 Weeks)
Phase 2 Analysis 2-3 Weeks Examiners
independently review Application, including
responses to Phase 1 questions, to identify
Strengths and potential Opportunities for
Improvement
  • Final Consensus
  • Team meets to develop final feedback
  • Strengths and OFIs
  • Key Themes

Phase 2
Feedback Review Meeting Applicant and GSQC/NNEAE
(optional)
69
Organizational Profile Phase 1 Worksheets
70
Organizational Profile Phase 2 Worksheets
71
(No Transcript)
72
Organizational Profile Assessment Examiner Roles
  • 2-4 Examiners
  • Team Leader, Team Members
  • No Item Leads
  • 1 Consensus Call for Each Phase
  • Generally no Site Visit
  • May be consultative discussion of feedback

73
Agenda Review
  • Quick Review
  • Independent Review Process
  • Consensus
  • Creating Consensus Comments and Rationale
  • Feedback (Applicant) Ready Comments
  • Practice, practice, practice
  • Examiner Roles, Ethics Conflict of Interest
  • The Organizational Profile Assessment
  • After Consensus
  • Key Themes
  • Introduction to Site Visit

74
The 50-Page Application Evaluation Process
Evaluate Each Criteria Item
Check for Conflict of Interest
Assemble Materials and Read Application
Key Themes
Draft Key Factors
Complete Checklist and Assemble Scorebook
75
The Process of Developing Feedback
Themes
Comments
Comments
Comments
Comments
Comments
76
Key Themes
  • Serve as an executive summary addressed to the
    applicants senior leaders
  • Summarize the most vital issues for the
    organization as a whole, including what the
    organization must do to remain or become
    competitive and to ensure long-term
    sustainability.
  • Strategic in nature may address how well the
    organization is managing a major change or
    improvement competitiveness or social issues or
    significant customer, market, product, or
    technological opportunities and challenges.
  • Must be traceable to Item comments

77
Key Themes
  • Developed by Team Leader or other Designated
    Examiner
  • Input and review by all team members
  • First Draft during Consensus Process
  • Finalized at/after Consensus Meeting
  • Revised and finalized during Site Visit
  • Meet Comment Guidelines

78
Key Themes Worksheet
  • Organize into four sections to address
  • most important strengths or outstanding practices
  • most significant opportunities, concerns, or
    vulnerabilities
  • Most significant strengths found in Results Items
  • most significant strengths, opportunities,
    vulnerabilities, and/or gaps found in its
    response to Results Items
  • 2-3 pages in length
  • Complete sentences and that meet the Comment
    Guidelines
  • Delineate comments with bullets

79
Sample Key Themes Process Strength
80
Sample Key Themes Process Opportunities,
concerns, vulnerabilities
81
Sample Key Themes Results Strengths
82
Sample Key Themes Results
Opportunities, vulnerabilities, and/or gaps
83
Agenda Review
  • Quick Review
  • Independent Review Process
  • Consensus
  • Creating Consensus Comments and Rationale
  • Feedback (Applicant) Ready Comments
  • Practice, practice, practice
  • Examiner Roles, Ethics Conflict of Interest
  • The Organizational Profile Assessment
  • After Consensus
  • Key Themes
  • Introduction to Site Visit

84
ReviewThe Baldrige Application Evaluation
Process
Goal Value Added Feedback To the Applicant
Site Visit Comments
Consensus Comments
Stage 3
Stage 2
Independent Observations
Stage 1
85
Site Visit
86
Planning the Site VisitPreparation, Preparation,
Preparation!
  • Your Input
  • Consensus Feedback
  • What are you looking for?
  • Accuracy (verify strengths)
  • Clarification of all areas for improvement
  • Breadth and depth of deployment
  • Integration
  • Maturity of systems
  • Linkages to key themes

87
Draft Site Visit Issues (SVIs)
  • What are Site Visit Issues?
  • Issues that need to be verified or clarified on
    site.
  • Related to a key factor, making it relevant and
    important to the applicant
  • Linked to comments making each relevant and
    important to the applicant
  • Focus on a Process
  • Normally 2-4 SVIs per Item

88
Draft Site Visit Issues (SVIs)
  • Verify Strengths
  • The applicants approach, the extent of
    deployment of the approach, and the results
    presented
  • Clarify OFIs
  • Things that were unclear or not addressed in the
    application
  • Central to the Item requirements
  • Relevant and important to the applicants
    organization
  • All OFIs, significant strengths, and all strength
    comments linked to a key theme must be verified
    or clarified (things that have an impact on
    scoring)

89
Site Visit to Final Feedback
Input Consensus Comments
Output Verified Clarified Comments Feedback
Report
Develop Site Visit Issues
Verify Walk the Wall
Create SVI Worksheets for Process and Results
Update Comments Site Visit Comment Worksheets
Create Strategy for each SVI
Complete SVI Worksheets
  • Conduct Site Visit
  • Document Findings
  • Scheduled Interviews
  • Walking Around Questions
  • Requested Document and Results

Denotes Worksheet
90
Summary From Consensus to Final Comments
Key Theme
Consensus Comment
Consensus Comment
Final Comment
Key Theme
91
At Site Visit
  • Professionalism is an absolute they are
    watching your every step.
  • Try to understand their business
  • Conduct interviews
  • Explain the purpose of interview
  • Say Thank You!
  • Dont say Great or other potentially judgmental
    phrases
  • Review Documentation (on-site only)
  • Document observations in relation to issues
  • May record evidence of non-SVI issues, but dont
    go looking
  • Record Site Visit Issue Evidence and Conclusions

92
Agenda Review
  • Quick Review
  • Independent Review Process
  • In Depth
  • Consensus Observations
  • Observations to Comments
  • Practice, practice, practice
  • Examiner Roles, Ethics Conflict of Interest
  • The Organizational Profile Assessment
  • After Consensus
  • Key Themes
  • Introduction to Site Visit

93
Agenda Review
  • Quick Review
  • Independent Review Process
  • Consensus
  • Creating Consensus Comments and Rationale
  • Feedback (Applicant) Ready Comments
  • Practice, practice, practice
  • Examiner Roles, Ethics Conflict of Interest
  • The Organizational Profile Assessment
  • After Consensus
  • Key Themes
  • Introduction to Site Visit

94
Complete NuGrain Packet
  • http//www.nist.gov/baldrige/publications/tillinga
    te.cfm

Tillingate Living Case Study Packet 2012Case
StudyScorebook Examiners productFeedback
Report Delivered to Applicant
95
Final Questions Feedback?
On-Line Evaluation is coming Check your E-mail
About PowerShow.com