Title: Performance Improvement Projects Technical Assistance Health Maintenance Organizations
1Performance Improvement Projects Technical
Assistance Health Maintenance Organizations
Provider Service NetworksWednesday, March 28,
2007300 p.m. 500 p.m.
- Cheryl L. Neel, RN, MPH, CPHQ
- Manager, Performance Improvement Projects
- David Mabb, MS
- Sr. Director, Statistical Evaluation
2Presentation Outline
- PIP Overall Comments
- Aggregate MCO PIP Findings
- Aggregate HMO Specific Findings
- Technical Assistance with Group Activities
- Study Design
- Study Implementation
- Quality Outcomes Achieved
- Questions and Answers
3Key PIP Strategies
- Conduct outcome-oriented projects
- Achieve demonstrable improvement
- Sustain improvement
- Correct systemic problems
4Validity and Reliability of PIP Results
- Activity 3 of the CMS Validating Protocol
Evaluating overall validity and reliability of
PIP results - Met Confidence/High confidence in reported PIP
results - Partially Met Low confidence in reported PIP
results - Not Met Reported PIP results not credible
5Summary of PIP Validation Scores
6Proportion of PIPs Meeting the Requirements for
Each Activity
7Aggregate Valid Percent Met
I
II
III
IV
V
VI
VII
VIII
IX
X
8HMO Specific Findings
- 53 PIPs submitted
- Scores ranged from 5 to 98
- Average score was 65
- Assessed evaluation elements were scored as Met
64 of the time
9Summary of HMO Validation Scores
10Study Design
- Four Components
- Activity I. Selecting an Appropriate Study Topic
- Activity II. Presenting Clearly Defined,
Answerable Study Question(s) - Activity III. Documenting Clearly Defined Study
Indicator(s) - Activity IV. Stating a Correctly Identified
Study Population
11Activity I. Selecting an Appropriate Study Topic
HMO Overall Score
12Activity I. Selecting an Appropriate Study Topic
- Results
- 92 percent of the six evaluation elements were
Met - 8 percent were Partially Met or Not Met
- None of the evaluation elements were Not
Applicable or Not Assessed
13Activity I Review the Selected Study Topic
- HSAG Evaluation Elements
- Reflects high-volume or high-risk conditions (or
was selected by the State). - Is selected following collection and analysis of
data (or was selected by the State). - Addresses a broad spectrum of care and services
(or was selected by the State). - Includes all eligible populations that meet the
study criteria. - Does not exclude members with special health care
needs. - Has the potential to affect member health,
functional status, or satisfaction. - Bolded evaluation elements show areas for
improvement
14Activity II. Presenting Clearly Defined,
Answerable Study Question(s) - HMO Overall Score
15Activity II. Presenting Clearly Defined,
Answerable Study Question(s)
- Results
- 43 percent of the two evaluation elements were
Met - 57 percent were Partially Met or Not Met
- None of the evaluation elements were Not
Applicable or Not Assessed
16Activity II Review the Study Question(s)
- HSAG Evaluation Elements
- States the problem to be studied in simple terms.
- Is answerable.
- Bolded evaluation elements show areas for
improvement
17HMO Overall Score for Each Evaluation Element
Activity III
18Activity III. Documenting Clearly Defined Study
Indicator(s)
- Results
- 56 percent of the seven evaluation elements were
Met - 27 percent were Partially Met or Not Met
- 16 percent were Not Applicable or Not Assessed
19Activity III Review Selected Study Indicator(s)
- HSAG Evaluation Elements
- Is well defined, objective, and measurable.
- Are based on practice guidelines, with sources
identified. - Allows for the study question to be answered.
- Measures changes (outcomes) in health or
functional status, member satisfaction, or valid
process alternatives. - Have available data that can be collected on each
indicator. - Are nationally recognized measure such as HEDIS,
when appropriate. - Includes the basis on which each indicator was
adopted, if internally developed. - Bolded evaluation elements show areas for
improvement
20Activity IV. Stating a Correctly Identified Study
Population - HMO Overall Score
21Activity IV. Stating a Correctly Identified
Study Population
- Results
- 53 percent of the three evaluation elements were
Met - 43 percent were Partially Met or Not Met
- 3 percent of the evaluation elements were Not
Applicable or Not Assessed
22Activity IV Review the Identified Study
Population
- HSAG Evaluation Elements
- Is accurately and completely defined.
- Includes requirements for the length of a
members enrollment in the managed care plan. - Captures all members to whom the study question
applies. - Bolded evaluation elements show areas for
improvement
23Group Activity
24Study Implementation
- Three Components
- Activity V. Valid Sampling Techniques
- Activity VI. Accurate/Complete Data Collection
- Activity VII. Appropriate Improvement Strategies
25Activity V. Presenting a Valid Sampling Technique
- HMO Overall Score
26Activity V. Presenting a Valid Sampling Technique
- Results
- 12 out of the 53 PIP studies used sampling.
- 11 percent of the six evaluation elements were
Met. - 12 percent were Partially Met or Not Met.
- 77 percent of the evaluation elements were Not
Applicable or Not Assessed.
27Activity V Review Sampling Methods
- This section is only validated if sampling is
used. - HSAG Evaluation Elements
- Consider and specify the true or estimated
frequency of occurrence. (N7) - Identify the sample size. (N7)
- Specify the confidence level to be used. (N5)
- Specify the acceptable margin of error. (N5)
- Ensure a representative sample of the eligible
population. (N5) - Ensure that the sampling techniques are in
accordance with generally accepted principles of
research design and statistical analysis. (N5) -
28Populations or Samples?
All?
Some?
- Generally,
- Administrative data uses populations
- Hybrid (chart abstraction) method uses samples
identified through administrative data
29Activity VI. Specifying Accurate/Complete Data
Collection - HMO Overall Score
30Activity VI. Specifying Accurate/Complete Data
Collection
- Results
- 34 percent of the eleven evaluation elements were
Met - 24 percent were Partially Met or Not Met
- 42 percent of the evaluation elements were Not
Applicable or Not Assessed
31Activity VI Review Data Collection Procedures
- HSAG Evaluation Elements
- Clearly defined data elements to be collected.
- Clearly identified sources of data.
- A clearly defined and systematic process for
collecting data that includes how baseline and
remeasurement data will be collected. - A timeline for the collection of baseline and
remeasurement data. - Qualified staff and personnel to collect manual
data. - A manual data collection tool that ensures
consistent and accurate collection of data
according to indicator specifications. - Bolded evaluation elements show areas for
improvement
32Activity VI Review Data Collection Procedures
(cont.)
- HSAG Evaluation Elements
- A manual data collection tool that supports
interrater reliability. Clear and concise written
instructions for completing the manual data
collection tool. An overview of the study in the
written instructions. - Administrative data collection algorithms that
show steps in the production of indicators. - An estimated degree of automated data
completeness (important if using the
administrative method). - Bolded evaluation elements show areas for
improvement
33Where do we look for our sources of data?
34Baseline Data Sources
- Medical Records
- Administrative claims/encounter data
- Hybrid
- HEDIS
- Survey Data
- MCO program data
- Other
35Activity VII. Documenting the Appropriate
Improvement Strategies - HMO Overall Score
36Activity VII. Documenting the Appropriate
Improvement Strategies
- Results
- 28 percent of the four evaluation elements were
Met - 21 percent were Partially Met or Not Met
- 51 percent of the evaluation elements were Not
Applicable or Not Assessed
37Activity Seven Assess Improvement Strategies
- HSAG Evaluation Elements
- Related to causes/barriers identified through
data analysis and Quality Improvement (QI)
processes. - System changes that are likely to induce
permanent change. - Revised if original interventions are not
successful. - Standardized and monitored if interventions are
successful. -
- Bolded evaluation elements show areas for
improvement
38Determining Interventions
- Once you know how you are doing at baseline,
what interventions will produce meaningful
improvement in the target population?
39First Do A Barrier Analysis
- What did an analysis of baseline results show ?
- How can we relate it to system improvement?
- Opportunities for improvement
- Determine intervention(s)
- Identify barriers to reaching improvement
40How was intervention(s) chosen?
- By reviewing the literature
- Evidence-based
- Pros cons
- Benefits costs
- Develop list of potential interventions -- what
is most effective?
41Types of Interventions
- Education
- Provider performance feedback
- Reminders tracking systems
- Organizational changes
- Community level interventions
- Mass media
42Choosing Interventions
- Balance
- potential for success with ease of use
- acceptability to providers collaborators
- cost considerations (direct and indirect)
- Feasibility
- adequate resources
- adequate staff and training to ensure a
sustainable effort
43Physician Interventions Multifaceted Most
Effective
- Most effective
- real-time reminders
- outreach/detailing
- opinion leaders
- provider profiles
- Less effective
- educational materials (alone)
- formal CME program without enabling or
reinforcing strategies
44Patient Interventions
- Educational programs
- Disease-specific education booklets
- Lists of questions to ask your physician
- Organizing materials flowsheets, charts,
reminder cards - Screening instruments to detect complications
- Direct mailing, media ads, websites
45Evaluating Interventions
- Does it target a specific quality indicator?
- Is it aimed at appropriate stakeholders?
- Is it directed at a specific process/outcome of
care or service? - Did the intervention begin after baseline
measurement period?
46Interventions Checklist
- Analyze barriers (root causes)
- Choose understand target audience
- Select interventions based on cost-benefit
- Track intermediate results
- Evaluate effectiveness
- Modify interventions as needed
- Re-Measure
47Group Activity
48Quality Outcomes Achieved
- Three Components
- Activity VIII. Presentation of Sufficient Data
Analysis and Interpretation - Activity IX. Evidence of Real Improvement
Achieved - Activity X. Data Supporting Sustained
Improvement Achieved
49Activity VIII. Presentation of Sufficient Data
Analysis and Interpretation - HMO Overall Score
50Activity VIII. Presentation of Sufficient Data
Analysis and Interpretation
- Results
- 35 percent of the nine evaluation elements were
Met - 15 percent of the evaluation elements Partially
Met or Not Met - 50 percent of the evaluation elements were Not
Applicable or Not Assessed
51Activity VIII Review Data Analysis and
Interpretation of Study Results
- HSAG Evaluation Elements
- Is conducted according to the data analysis plan
in the study design. - Allows for generalization of the results to the
study population if a sample was selected. - Identifies factors that threaten internal or
external validity of findings. - Includes an interpretation of findings.
- Is presented in a way that provides accurate,
clear, and easily understood information.
52Activity VIII Review Data Analysis and
Interpretation of Study Results (cont.)
- HSAG Evaluation Elements
- Identifies initial measurement and remeasurement
of study indicators. - Identifies statistical differences between
initial measurement and remeasurement. - Identifies factors that affect the ability to
compare initial measurement with remeasurement. - Includes the extent to which the study was
successful. - Bolded evaluation elements show areas for
improvement
53Changes in Study Design?
- Study design should be same as baseline
- Data source
- Data collection methods
- Data analysis
- Target population or sample size
- Sampling methodology
- If change
- rationale must be specified appropriate
54Activity IX. Evidence of Real Improvement - HMO
Overall Score
55Activity IX. Evidence of Real Improvement
- Results
- 22 percent of the four evaluation elements were
Met - 29 percent were Partially Met or Not Met
- 50 percent of the evaluation elements were Not
Applicable or Not Assessed
56Activity IX Assess the Likelihood that Reported
Improvement is Real Improvement
- HSAG Evaluation Elements
- The remeasurement methodology is the same as the
baseline methodology. - There is documented improvement in processes or
outcomes of care. - The improvement appears to be the result of
intervention(s). - There is statistical evidence that observed
improvement is true improvement. - Bolded evaluation elements show areas for
improvement
57Statistical Significance Testing
Time Periods Measurement Periods Numerator Denominator Rate or Results Industry Benchmark Statistical Testing and Significance
CY 2003 Baseline 201 411 48.9 60 N/A
CY 2004 Re-measurement 1 225 411 54.7 60 Chi-square 2.8 P-value 0.09387 NOT SIGNIFICANT AT THE 95 CONFIDENCE LEVEL
58Activity X. Data Supporting Sustained Improvement
Achieved - HMO Overall Score
59Activity X. Data Supporting Sustained
Improvement Achieved
- Results
- 6 percent of the one evaluation element was Met
- 36 percent was Partially Met or Not Met
- 58 percent of the evaluation element was Not
Applicable or Not Assessed
60Activity X Assess Whether Improvement is
Sustained
- HSAG Evaluation Elements
- Repeated measurements over comparable time
periods demonstrate sustained improvement, or
that a decline in improvement is not
statistically significant. - Bolded evaluation elements show areas for
improvement
61Quality Outcomes Achieved
Baseline 1st Yr
Demonstrable Improvement
Sustained Improvement
62Sustained Improvement
- Modifications in interventions
- Changes in study design
- Improvement sustained for 1 year
63HSAG Contact Information
Cheryl Neel, RN, MPH,CPHQ Manager, Performance
Improvement Projects cneel_at_hsag.com 602.745.6201
Denise Driscoll Administrative Assistant ddriscoll
_at_hsag.com 602.745.6260
64Questions and Answers