Performance and outcome measurement I: Epidemiological and statistical concepts - PowerPoint PPT Presentation

Loading...

PPT – Performance and outcome measurement I: Epidemiological and statistical concepts PowerPoint presentation | free to view - id: 13f474-N2M0M



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

Performance and outcome measurement I: Epidemiological and statistical concepts

Description:

1. Clarify the purpose of measurement. 2. Identify the concepts to be measured ... 1. Clarify the purpose. Establish accountability for Title V funds. expenditures ... – PowerPoint PPT presentation

Number of Views:79
Avg rating:3.0/5.0
Slides: 65
Provided by: mikes178
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Performance and outcome measurement I: Epidemiological and statistical concepts


1
Performance and outcome measurement
IEpidemiological and statistical concepts
  • 1999 AMCHP skill building session
  • Michael A. Stoto, PhD
  • The George Washington University
  • School of Public Health Health Services

2
Outline of presentation
  • Measurement theory and methods
  • Epidemiological methods
  • surveillance and public health assessment
  • rates and proportions
  • Statistical concepts
  • population and measurement variability
  • sampling theory and survey methods
  • confidence intervals
  • Binomial and Normal distributions
  • simple regression analysis
  • Practical examples
  • validity and reliability
  • promoting successful birth outcomes

3
Measurement theory methods
  • Steps for developing measures
  • 1. Clarify the purpose of measurement
  • 2. Identify the concepts to be measured
  • 3. Identify specific indicators of these concepts
  • 4. Assess validity, reliability, responsiveness
    to change
  • will discuss this in detail later

4
1. Clarify the purpose
  • Establish accountability for Title V funds
  • expenditures
  • performance measures
  • impact on outcomes
  • Improve processes and health outcomes

5
2. Identify the concepts
  • Identify responsibility and accountability for
    performance
  • Evidence-based link between performance and
    health
  • Balance between short- and long-range goals
  • Balance among levels and types of service

6
Levels and types of service
7
3. Identify specific indicators
  • Timely availability of data at a reasonable cost
  • Explicit definitions
  • Inclusion in other indicator sets

8
Concepts vs. indicators
  • Concept
  • mortality
  • presence of disease
  • health risks
  • costs
  • quality
  • access
  • Operational form
  • disease specific mortality rate
  • disease prevalence rate
  • risk factor prevalence rate
  • treatment costs per patient
  • patient satisfaction ratings
  • percent of population with health insurance

9
Concepts vs. indicators(capacity measures)
  • SSI with CSHCN
  • Percent of state SSI beneficiaries lt 16 years old
    receiving services from the state Children with
    Special Health Care Needs program
  • Medical home
  • Percent of CSHCN who have a medical/health home

10
Concepts vs. indicators(process measures)
  • Medicaid children
  • Percent of potentially Medicaid eligible children
    who have received a service paid by the Medicaid
    program
  • Lead screening
  • Proportion of children aged 6 mo. to 5 years
    screened for excess blood lead

11
Concepts vs. indicators(outputs, intermediate
outcomes)
  • Prenatal care
  • Percent of infants born to pregnant women
    receiving prenatal care beginning in the first
    trimester
  • Newborn screening
  • Percent of newborns with at least one screening
    for each of PKU, hypothyroidism, galactosemia,
    hemoglobinopathies
  • Vaccine coverage
  • Percent of children though age 2 who have
    completed MMR, DPT, polio, Hib, and hepatitis B
    immunizations

12
Concepts vs. indicators(health outcomes)
  • Infant mortality
  • Number of infant deaths (lt 1 year) divided by
    number of live births (per 1,000)
  • Low birth weight
  • Percent of live births lt 1500 grams at birth
  • Teen pregnancy
  • Birth rate (per 1,000) for teenagers aged 15
    through 17 years
  • School success
  • Average reading scale scores of 9 year olds

13
Sources of data for performance measurement
  • Vital statistics
  • maternal and infant mortality
  • birth certificate information
  • birth/death match
  • Examples
  • prenatal care
  • low birthweight births
  • low birthweight children born at level iii
    facilities
  • teenage fertility rate
  • motor vehicle crash deaths
  • suicide deaths among youths

14
Birth certificate information
  • Location information
  • state, county, and place of residence
  • place of birth and attendance
  • Infant information
  • race, sex, gestation, birthweight, Apgar
  • Mother age, race/Hispanic, education, marital
    status
  • Father age, race/Hispanic
  • Prenatal care month began, number of visits
  • Medical/health data
  • tobacco alcohol use, weight gain during pregnancy
  • medical risks, complications of labor

15
Sources of data for performance measurement
  • Surveillance data
  • Special purpose surveys
  • Examples
  • immunization coverage
  • dental sealants
  • breastfeeding at hospital discharge
  • hearing screening before hospital discharge
  • children with health insurance

16
Sources of data for performance measurement
  • Program data
  • Title V, other MCH, other programs
  • Examples
  • CSHCN with
  • medical/health home
  • insurance for primary and specialty care
  • Medicaid services to Medicaid eligible
  • Newborn screening

17
Epidemiological methods
  • Surveillance and public health assessment
  • data sources
  • Mortality and morbidity analyses
  • rates and proportions

18
Surveillance
  • Purposes
  • identification of emerging health problems
  • identification of affected individuals
  • continued watchfulness over disease in the
    population
  • Types of surveillance
  • active
  • passive
  • sentinel health events

19
Surveillance data sources
  • Vital statistics (mortality and fertility)
  • Disease control programs
  • case reporting/surveillance systems
  • case finding for selected populations
  • Administrative data
  • employers and schools
  • absenteeism, periodic physical exams
  • health plans, hospitals, pharmacies
  • administrative data
  • provider and patient surveys

20
Surveillance data sources
  • Prevalence surveys
  • Purpose to provide information on the frequency
    of disease in a population
  • Types of cross-sectional surveys
  • Interview National Health Interview Survey
    (NHIS)
  • Examination National Health and Nutrition
    Examination Survey (NHANES)
  • Health records National Hospital Discharge Survey

21
Public health assessment
  • Definition
  • The regular collection, analysis,
    interpretation, and communication of information
    about health conditions, risks, and assets in a
    community (IOM, 1988)
  • Other assessment data
  • census and general population surveys
  • administrative data from health sector
  • also social services, housing, education,
    transportation, etc.

22
Epidemiologic analyses
  • Time e.g. seasonality
  • Place spot and area maps
  • Time and space clusters
  • Person
  • age, gender
  • race and ethnicity
  • genetic background
  • social class and SES

23
Mortality analyses
  • Mortality rates
  • defined population group -- denominator
  • time period
  • number of deaths -- numerator
  • in that population during that time period
  • Crude death rate -- all ages and causes
  • Age- and sex-specific death rates
  • restrict numerator and denominator
  • Cause-specific death rates
  • restrict numerator

24
Mortality analyses
  • Age adjustment
  • direct
  • indirect
  • Survival/life table measures
  • life expectancy
  • Other mortality measures
  • case fatality rate
  • proportionate mortality rate/ratio (PMR)

25
Measures of disease frequency
  • Incidence rate
  • of new cases in a specified time period
  • of persons at risk of developing the disease
  • Prevalence rate
  • of cases present in a specified time period
  • of persons at risk of having the disease
  • Point prevalence number of cases at a specified
    moment
  • Period prevalence number of cases that occur
    during a specified period

26
Measures of disease frequency
  • Proportion ratio with
  • denominator of individuals in group
  • numerator of these with specified
    characteristics
  • Rates that are really proportions
  • attack rate proportion of individuals in some
    group that get a specified disease
  • case fatality rate proportion of individuals
    with a specified disease who die of that disease

27
Statistical concepts
  • Unit of analysis
  • Level of measurement
  • Population and measurement variability
  • Binomial and Normal distributions
  • Confidence intervals
  • Sampling theory and survey methods

28
Unit of analysis
  • Individual
  • Defined population
  • nation, county, community, program
    participants, insured population, etc.
  • Institution
  • hospital, health plan, clinic, provider group,
    WIC program, etc.

29
Level of measurement
  • Nominal
  • categories without order
  • Ordinal
  • categories with order
  • Interval
  • continuous without natural zero
  • Ratio
  • continuous with natural zero

30
Population (subject) variability and measurement
error
  • Subject variability Measurement error
  • Outside temperature
  • Daily variation Thermometer error
  • Body temperature
  • Interpersonal variation Thermometer error
  • Body weight
  • Interpersonal variation Scale error
  • Self-reporting error

31
Normal/Gaussian distribution
  • Many continuous variables are approximately
    Normally distributed
  • but not all!
  • subject variability and measurement error
  • Normal distributions can be fully characterized
    by mean (?) and standard deviation (?)
  • variance (?2) (standard deviation) 2

32
Normal/Gaussian distribution
  • In a Normal distribution
  • about 2/3 of subjects are within 1 ? of ?
  • about 95 of subjects are within 2 ? of ?

33
Normal/Gaussian distribution
  • Example Distribution of birthweight (X)
  • ? 4000 g, ? 1000 g
  • What is probability that X gt 5000g?
  • Z (X - ?)/ ? (5000 - 4000)/1000 1
  • Prob(Zgt1) 1/2 1/3 1/6
  • What is probability that X lt 2000g?
  • Z (X - ?)/ ? (2000 - 4000)/1000 -2
  • Prob(Zgt-2) 1/2 0.05 0.025

34
Sampling theory
  • Simple random sample
  • each element in population has same, and
    independent, chance of being in a sample
  • Sample average ?xi / n
  • Central limit theorem
  • has a normal distribution with
  • mean ? and standard deviation ? / ?n

35
Sampling theory
  • Sample size n 1 and 10

?
36
Sampling theory
  • Sample size n 1, 10, and 100

?
37
Sampling theory
  • Example Average birthweight ( )
  • standard deviation of ? / ?n
  • n ?n ? / ?n
  • 1 1 1000/1 1000
  • 10 3.16 1000/3.16 316
  • 100 10 1000/10 100

38
Confidence intervals
  • Basic idea
  • is in the interval ? 1.96 ?/?n in 95 of
    samples
  • 1.96 ?/?n covers ? 95 of the time, and
    is called a 95 confidence interval
  • must use a t-value greater than 1.96 when ? is
    estimated from the data
  • e.g. t 2.6 when n 5

39
Confidence intervals
  • Example Average birthweight ( )
  • Sample values 3000, 5500, 2500, 4600
  • 15,600/4 3900
  • ?/?n 1000 / ?4 500
  • 95 Confidence interval (n 4)
  • 1.96 ?/?n 3900 1.96(500)
  • 3900 980 (2920, 4880)
  • 95 Confidence interval (if n 25, 3900)
  • 1.96 ?/?n 3900 1.96(1000)/ ?25
  • 3900 392 (3508, 4292)

40
Statistics for proportions
  • Many performance measures are in the form of
    proportions, i.e. a ratio with
  • of individuals with specified characteristics
  • p
  • of individuals in group
  • Counts have a Binomial distribution if
  • fixed n of binary outcomes
  • independent outcomes with same probability ?
  • As n increases, distribution of p is
    approximately Normal with
  • ? n? and ? ?(1-?)/n

41
Statistics for proportions
  • Example standard deviation of p

?
n
42
Statistics for proportions
  • n 10, ? .3

43
Statistics for proportions
  • n 50, ? .3

44
Regression analysis
  • What is the relationship between Y and X?
  • If a straight line, can be expressed as Y a
    bX
  • What is the slope, i.e. the increase in Y for a
    unit increase in X?
  • Is the slope significantly different than 0?
  • Since Y and Y are never exactly linear, use
  • Y a bX e, where e has a Gaussian
    distribution with ? 0, and e are independent
  • Choose a and b to minimize squared deviations
    ?(y - a - bx)2

45
Regression analysis
46
Regression analysis
47
Regression analysis
  • Example IMR trend analysis

48
Regression analysis
  • Example IMR trend analysis

49
Practical examples
  • Assessing proposed measures
  • Validity and reliability tradeoffs
  • Differentials and disparities
  • Successful birth outcomes

50
4. Assess the proposed measures
  • Validity
  • is the indicator measuring the right concept?
  • Reliability
  • is the indicator consistently measuring the
    concept?
  • Is measurement error small compared to population
    variability?
  • Robustness and responsiveness to change
  • will the indicator change if and only if the
    concept being measured changes?

51
Reliability and validity
Reliable
Valid
Reliable and valid
52
Increasing reliability
  • Reduce error variance
  • increase number of observers
  • increase number of observations
  • observer training
  • improve scales
  • Enhance true variance
  • improve scales

53
Reliability and validity
  • Performance measures are not the same as practice
    recommendations
  • example prenatal care in first trimester rather
    than by PHS recommendations
  • trade validity for availability

54
Reliability and validity
  • Use running averages and statistical smoothing
  • example average infant mortality rate over 3 or
    5 years
  • trade timeliness for reliability

55
Reliability and validity
  • Use proxy measures that reflect trends and
    differences
  • example low birth weight rather than infant
    mortality
  • trade validity for reliability

56
Differentials and disparities
  • Increasing national concern about racial, ethnic
    and other differentials in health outcomes
  • Healthy People 2010 focus on eliminating
    disparities
  • Use parallel indicators for different groups (not
    disparity measures)
  • Use appropriate independent variables to
    understand problem and solutions

57
Promoting successful birth outcomes
  • Example based on Access to Health Care in America
    (IOM, 1993)
  • One of five objectives Promoting successful
    birth outcomes
  • Combination of 4 utilization and outcome
    indicators

58
Promoting successful birth outcomes
  • Indicator 1 (utilization measure)
  • Percentage of pregnant women obtaining adequate
    prenatal care
  • Strength direct measure of access
  • Weaknesses
  • Measured by initiation and frequency
  • neither alone sufficiently measures adequacy
  • nothing on distribution of visits, content,
    continuity
  • large number could represent a problem
  • more complex indices can be confounded by missing
    or incomplete data
  • recall problems

59
Promoting successful birth outcomes
  • Indicator 2 (outcome measure)
  • Infant mortality rate
  • Strengths
  • commonly used measure of access
  • available everywhere from vital statistics
  • Weaknesses
  • provides little information about access barriers
  • rate includes causes of death that cannot be
    affected by the health care system
  • high variability in infant deaths in some areas

60
Promoting successful birth outcomes
  • Indicator 3 (outcome measure)
  • Low birthweight rate
  • proportion of infants weighing less than 2500 g
  • Specific to adequate prenatal care and access to
    nutrition services
  • Strengths
  • important predictor of infant survival
  • numerator not as rare, so more stable
  • Weakness
  • timeliness of published data

61
Promoting successful birth outcomes
  • Indicator 4 (outcome measure)
  • Congenital syphilis rate
  • Strength
  • reportable condition in most states
  • very specific to lack of or inadequate prenatal
    care
  • Weaknesses
  • reporting may be incomplete
  • syphilis is rare in most states

62
Conclusions
  • Performance measurement can be an effective tool
    for accountability and learning organizations
  • Depends on availability of valid and reliable
    indicators
  • Need a limited yet comprehensive set of coherent
    and significant indicators
  • which can be monitored over time and
  • disaggregated to relevant social units

63
References
  • IOM/NAS reports (www.nap.edu)
  • Also www2.nas.edu/hpdp and www2.nas.edu/bocyf
  • Improving Health in the Community (1997)
  • Access to Health Care in America (1993)
  • America's Children Health Insurance and Access
    to Care (1998)
  • Systems of Accountability Implementing
    Children's Health Insurance Programs (1998)
  • Paying Attention to Children in a Changing Health
    Care System (1996)

64
References
  • Additional IOM/NAS reports
  • Reducing the Odds Preventing Perinatal
    Transmission of HIV in the United States (1999)
  • From Generation to Generation The Health and
    Well-Being of Children in Immigrant Families
    (1998)
  • The Best Intentions Unintended Pregnancy and the
    Well-being of Children and Families (1995)
  • Overcoming Barriers to Immunization (1994)
  • Prenatal Care (1988)
  • Preventing Low Birthweight (1985)
About PowerShow.com