Estimating Treatment Effects with Observational Data using Instrumental Variable Estimation: The Ext

1
Estimating Treatment Effects with Observational
Data using Instrumental Variable Estimation The
Extent of Inference John M Brooks. Ph.D. Health
Effectiveness Research Center (HERCe) Colleges of
Pharmacy and Public Health University of
Iowa June 8, 2004
Health Effectiveness Research
Center
2
Research Goal Estimate casual relationships
between "treatment" and outcome in
healthcare... treatment on
outcome behavior on outcome system change
on outcome
3
The best estimation method to make inferences
about these relationships is a function of 1.
the manner in which the researcher collects
data and 2. the approach used to control
for confounding factors confounding
factors factors related to both the
treatment and
outcome.
4
(No Transcript)
5
Sources of Treatment Variation in Health
Care 1. Randomized Controlled Trials study of
patients with a given medical condition in
which treatment is randomly assigned.
Why randomly assign treatment to patients?
To help ensure that estimated treatment affects
are attributable to the treatment and not
unmeasured confounders. The Gold Standard
6
Why dont we do more Randomized Controlled
Trials between approved treatments? ?
ethical problems ? expensive and
time-consuming ? little
motivation ? inability to generalize
7
2. Observational Healthcare Databases
Database Types ? Claims medical
service treatment claims from
individuals with health insurance ?
Provider-Specific databases describing the

utilization of a set of providers. ?
Health Care Surveys surveys of patients or

providers detailing health
care utilization.
8
Strengths ? plenty of variation in
treatment choice ? potentially enhanced
ability to generalize reveals
variation in treatment choice across a variety
of clinical scenarios ? can
assess treatments in practice estimate
effectiveness ? unobtrusively
collected ? the power of large numbers and
time.
9
Weaknesses ? data usually not collected
for researcher purposes ? missing
information - care not covered is not
observed - care not claimed is not
observed - claim form limitations -
nuances of illness, treatment, and patient that
cant be recorded on claims forms ? patient
enrollment variation ? confounding information
may be unobserved.
10
Is the Main Source of Weakness with Observational
Data Unmeasured Confounders or Treatment
Selection Bias? 1. Unmeasured Confounders
Unmeasured Confounders argument ?
homogenous treatment effect ? unmeasured
factors related to both treatment and
outcome is the source of bias.
11
Assume true outcome relationship is
Y ao a1T a2L e
where Y measure of outcome (e.g.
1 if survive to a certain time
period, 0 otherwise) T 1 if receive
treatment, 0 otherwise and L
additional factor (e.g. severity, other
treatments). Goal is to estimate a1 the effect
of treatment on outcome.
12
For Estimation Suppose ? L is not
measured and the estimation model is Y ao
a1T u where u (a2L
e) ? L is related to Y (a2 ? 0) and
? T and L are related (Cov(T,L) ?
0). Cov(T,L) covariance of T L. Cov(T,L) ? 0
essentially means that T L move together.
13
Define the ordinary least squares (ANOVA)
estimate of a1 as . ? It can be
shown that under these assumptions is
a biased estimate of a1 through its expected
value
a1 Cov(T,L)a2 ? Also note that
will equal a1 if either -- Cov(T,L) 0
or -- a2 0.
14
Suppose theory about the unmeasured variable
L suggests ? a2 lt 0 (patients with
higher severity have lower
cure rate). ? Cov(T,L) gt 0 (treated
patients are generally more
severe). Plug in signs into our
expected value formula to find
lt a1.
15
Problem with the Unmeasured Confounders
argument to describe bias in observational
data ? It does not provide a theoretical
foundation to link treatments to
unmeasured factors.... Why is
Cov(T,L) ? 0? ? In the case we just
described, if treatment effect (a1) is
the same for all patients, why would Cov(T,L) gt
0? Perhaps patients getting
treatments -- live in areas with
high/low poverty -- live in areas with
more pollution or -- also tend to get
other unmeasured treatments.
16
2. Treatment Selection Bias (the gestalt
underlying most negative reviewers
comments) Treatment Selection Bias
argument ? heterogeneous treatment
effect -- Cov(T,L) reflects the
decision-makers beliefs about the
differences in treatment effectiveness across
patients and ? bias comes
from unmeasured factors (severity,
other treatments) related to the treatments
expected effectiveness that affects both
treatment choice and outcome.
17
Assume true outcome relationship is Y
bo (b1 b2L) T b3L e where
Y measure of outcome (e.g. 1 if survive
to a certain time period, 0
otherwise) T 1 if receive
treatment, 0 otherwise L
unmeasured factor (e.g. severity, other
treatment) b3 the
direct effect of L on Y and (b1 b2L)
effect of T on Y that depends on L.
18
? L is now related to T through theory linking
"treatment choice" to the decision-makers
expectations of treatment benefits across
patients with different L. T co c1L
c2W v where T 1 if
receive treatment, 0 otherwise L
unmeasured factor (e.g. severity, other
treatment) affecting treatment choice
through expected treatment
effectiveness and W other factors
affecting treatment choice. If decision
makers use L in treatment decisions, c1
? 0 and Cov(T,L) ? 0.
19
Ultimate goal should be to estimate (b1
b2L) the effect of treatment T on outcome Y
across levels of L. For estimation suppose
? L is not measured and it is wrongly assumed
by the researcher that the effect of T
is homogenous, the estimation model
is Y ao a1T u where
u (b2LT b3L e)
20
Define the ordinary least squares (ANOVA)
estimate of a1 as . ? It can be
shown that the expected value of is
? If c1 0 (no selection based on L), then
becomes Yields an average estimate
that depends on the mix of L in the
population (e.g. RCT using a broad population).
21
How does c1 (b2 b3) affect this
estimate? ? Assume that L is unmeasured
illness severity and that higher L
means more severe illness. ? Higher L
lowers survival which implies b3 lt 0. ? If
treatment benefit is less for more severe cases
(e.g. surgery for heart attacks)
then benefit falls less
treatment with higher in
more severity severe
cases Estimate of average population
treatment benefit will be biased high.
22
? If treatment benefit is greater for more
severe cases (e.g. antibiotics for
otitis media) then benefit increases more
treatment with higher in
more severity severe
cases Estimate of average population
treatment affect is biased but sign can
not be determined.
23
So what do we have here? ? Observational
data contains enormous treatment
variation. ? Treatment choice may be related
to the selection or sorting of patients
using unmeasured (to the researcher)
characteristics that are related to
expected outcomes. ? Under selection,
standard statistical techniques yield
biased estimates that dont apply to anyone
anyway. Do we have any alternatives?
24
Instrumental Variables (IV) Estimation and
Subset B IV estimation offers consistent
estimates for a subset of patients (McClellan,
Newhouse 1993) Marginal Patients patients
whose treatment choices vary
with measured factors called
instruments
that do not directly affect outcomes.
McClellan and Newhouse argue that estimates of
treatment effects for Marginal Patients are
useful ? They are estimates for patients for
whom the benefits of treatment are the
least certain patients least like those in
RCTs. ? Estimates may be more suitable
than RCT estimates to address the question
of whether existing treatment rates should
change.
25
(No Transcript)
26
Patients in Subset B are interesting
because ? the best treatment choice
(treat or dont treat) is least
certain ? treatment or no-treatment for a
patient in this subset is not considered
bad medicine the art of medice ? the
possibility of gaining new RCT evidence for
patients in this subset is remote
(ethics, motivation) ? McClellan et al.
1994 argue that policy interventions
affect mainly the treatment choices for patients
in this subset and ? Non-clinical
factors (e.g. provider access, market
pressures) affect mainly the treatment choices
of patients in this subset.
27
(No Transcript)
28
(No Transcript)
29
IV estimation involves 1. Finding
measured variables or instruments (Z) that
a. are related to the possibility of a
patient receiving treatment (cov(T,Z)
? 0) and b. are assumed (through theory)
unrelated directly to Y or to
unmeasured confounding variables (cov(Z,L)
0). The theoretical basis for Z variables
should come from a model of treatment choice
the W variables in T co c1L
c2W v where W other
factors affecting treatment choice.
30
IV estimation involves cont 2.
Grouping patients using values of the
instrument. 3. Estimate treatment
effects for marginal patients by
exploiting treatment variation rate differences
across patient groups. Local
Average Treatment Effect --
(Imbens Angrist 1994)
31
For example, if an instrument divides patients
into two groups, a simple IV estimate can be
found by calculating 1. the overall
treatment rate in each group (ti treatment
rate in group i) and 2. the
overall outcome rate in each group (yi outcome
rate in group i) and estimate
where average treatment effect for the
marginal patients specific to
the instrument used in the analysis
only those patients whose treatment choices
were affected by the instrument
who must have come from Subset B.
32
(No Transcript)
33
We have treatment rates for each group
Closer Group Treatment Rate .60
Further Group Treatment Rate .50 Suppose
we also measured cure rates in both groups
Closer Group Cure Rate .40
Further Group Cure Rate .38 Four numbers
lead to the following IV estimate
34
Strict Interpretation ? If the
treatment rate in the Further Group was increased
.01 percentage point (e.g. .50 to .51)
by increasing treatment for the M
patients in the Further Group, the Cure rate in
the Further Group would increase .002
(.01 .2) from .38 to .382.
Stretched Policy-Relevant Interpretation
(McClellan et al. 1994) ? A behavioral
intervention that increases the overall
treatment rate by .01 percentage point (e.g. .55
to .56) would lead to an increase in
the cure rate of .002 (.01 .2).
35
Stretched interpretation assumes that the
treatment effect for patients in Subset B is
fairly homogenous and an IV estimate from a
single instrument can be generalized to all
patients in Subset B. This allows one to say
Stretched interpretation is not perfectly
accurate if treatment effects are
heterogeneous within Subset B and different
instruments affect treatment choices from
different patients within Subset B. ?
Results from a single instrument may still be
more appropriate than assuming RCT
results apply to Subset B. ? Ability to
generalize results may increase if more than one
instrument is used in an IV analysis.

36
IV qualifiers to remember ? second
property of IV variables (cov(Z,L) 0) is
forever an assumption (unless more data are
obtained) and ? unmeasured but
correlated treatments may still bias
estimated treatment benefits. Researchers should
fully qualify their IV estimates don't oversell.
37
Hypothetical Example to Demonstrate 4-Number
Result Suppose 2100 children with Acute
Otitis Media (AOM) in a population. Two
treatment possibilities 1. antibiotics 2. wat
chful waiting. The patients in our sample are
in one of three severity types low,
medium, and high Severity type is
observed by the provider/patient but is not
observed by the researcher.
38
The 2100 patients are distributed across
severity type in the following manner

severity type
High Medium Low number of patients
800 800 500 The actual
underlying cure rates for each severity type by
treatment are
severity type treatment
High Medium Low
antibiotics .95 .97 .98
watchful waiting .80 .90
.98
39
? Higher severity means a lower the cure rate
in general (b3 lt 0). ? Antibiotics
have a higher curative effect in more severe
patients and offer no advantage to the less
severe (b2 gt 0). ASSUMPTION
Treatment effects are heterogenous. ? All
providers have inclination that antibiotics work
well in the "high" severity patients
have little effect on the "low"
severity patients but the effect in the "medium"
type is unknown to providers.
Leads to selection bias...the more severe
kids are treated (c1 gt 0).
40
Potential Methods to analyze 1. Randomize
Patients Into Treatments -- ANOVA 2.
Providers Assign Treatments -- ANOVA 3.
Instrumental Variable Grouping
41
1. Randomize Patients Across Population
ANOVA. Patient Treatment Assignments After
Randomization by Severity Type

severity type patient groups High
Medium Low antibiotics
400 400 250
watchful waiting 400 400 250
42
Expected average cure rates for each
group Unbiased average antibiotic
treatment rate for the entire population
(.965-.881 .084), but To whom does it
apply? A patient randomly chosen from an
urn? Are patients chosen from urns?
43
2. Providers Assign Treatments -- ANOVA If
providers follow inclinations, we may end up
with something like Number of Patients Assigned
by Providers to Each Treatment Group by Severity
Type
severity type patient group
High Medium Low antibiotics
800 400 0 watchful
waiting 0 400 500
44
Expected average cure rates for each
group For this population the
average treatment effect is (.084). We find a
biased low estimate of the antibiotic treatment
effect for the average patient (.957 - .944
.013 lt .084). To which patients does this
estimate apply?
45
3. Instrumental Variable Grouping -- Further
a. Assume information is available to
approximate distances from patients
to providers address of
patient supply of providers in area around
patients b. Evidence suggests that
patients in areas with more
physicians per capita have a higher probability
of being treated with antibiotics for
their AOM than patients in areas with
fewer physicians per capita.
46
If b is true, divide 2100 patients into two
groups based on the physicians per capita in the
area around their home Group 1 the group of
patients living in areas with a higher
number of physicians per capita Group 2
the group of patients living in areas with a
lower number of physicians per
capita
47
Using our assumptions, does this grouping qualify
as an instrument? 1. Doc supply related to
treatment? Yes, if patients tend to go to

the closest provider for

treatment. If true, and providers follow
inclinations we may see treatment patterns
something like Patient Treatment Assignments by
Severity Type patient severity type
group High Medium
Low Group 1 100 antibiotics 80
antibiotics 100 W.W.
20 W.W. Group
2 100 antibiotics 30 antibiotics
100 W.W.
70 W.W.
48
2. Is grouping related to unmeasured
confounding variables (e.g. severity)?
Related to severity only if parents chose
residences in expectation of the severity of a
future acute condition. If not related
to severity, we assume equivalent severity
distributions across groups
Number of Patients in Each Group by Severity
Type
severity type patient
group High Medium Low
Group 1 400
400 250 Group
2 400 400
250
49
Expected average estimated cure rates for these
groups Well, (.959428 - .946092)
.013336 doesn't appear to reveal much of
anything
50
Now look at the antibiotic treatment rate in each
group 720/1050 .68571 in Group
1 520/1050 .4952381 in Group 2 These
differences also don't look very
informative. The IV change in the cure rates
resulting from a one unit increase in the drug
treatment rate equals This estimate is
the average difference in the antibiotic cure
rate for the marginal or in this example the
Medium severity patients.
51
Remember the actual unknown cure rates for
each group by treatment are
severity type treatment High
Medium Low antibiotics .95
.97 .98 watchful waiting
.80 .90 .98
.07 This estimate was found using
only measured treatment rates and outcome
rates across groups that are defined by the
instruments. Which of the estimates above is
the most important for policy-makers
wondering about over/underutilization of a
treatment?
52
IV Brass Tacks Where do instruments come
from? ? Theory on what motivated
choices, not theory on how choices can
be motivated. ? Observed differences in
-- guideline implementation
(timing/interpretation) -- product
approval rules across payers --
reimbursement across payers/geography
-- area provider treatment signatures
-- geographic access to relevant providers
-- provider market structure/competition
? Generally, Natural Experiments (Angrist and
Krueger, 2001)
53
General IV Estimation Model Treatment
Choice Equation (1st stage) Outcome Equation
(2nd stage) Yi 1 if health outcome
occurs, 0 otherwise Xi measured patient
clinical characteristics Ti 1 if
patient received treatment, 0 otherwise
predicted treatment from 1st stage Zi
a set of binary variables to grouping
patients based on values of
instrumental variables (from W) and Li
unmeasured confounding variables assumed related
to both Y and T but not Z (from
W). The only variation in T used to estimate a1
comes from Z.
54
? The estimate of a1 can only be definitively
generalized to the patients whose treatment
choices were affected by Z (Angrist,
Imbens, Rubin 1996). ? F-test of whether the
parameters within c3 are simultaneously
equal to zero provides a test of the first
instrumental variable criterion Finding
measured variables or instruments (Z) that
a. are related to the possibility of a
patient receiving treatment
(cov(T,Z) ? 0)
55
? Model can be estimated via --
Two-Stage Least Squares (2SLS) PROC
SYSLIN in SAS. -- Bivariate
Probit BIPROBIT function in STATA.
-- Two-Stage Replacement (e.g. Beenstock
Rahav, 2002). ? 2SLS offers
consistent estimates that are asymptotically
normal with the fewest assumptions (Angrist
2001). -- essentially regressing
group-level outcome rate changes on
group-level treatment rate changes.
56
How many groups? ? Z can be specified as
a continuous variable, but results are
then conditional on this assumption and is less
interpretable. ? Creating many
groups from an instrument (more binary
variables in Z) uses more information and yields
a weighted average of many two-group
comparisons, e.g. -- low/high
groups using the median of the instrument
VS -- low/med low/med
high/high groups using the
quartiles of the instrument. ? Too many
groups may introduce bias. ? Best to report
estimates for several grouping strategies.
57
Example The effect of breast-conserving
surgery (BCS) relative to
mastectomy (MAS) for stage
II breast cancer patients (Brooks et al. 2003).
? Sample ESBC Stage II patients (N 2,905)
from the Iowa SEER Cancer
Registry, 1989-1994 that
had either BCS or MAS. ? Measures
-- Treatment Had BCS plus irradiation.
-- Outcomes Survival 1, 2, 3 and 4 years.
? Instrument BCS percentage for all other
early-stage breast
cancer patients in 50-mile radius
of patient zip code in diagnosis
year.
58
Comparison of Characteristics of ESBC Patient
Groups
In Iowa, 1989-1994 Treatment vs. Area BCS
Rates Group based on Group based on
area actual treatment choice
treatment signature Patient
High Low Chars BCS
Mastectomy BCS areaa BCS areaa
. BCS 100
0 12
8 Under 65 67 44
53 48 65 to 74
22 25 23
25 Over 74 9
27 24
27 Stage IIb 21
35 35
33 Comor Indexb .15 .31
.31 .28 Number
2622 283
1225 1680 ,,
significant differences at the .01, .05 and .10
percent confidence levels, respectively. a.
Based on 50-mile radius around patients zip code
in year of diagnosis. High areas have BCS
percentage greater than or equal to 22 (includes
stage I patients). Low areas have BCS
percentages less than 22. Rates are calculated
excluding the patient. b. Modified version of
Charlson Co-morbidity index using non-cancer
ICD-9 codes from patients hospital
discharge abstracts. Equals one if index is
greater than zero, zero otherwise.
59
(No Transcript)
60
? IV estimates using area BCS rate as
instrument. ,, statistically
significant at .99, .95, and .90 confidence,
respectively.
61
How many instruments? ? Patients in
Subset B affected by instruments may vary
across instruments, so IV estimates may vary.
? IV estimates using Distance to Radiation
as an instrument ,,
statistically significant at .99, .95, and .90
confidence, respectively.
62
? IV estimates using both area BCS rate and
distance to radiation ,,
statistically significant at .99, .95, and .90
confidence, respectively. ? Each
instrument remained independently significant.
? Estimates are weighted average.
63
Which Sample? ? Estimates for Marginal
Patients may vary by sample.
,, statistically significant
at .99, .95, and .90 confidence, respectively.
64
Which Sample (Example 2)? ? Effects of
Catheterization on AMI Patient Mortality
by Insurance Status using Differential Distance
as an Instrument (Brooks et al. 2000).
? Data from Washington State
1989-1993 ? Lower catheterization
rate reveals higher benefit for marginal
patients.
,, statistically significant at .99, .95,
and .90 confidence, respectively.
65
Summary The foundation of IV estimation is
theory that suggests instruments what
factors motivated treatment choices. Ability
to generalize is limited, but IV estimates offer
a more natural estimate of the effects of rate
changes than RCT estimates. Estimates can
vary by sample and instrument used. Estimates
are conditional on the truth (and acceptance)
of a known identification restriction. The
source of the treatment variation is known.
The relationship between this variation and
unmeasured confounders can be debated.
66
References Angrist JD, 2001. Estimation of
Limited Dependent Variable Models with Dummy
Endogenous Regressors Simple Strategies for
Empirical Practice. Journal of Business
Economic Statistics. 19(1)2-16 Angrist, JD,
Imbens GW, Rubin, DB. 1996. Identification of
Causal Effects Using Instrumental Variables.
Journal of the American Statistical Association.
91444-454. Angrist JD, Krueger AB. 2001.
Instrumental Variables and the Search for
Identification From Supply and Demand to Natural
Experiments. Journal of Economic Perspectives.
15(4) 69-85. Brooks JM, Chrischilles E, Scott
S, Chen-Hardee S. 2003. Was Lumpectomy
Underutilized for Early Stage Breast Cancer?
Instrumental Variables Evidence for Stage II
Patients from Iowa. Health Services Research,
38(6)1385-1402. Brooks JM, McClellan M, Wong H.
2000. The Marginal Benefits of Invasive Treatment
for Acute Myocardial Infarction Does Insurance
Coverage Matter? Inquiry, 37(1)75-90. Imbens
GW, Angrist, JD. 1994. Identification and
Estimation of Local Average Treatment Effects,
Econometrica. 62(2)467-475. McClellan M, McNeil
BJ, Newhouse JP. 1994. Does More Intensive
Treatment of Acute Myocardial Infarction in the
Elderly Reduce Mortality Analysis Using
Instrumental Variables", Journal of the American
Medical Association. 272859-866. McClellan M,
Newhouse JP. 1993. The Marginal Benefits of
Medical Treatment Intensity. Cambridge,Mass
National Bureau of Economic Research Working
Paper. McClellan M, Newhouse JP. 1997. The
Marginal Cost-Effectiveness of Medical Technology
- a Panel Instrumental Variables Approach,
Journal of Econometrics. 7739-64.