Estimating Treatment Effects with Observational Data using Instrumental Variable Estimation: The Extent of Inference

1
Estimating Treatment Effects with Observational
Data using Instrumental Variable Estimation The
Extent of Inference John M. Brooks, Ph.D. Health
Effectiveness Research Center (HERCe) Colleges of
Pharmacy and Public Health University of
Iowa June 26, 2005
Health Effectiveness Research
Center
2
Research Goal Estimate casual
relationships between "treatment" and
outcome in healthcare... ? treatment on
outcome ? behavior on outcome ? system change
on behavior (e.g. guideline
implementation) ? system change on outcome.
3
Written as a linear relationship
Y a0 a1 T our goal is to
obtain estimate(s) of a1. To estimate a1
T must move or vary. To make inferences about
a1 the source of the variation in T must be
scrutinized relative to your research goal.
4
Key research design issues for isolating and
using T variation 1. the manner in
which the researcher collects data
and 2. the approach to deal with
confounding factors confounding
factors factors that vary both
with T and Y.
5
(No Transcript)
6
Sources of Treatment Variation in Health
Care 1. Randomized Controlled Trials study of
patients with a given medical condition in
which treatment is randomly assigned.
Why randomly assign treatment to patients?
To help ensure that estimated treatment effects
result from the treatment variation and not
unmeasured confounders. The Gold Standard
7
Why not more Randomized Controlled Trials?
? ethical problems once treatment is
approved ? expensive and
time-consuming ? little motivation ?
patient sampling problems when comparing
existing treatments (so who wants to be
randomized?)
8
2. Observational Healthcare Databases Containing
Healthcare Treatment Choices
Secondary ? Claims medical service
treatment claims from
individuals with health insurance ?
Provider-Specific databases describing the

utilization of a set of providers.
Primary ? Health Care Surveys
surveys of patients or
providers detailing
health
care utilization.
9
Strengths ? plenty of variation in
treatment choice ? ability to study effects
of treatment across a variety of
clinical scenarios ? can assess treatments
in practice estimate
effectiveness ? often unobtrusively
collected ? the power of large numbers and
time.
10
Weaknesses ? often data usually not
collected for researchers purpose
(secondary) ? patient enrollment variation ?
confounding information may be unobserved.
- care not covered is not observed -
care not claimed is not observed - claim
form limitations - nuances of illness,
treatment, and patient that cant be recorded
on claims forms
11
Is the Main Weakness with Observational
Data Unmeasured Confounders or Treatment
Selection Bias? 1. Unmeasured Confounders
Unmeasured Confounders argument ?
homogenous treatment effect (a1 same for all
patients) and ? unmeasured
factors related to both treatment and
outcome is the source of bias.
12
Assume true outcome relationship is
Y ao a1T a2L e
where Y measure of outcome (e.g.
1 if survive to a certain time
period, 0 otherwise) T 1 if receive
treatment, 0 otherwise and L
additional factor (e.g. severity, other
treatments). Goal is to estimate a1 the effect
of treatment on outcome.
13
For Estimation Suppose ? L is not
measured and the estimation model is Y ao
a1T u where u (a2L
e) ? L is related to Y (a2 ? 0) and
? T and L are related (Cov(T,L) ?
0). Cov(T,L) covariance of T L. Cov(T,L) ? 0
essentially means that T L move together.
14
Define the ordinary least squares (ANOVA)
estimate of a1 as . ? It can be
shown that under these assumptions is
a biased estimate of a1 through its expected
value
a1 Cov(T,L)a2 ? Also note that
will equal a1 if either -- Cov(T,L) 0
or -- a2 0.
15
Suppose theory about the unmeasured variable
L suggests ? a2 lt 0 (patients with
higher severity are less likely
to survive). ? Cov(T,L) gt 0 (treated
patients are generally more
severe). Plug in signs into our
expected value formula to find
? lt a1.
16
Problem with the Unmeasured Confounders
argument to describe bias in observational
data ? No theoretical foundation linking
treatments to unmeasured factors....
Why is Cov(T,L) ? 0? ? In the
example above, if treatment effect (a1) is the
same for all patients, why would Cov(T,L) gt
0? Perhaps patients getting treated
-- live in areas with high/low poverty
-- live in areas with more pollution or
-- also tend to get other unmeasured
treatments.
17
2. Treatment Selection Bias (the gestalt
underlying most negative reviewers
comments) Treatment Selection Bias
argument ? Heterogeneous treatment
effect -- Cov(T,L) is a reflection
of decision-makers beliefs about the
treatment effectiveness across patients
related to unmeasured factors
L. ? Bias comes from unmeasured
factors (L) being related to the
treatment choice and outcome. ?
Researcher must address both bias and ability
to generalize (to whom do the results
apply?).
18
Assume true outcome relationship is Y
bo (b1L) T b2L e where Y
measure of outcome (e.g. 1 if survive to a
certain time period, 0
otherwise) T 1 if receive
treatment, 0 otherwise L
unmeasured factor (e.g. severity, other
treatment) b2 the
direct effect of L on Y and (b1L) effect
of T on Y that depends on L.
19
? L is now related to T through theory linking
"treatment choice" to the decision-makers
expectations of treatment benefits across
patients with different L. T co c1L
c2W v where T 1 if
receive treatment, 0 otherwise L
unmeasured factor (e.g. severity, other
treatment) affecting treatment choice
through expected treatment
effectiveness and W other factors
affecting treatment choice. If decision
makers use L in treatment decisions, c1 ?
0 and Cov(T,L) ? 0.
20
Ultimate goal should be to estimate (b1L)
the effect of treatment T on outcome Y across
levels of L. For estimation suppose
? L is not measured and it is wrongly assumed
by the researcher that the effect of T is
homogenous, and the estimation model
is Y ao a1T u where
u f(L,T, e, b1,b2)
21
Define the ordinary least squares (ANOVA)
estimate of a1 as . ? It can be
shown that the expected value of is
? If b2 0 (L has no direct effect on Y) or c1
0 (no selection based on L), then
becomes Yields an average estimate
of the treatment effect for the treated in
the sample. Result can be generalized only to
those with L similar to those treated.
22
How does c1 b2 affect this estimate? ?
Assume that L is unmeasured illness severity
and that higher L means more severe
illness. ? Higher L lowers survival which
implies b2 lt 0. ? If treatment benefit is
less for more severe cases (e.g.
surgery for heart attacks) then
benefit falls less treatment
with higher in more
severity severe cases
Estimate of the effect of the treatment on the
treated will be biased high.
23
? If treatment benefit is greater for more
severe cases (e.g. antibiotics for
otitis media) then benefit increases more
treatment with higher in
more severity severe
cases Estimate of the effect of the
treatment on the treated will be biased
low.
24
So what do we have here? ? Observational
data contains treatment variation. ? If
treatment benefits are heterogeneous the best
you can get is an estimate of the
treatment effect on the treated (Does this
address the benefits from expanding
treatments?). ? Treatment selection may be
based on unmeasured factors related to
both treatment effectiveness and
outcomes. ? If unmeasured factors affecting
selection also effect outcomes directly,
estimate will be biased. Do we have any
alternatives?
25
Instrumental Variables (IV) Estimation and
Subset B IV estimation offers consistent
estimates for a subset of patients (McClellan,
Newhouse 1993) Marginal Patients patients
whose treatment choices vary
with measured factors called
instruments
that do not directly affect outcomes.
McClellan and Newhouse argued that estimates of
treatment effects for Marginal Patients are
useful. ? Estimates may be more suitable than
RCT estimates to address the question of
whether existing treatment rates should
change.
26
(No Transcript)
27
Patients in Subset B are interesting
because ? the best treatment choice
(treat or dont treat) is least
certain ? treatment or no-treatment for a
patient in this subset is not considered
bad medicine the art of medicine
? the possibility of gaining new RCT evidence
for patients in this subset is remote
(ethics, motivation) ? McClellan et al.
1994 argue that (1) policy interventions
and (2) non-clinical factors (e.g.
provider access, market pressures) affect mainly
the treatment choices of patients in
this subset.
28
(No Transcript)
29
(No Transcript)
30
IV estimation involves 1. Finding
measured variables or instruments (Z) that
a. are related to the possibility of a
patient receiving treatment (cov(T,Z)
? 0) and b. are assumed (through
theory) unrelated directly to Y or to
unmeasured confounding variables (cov(Z,L)
0). The theoretical basis for Z variables
should come from a model of treatment choice
the W variables in T co c1L
c2W v where W other
factors affecting treatment choice.
31
IV estimation involves cont 2.
Grouping patients using values of the
instrument. 3. Estimate treatment effects
for marginal patients by exploiting
treatment rate differences across patient
groups. Local Average Treatment Effect
-- (Imbens Angrist 1994)
32
For example, if an instrument divides patients
into two groups, a simple IV estimate can be
found by calculating 1. the overall
treatment rate in each group (ti treatment
rate in group i) and 2. the
overall outcome rate in each group (yi outcome
rate in group i) and estimate
where average treatment effect for the
marginal patients specific to
the instrument used in the analysis
only those patients whose treatment choices
were affected by the instrument
who must have come from Subset B.
33
(No Transcript)
34
We have treatment rates for each group
Closer Group Treatment Rate .60
Further Group Treatment Rate .50 Suppose
we also measured cure rates in both groups
Closer Group Cure Rate .40
Further Group Cure Rate .38 Four numbers
lead to the following IV estimate
35
Strict Interpretation ? If the
treatment rate in the Further Group was increased
.01 percentage point (e.g. .50 to .51)
by increasing treatment for the M
patients in the Further Group, the Cure rate in
the Further Group would increase .002
(.01 .2) from .38 to .382.
Stretched Policy-Relevant Interpretation
(McClellan et al. 1994) ? A behavioral
intervention that increases the overall
treatment rate by .01 percentage point (e.g. .55
to .56) would lead to an increase in
the cure rate of .002 (.01 .2).
36
Stretched interpretation assumes that the
treatment effect for patients in Subset B is
fairly homogenous and an IV estimate from a
single instrument can be generalized to all
patients in Subset B. Stretched
interpretation may not be accurate if treatment
effects are heterogeneous within Subset B and
different instruments affect treatment choices
from different patients within Subset B.
? Results from a single instrument may still be
more appropriate than assuming RCT
results apply to Subset B. ? Ability to
generalize results may increase if more than one
instrument is used in an IV analysis.

37
IV qualifiers to remember ? second
property of IV variables (cov(Z,L) 0) is
forever an assumption (unless more data are
obtained) ? unmeasured but
correlated treatments may still bias
estimated treatment benefits and ?
ability to generalize is limited. Researchers
should fully qualify their IV estimates don't
oversell.
38
Hypothetical Example to Demonstrate 4-Number
Result Suppose 2100 children with Otitis
Media (OM) in a population. Two treatment
possibilities 1. antibiotics 2. watchful
waiting. The patients in our sample are in
one of three severity types low, medium,
and high Severity type is observed by the
provider/patient but is not observed by the
researcher.
39
The 2100 patients are distributed across
severity type in the following manner

severity type
High Medium Low number of patients
800 800 500 The actual
underlying cure rates for each severity type by
treatment are
severity type treatment
High Medium Low
antibiotics .95 .97 .98
watchful waiting .80 .90
.98
40
? Higher severity means a lower the cure rate
in general (b2 lt 0). ? Treatment
effects are heterogeneous and antibiotics have
a higher curative effect in more severe
patients and offer no advantage to the
less severe (b1 gt 0). ? All providers have
inclination that antibiotics work well in
the "high" severity patients have little
effect on the "low" severity patients
but the effect in the "medium" type is
unknown. ? Leads to treatment selection
bias...the more severe kids are treated
(c1 gt 0) and more severe kids are less likely
cured (b2 lt 0).
41
Potential Methods to Get Treatment Variation for
Analysis 1. Randomize Patients Into Treatments
-- ANOVA 2. Providers Assign Treatments --
ANOVA 3. Instrumental Variable Grouping
42
1. Randomize Patients Across Population
ANOVA. Patient Treatment Assignments After
Randomization by Severity Type

severity type patient groups High
Medium Low antibiotics
400 400 250
watchful waiting 400 400 250
43
Expected average cure rates for each
group Unbiased average antibiotic
treatment effect for the entire population
(.965-.881 .084), but Estimate will vary
with the average severity in the
population...ELT1. To whom does it
apply? A patient randomly chosen from an
urn? Are patients chosen from urns?
44
2. Providers Assign Treatments -- ANOVA If
providers follow inclinations, we may end up
with something like Number of Patients Assigned
by Providers to Each Treatment Group by Severity
Type
severity type patient group
High Medium Low antibiotics
800 400 0 watchful
waiting 0 400 500
45
Expected average cure rates for each
group For this population the
average treatment effect is on the treated
(800/1200.15 400/1200.07.123). We find a
biased low estimate of the antibiotic treatment
effect for the average treated patient (.957 -
.944 .013 lt .123). Biased low follows our
theory as...
46
3. Instrumental Variable Grouping Further
assume a. Information is available to the
researcher to approximate distances
from patients to providers
address of patient supply of providers in
area around patients b. Evidence suggests
that patients in areas with more
physicians per capita have a higher probability
of being treated with antibiotics for
their OM than patients in areas with
fewer physicians per capita.
47
If b is true, divide 2100 patients into two
groups based on the physicians per capita in the
area around their home Group 1 the group of
patients living in areas with a higher
number of physicians per capita. Group 2
the group of patients living in areas with a
lower number of physicians per
capita.
48
Using our assumptions, does this grouping qualify
as an instrument? 1. Doc supply related to
treatment? Yes, if patients tend to go to

the closest provider for

treatment. If true, and providers follow
inclinations we may see treatment patterns
something like Patient Treatment Assignments by
Severity Type patient severity type
group High Medium
Low Group 1 100 antibiotics 80
antibiotics 100 W.W.
20 W.W. Group
2 100 antibiotics 30 antibiotics
100 W.W.
70 W.W.
49
2. Is grouping related to unmeasured
confounding variables (e.g. severity)?
Related to severity only if parents chose
residences in expectation of the severity of a
future acute condition. If not related
to severity, we assume equivalent severity
distributions across groups
Number of Patients in Each Group by Severity
Type
severity type patient
group High Medium Low
Group 1 400
400 250 Group
2 400 400
250
50
Expected average estimated cure rates for these
groups Well, (.959428 - .946092)
.013336 doesn't appear to reveal much of
anything!
51
Now look at the antibiotic treatment rate in each
group 720/1050 .68571 in Group
1 520/1050 .4952381 in Group 2 These
differences also don't look very
informative. The IV change in the cure rates
resulting from a one unit increase in the drug
treatment rate equals This estimate is
the average difference in the antibiotic cure
rate for the marginal or in this example the
Medium severity patients.
52
Remember the actual unknown cure rates for
each group by treatment are
severity type treatment High
Medium Low antibiotics .95
.97 .98 watchful waiting
.80 .90 .98
.07 This estimate was found using
only measured treatment rates and outcome
rates across groups that are defined by the
instruments. Which of the estimates above is
the most important for policy-makers
wondering about over/underutilization of a
treatment?
53
IV Brass Tacks Where do instruments come
from? ? Theory on what motivated
choices, not theory on how choices can
be motivated. ? Observed differences in
-- guideline implementation
(timing/interpretation) -- product
approval rules across payers --
reimbursement differences across
payers/geography -- area provider
treatment signatures -- geographic
access to relevant providers --
provider market structure/competition ?
Generally, Natural Experiments (Angrist and
Krueger, 2001)
54
General IV Estimation Model Treatment
Choice Equation (1st stage) Outcome Equation
(2nd stage) Yi 1 if health outcome
occurs, 0 otherwise Xi measured patient
clinical characteristics Ti 1 if
patient received treatment, 0 otherwise
predicted treatment from 1st stage Zi
a set of binary variables grouping patients
based on values of instrumental
variables (from W) and Li unmeasured
confounding variables assumed related
to both Y and T but not Z. The only variation
in T used to estimate a1 comes from Z.
55
Define the IV estimate of a1 as .
? It can be shown that the expected value of
is Yields an average estimate of
the treatment effect for the set of patients
whose treatment choices were dependent on
their value of Z.
56
? The estimate of a1 can only be definitively
generalized to the patients whose treatment
choices were affected by Z (Angrist,
Imbens, Rubin 1996). ? F-test of whether the
parameters within c3 are simultaneously
equal to zero provides a test of the first
instrumental variable criterion Finding
measured variables or instruments (Z) that
a. are related to the possibility of a
patient receiving treatment
(cov(T,Z) ? 0)
57
? Model can be estimated via --
Two-Stage Least Squares (2SLS) PROC
SYSLIN in SAS. -- Bivariate
Probit BIPROBIT function in STATA.
-- Two-Stage Replacement (e.g. Beenstock
Rahav, 2002). ? 2SLS offers
consistent estimates that are asymptotically
normal with the fewest assumptions (Angrist
2001). -- essentially regressing
group-level outcome rate changes on
group-level treatment rate changes.
58
How many groups? ? Z can be specified as
continuous variables, but results are
then conditional on this assumption and are less
interpretable. ? Creating many
groups from an instrument (more binary
variables in Z) uses more information and yields
a weighted average of many two-group
comparisons, e.g. -- low/high
groups using the median of the instrument
VS -- low/med low/med
high/high groups using the
quartiles of the instrument. ? Too many
groups may introduce bias. ? Best to report
estimates for several grouping strategies.
59
Effect of Dialysis Center Profit-Status on
Patient Survival An Instrumental Variables
Approach Brooks, Irwin, Pendergast,
Chrischilles, Flanigan, Hunsicker
60
Introduction In a meta-analysis of observational
studies, Devereaux et al (1) found that patient
survival at for-profit dialysis centers was
poorer than non-profit centers. Objective Compa
re estimates of the effect of dialysis center
profit status on patient survival using
risk-adjustment and IV estimation.
61

Sample N 101,669 incident ESRD
patients from United States Renal
Data System (USRDS) from 1996-1999 that
-- were between 67 and 100 years old at dialysis
initiation -- had hemodialysis as
initial modality -- obtained dialysis
in a non-government dialysis facility
-- had complete information on all model
variables -- zip codes linked to 1990
census data. Key Variable Definitions
Outcome one-year survival after dialysis
initiation 1, 0
otherwise. Treatment Setting patient
initiated dialysis in a for-profit
dialysis center 1, 0
otherwise.
62
Instrumental Variable
Strategy Followed McClellan et al. (1994) and
grouped patients based on Differential
Distance (DD) to various hospital
classifications DD (DFP - DNP)
where DFP distance from patient residence
to the nearest for profit dialysis
center and DNP distance from patient
residence to the nearest non-profit
dialysis center. Assessed whether IV
estimates were robust to the number of patient
groups defined using differential distance.
63
Percent Initial For-Profit and Number of
Comorbidities by Patient Differential Distance
64
Table 1 Attributes of Dialysis Patient Groups,
1996-1999
Differential
Distance (DD) Patient Treatment Setting
Patient Closer to
Characteristics For-Profit Non-Profit
For-Profit Non-Profit For Profit
100 0
92.8 48.1 White
70.6 73.3
74.9 67.9 Black
23.5 20.0
19.8 25.1 Cardiac Failure
42.6 44.2 43.1
43.1 Diabetes
45.1 41.3 44.8
43.1 CerebroVasc Dis 11.9
12.5 12.0
12.1 Isch Heart Disc 32.1
36.1 33.5
33.1 AMI 11.6
13.2 12.2
12.0 Reside ina High Hlth State 61.2
47.4 61.4
52.9 Med Hlth State 16.4
40.3 13.7 33.3 Low
Hlth State 22.4 12.3
24.9 13.9 Number
73,480 30,678
52,443 51,715 a.Subramanian, S, Kawacki
I, et al. (2001). Does the state you live in
make a difference? A multilevel analysis of
self-rated health in the US. Social Science
Medicine 53(1) 9-19. , statistically
significant at the .01 and .05 levels,
respectively
65
(No Transcript)
66
Table 2 F-Statistics Testing Factors in Center
Choice Model are Related to the Use
of For-Profit Dialysis Facilities,
1996-1999. Factora
Partial F-Statistics .
Differential Distance (instrument)
2150.53 Year
59.53 Gender

5.81 Age

7.29 Race
6.26 Comorbidity

8.55 Previous Healthcare Use
2.63 State of Residence
212.00 Distance
to Nearest Center
75.41 Area Socioeconomic Status
21.83
a. specified using binary variables reflecting
differences in respective characteristic.
Differential distance was used to group patients
into 20 separate groups. , statistically
significant at the .01 and .05 levels,
respectively
67
Table 3 2SLS/IV and Ordinary Least Squares (OLS)
Estimates of the Effect of Initial
For-Profit Initial Dialysis Provider Relative to
a Non-Profit Provider on 1-Year
Patient Survival Estimation Model
Number of Instrument and
Specification Groups Specified
Estimate (P-value) OLS no covariates
na -0.0031c
(0.3450) OLS Devereaux covariatesa
na -0.0122c (lt.0001)
OLS Devereuax covariates plusb na
-0.0071c (0.0511) 2SLS/IVb
2
0.0009 (0.9264) 2SLS/IVb
5
0.0025 (0.7373) 2SLS/IVb
10
-0.00004 (0.9953) 2SLS/IVb
20
-0.0002 (0.9823) 2SLS/IVb
40
0.0006 (0.9349) a. Factors consistently
controlled for in the studies within the
Devereaux meta-analysis age, gender, race,
comorbidities. b. Factors consistently controlled
for in the studies within the Devereaux
meta-analysis age, gender, race,
comorbidities, plus dialysis year, state of
residence, previous healthcare utilization,
provider access (distance to nearest dialysis
center), socioeconomic status (patient zip
percent rural, percent poverty, and per
capita income). c. Logistic regression estimates
were consistent in both magnitude and statistical
significance. OLS estimates were reported
because their interpretation is more consistent
with IV estimates.
68
Summary The foundation of IV estimation is
theory that suggests instruments what
factors motivated treatment choices. Ability
to generalize is limited, but IV estimates offer
a more natural estimate of the effects of rate
changes than RCT estimates. Estimates can
vary by sample and instrument used. Estimates
are conditional on the truth (and acceptance)
of a known identification restriction. The
source of the treatment variation is known.
The relationship between this variation source
and unmeasured confounders can be debated.
69
References Angrist JD, 2001. Estimation of
Limited Dependent Variable Models with Dummy
Endogenous Regressors Simple Strategies for
Empirical Practice. Journal of Business
Economic Statistics. 19(1)2-16 Angrist, JD,
Imbens GW, Rubin, DB. 1996. Identification of
Causal Effects Using Instrumental Variables.
Journal of the American Statistical Association.
91444-454. Angrist JD, Krueger AB. 2001.
Instrumental Variables and the Search for
Identification From Supply and Demand to Natural
Experiments. Journal of Economic Perspectives.
15(4) 69-85. Beenstock M, Rahav G. Testing
Gateway Theory do cigarette prices affect
illicit drug use? Journal of Health Economics
200221679-98. Brooks JM, Chrischilles E, Scott
S, Chen-Hardee S. 2003. Was Lumpectomy
Underutilized for Early Stage Breast Cancer?
Instrumental Variables Evidence for Stage II
Patients from Iowa. Health Services Research,
38(6)1385-1402. Brooks JM, McClellan M, Wong H.
2000. The Marginal Benefits of Invasive Treatment
for Acute Myocardial Infarction Does Insurance
Coverage Matter? Inquiry, 37(1)75-90. Devereaux,
P., H. Schunemann, et al. (2002). Comparison of
Mortality Between Private For-Profit and Private
Not-For-Profit Hemodialysis Centers. A
Systematic Review and Meta-analysis. JAMA
288(19) 2449-2457. Imbens GW, Angrist, JD.
1994. Identification and Estimation of Local
Average Treatment Effects, Econometrica.
62(2)467-475. McClellan M, McNeil BJ, Newhouse
JP. 1994. Does More Intensive Treatment of Acute
Myocardial Infarction in the Elderly Reduce
Mortality Analysis Using Instrumental
Variables", Journal of the American Medical
Association. 272859-866. McClellan M, Newhouse
JP. 1993. The Marginal Benefits of Medical
Treatment Intensity. Cambridge, Mass National
Bureau of Economic Research Working
Paper. McClellan M, Newhouse JP. 1997. The
Marginal Cost-Effectiveness of Medical Technology
- a Panel Instrumental Variables Approach,
Journal of Econometrics. 7739-64. Subramanian,
S, Kawacki I, et al. (2001). Does the state you
live in make a difference? A multilevel analysis
of self-rated health in the US. Social Science
Medicine 53(1) 9-19.