Econometrics with Observational Data: Research Design

1
Econometrics with Observational Data Research
Design

• Todd Wagner

2
Research Design

• Goal evaluate behaviors and identify causation
• Policy X caused effect Y
• Medication A resulted in B hospitalizations
• Unit of analysis can be individual or
  organizational

3
Research Methods
Random assignment?
Yes
Intent to Treat?
4
Research Methods
Random assignment?
Yes
Intent to Treat?
No
Yes
On Treatment
Basic RCT Analysis
5
On Treatment

• RCT comparing drug A to drug B
• Adherence for drugs
• A is 70
• B is 40
• What does a comparison of A versus B tell us?

6
Research Methods
Random assignment?
No
Yes
Intent to Treat?
Is there a control group?
7
Research Methods
Is there random assignment
Randomized Trial
Is there a control group
Quasi-experimental Design
Descriptive Study
8
Research Methods
Is there random assignment
Randomized Trial
Is there a control group
Quasi-experimental Design
Descriptive Study
9
Quasi-Experimental Designs

• Difference-in-differences
• Regression discontinuity
• Switching replications
• Non-equivalent dependent variables

Most common In health
10
Difference-in-Differences

• AKA DD, D in D, or Diff in Diff
• Differences across time and arms
• Usually two arms treatments, controls
• In theory can be used with 3 arms

11
Methods for Identifying Controls

• Inherent matching Find similar individuals not
  getting treatment to serve as controls (e.g.,
  twins)
• Statistical use statistical techniques to
  identify best comparison groups
• Location use other geographic sites, states or
  regions as controls

12
Unit of Analysis

• D in D works for different units of analysis
• Personpeople followed over time
• Site sites followed over time
• State states followed over time
• May need to make some analytical changes
  depending on unit of analysis

13
Diff in Diff example

• Gruber, Adams and Newhouse (1997)
• Tennessee increased Medicaid fees for primary
  care services (goal encourage office care
  decrease hospital-based ambulatory care)
• What is the effect of this policy change?

14
(No Transcript)
15
Research Designs

• Difference-in-differences
• Regression discontinuity
• Switching replications
• Nonequivalent dependent variables

16
Regression Discontinuity

• Participants are assigned to program or
  comparison groups solely on the basis of an
  observed measure (education test or means test)
• Appropriate when we wish to target a program or
  treatment to those who most need or deserve it

17
Regression Discontinuity

• Partial coverage (not everyone gets the
  treatment)
• Requires the selection mechanism to be fully
  known
• Selection mechanism must be consistently applied
  to all persons

18
RD Design Graphically
Test for significance
Source Urban Institute
Threshold MUST be known and consistently applied
19
Research Designs

• Difference-in-differences
• Regression discontinuity
• Switching replications
• Nonequivalent dependent variables

20
Switching Replications

• Has two groups and three waves of measurement
• AKA waitlist control group
• This design is sometimes used in randomized trials

21
Example from Pap Smear Study
100
treat
90
80
70
Immediate treatment
60
50
Cumulative Followed Up
40
30
20
delayed treatment
10
0
1
2
3
4
5
6
7
8
9
10
11
12
gt 12
Months since Initial Pap
Intervention
Control
22
Research Designs

• Difference-in-differences
• Regression discontinuity
• Switching replications
• Nonequivalent dependent variables

23
Non-Equivalent DVs

• Analyze dependent variable that should not be
  affected by the intervention
• Example Intervention is designed to affect
  quality of diabetes care, but could also see if
  intervention affected quality of asthma care

24
Notes on the Analysisof DD data
25
Analytical Methods

• Plot or graph unadjusted data
• Graduate to more complex models
• Address, if possible, model limitations

26
DD Raw Data

Baseline
1-Year Follow-Up Exp. Control
Exp Control
DD ----------------------------------------
-------------------------------------------------
Utilization Entry ( yes) 84.5
86.1 88.9 86.8
3.7 (36.2) (34.6)
(31.4) (33.9) No. of
visits (0-16) 3.69 3.84
3.73 3.67
0.21 (4.28) (4.36)
(4.00) (4.07)
--------------------------------------------------
----------------------------------------
Standard deviations in parentheses DD
(Expfollowup- Expbaseline)-(Controlfollowup-
Controlbaseline) unadjusted estimates
27
Diff n Diff Model

• Y a b1G b2T b3GT gX e
• Youtcome
• G group (0control, 1treatment)
• T time (0baseline, 1follow-up)
• X characteristics of person, place, etc.
• e error term

28
Program Effect
Outcome a b1G b2T b3GT gX e

• If b3 0 then the program has no effect
• Limited statistical power. Testing interactions
  increases risk of type 2 error.

29
Organizing the Dataset
------------------------------ avgcost sta3n
exp yr_d year --------------------------------
. 358 0 0 93 . 358
0 1 94 318.2305 402 1 0
93 323.2815 402 1 1 94 472.0291
405 1 0 93 480.1368 405 1 1
94 364.0456 436 0 0 93 398.9824
436 0 1 94 369.9669 437 0 0
93 346.4565 437 0 1 94 270.0007
438 0 0 93 322.2588 438 0 1
94 292.7632 442 1 0 93 .
442 1 1 94 475.6746 452 1 0
93 494.9601 452 1 1 94
Note Data Listed in Stata
30
Identification
Outcome a b1G b2T b3GT gX e

• How do you obtain an unbiased estimate of b3?
• For an unbiased estimate of GT, G must not be
  correlated with e that is, G must be exogenous

31
Identification
Outcome a b1G b2T b3GT gX e

• G may be endogenous
• Selection bias
• Selection bias is type of endogeneity
• Caused by non-random assignment
• Outcome and G (group) affect each other --
  causality runs both ways
• Impact b3 is biased

32
Example VA Residential Treatment
Wagner, T. H., Chen, S. (2005). An economic
evaluation of inpatient residential treatment
programs in the department of veterans affairs.
Med Care Res Rev, 62(2), 187-204.
33
Residential Treatment Programs

• RTPs provide mental health and substance use
  treatment
• RTPs were designed to
• treat eligible veterans in a less-intensive and
  more self-reliant setting.
• to provide cost-effective care that promotes
  independence and fosters responsibility.

34
Objectives

1. Did the RTPs save money?
2. Were savings a one-time event or do they
   continue to accrue?

35
Design Choice

• Selection mechanism is not observed cant use
  regression discontinuity
• We know who adopted RTP and when DD is feasible

36
Methods

• Built a longitudinal dataset for 1993-1999 for
  all VA medical centers
• Tracked approved RTP programs (N43)
• We merged data from the PTF and CDR to track
• Total MH inpatient days (PTF) and dollars (CDR)
• Total SA inpatient days (PTF) and dollars (CDR)

37
Outcomes

• Department-level costs
• Average cost per MH day
• Average cost per SA day
• Total MH/SA department costs
• Sensitivity analysis
• Outpatient MH/SA costs
• FTE

38
Multivariate models

• Fixed-effects models1
• DV Department-level costs
• Controlled for medical center size
• Inflation adjusted to 1999 using CPI
• Year dummies
• Wage index

1 Random effects were similar Hausman tests were
not significant. Fixed effects were more
conservative.
39
Results Mental Health

• Average cost savings of 81 per day (plt0.01).
• Savings do not appear to be increasing over time.

40
Mental Health Costs
41
Results Substance Abuse

• Average cost savings of 112 per day (plt0.01).
• Savings do not appear to be increasing over time.

42
Mental Health Costs
43
Sensitivity Analysis

• RTPs were associated with a slight decrease in
  the costs of outpatient psychiatry.
• RTPs were associated with a decrease in FTE

44
Limitations

• Not clear if RTPs could be better are they
  treating the right patient?
• Endogeneity of RTPs
• 1 and 2 year lags (medical centers with RTPs in
  1994 and 1995) are not associated with costs
• There does not appear to be self-selection in
  RTPs.

45
Any Questions?
46
Design References
Trochim, W. Research Methods Knowledge Database
http//www.socialresearchmethods.net/kb/ Rossi,
PH, and HE Freeman. Evaluation A systematic
approach. 5th ed. New York Sage, 1993.
47
Regression References

• Wm. Greene. Econometric Analysis.
• J Wooldridge. Econometric Analysis of Cross
  Section and Panel Data.

48
Youve Almost Made It

• June11th Mark Smith, Endogeneity
• TBA Todd Wagner Using Stata