EC3333 Lecture 6 Experiments

1
EC3333 Lecture 6 Experiments

• Review
• Randomized Experiments
• Natural/Quasi Experiments
• Differences-in-Differences
• Link with IVs
• LATE
• Regression Discontinuity

2
Review

• Under the GaussMarkov assumptions, OLS is
  consistent, unbiased, and efficient.
• Under heteroskedasticity or serial correlation,
  OLS is still consistent and unbiased, but
  inefficient (and the OLS formula for estimated
  standard errors is incorrect)
• When X is correlated with e, then OLS is
  inconsistent and biased.

3
Review (cont.)

• There are many possible reasons why X could be
  correlated with e?
• Omitted Variables
• Simultaneity
• Measurement Error

4
Review (cont.)

• Solutions to contemporaneous correlation must
  break the link between X and e.
• Instrumental Variables breaks this link ex post,
  isolating part of the variation observed in X
  that is known to be uncorrelated with e.
• Experiments break the link ex ante.

5
Estimating Treatment Means

• Typically we are interested in knowing the effect
  of one or more treatments.
• We want to know what outcomes are caused by the
  treatment/s.

6
Estimating Means (cont.)

• We really want to know the causal effect of the
  treatment T.
• We want to know what would happen on average to a
  person randomly chosen from the population if we
  gave him/her the treatment, as opposed to NOT
  giving him/her the treatment.

7
Estimating Means (cont.)

• Unfortunately, we do not get to observe the same
  individuals in each state.
• A naïve analyst might simply compare the outcomes
  for those observed in the treatment state to
  those not observed in the treatment state.
• In an observational study, we cannot DIRECTLY
  compare groups that receive the treatment to
  groups that do not.

8
Estimating Means (cont.)

• Whenever selection into a treatment is
  non-random, researchers must worry about
  unobserved heterogeneity among subjects.
• Some subjects have greater ability, motivation,
  resources, etc., that make them more likely to
  seek out and gain access to helpful treatments
  (and to avoid unhelpful ones).

9
Estimating Means (cont.)

• Treatments also tend to attract individuals who
  derive the most benefit from them.
• The selection bias is a special case of omitted
  variables bias.
• The goal of experiments is to break the selection
  bias.

10
Estimating Means (cont.)

• Breaking the selection bias requires the
  experimenter to intervene in the agents world in
  some way.
• The greater the intervention, the greater the
  control the economist possesses, and the more
  certain the economist is to eliminate selection
  biases.
• The greater the intervention, the greater the
  danger that the results will not generalize to a
  more authentic situation.

11
Estimating Causal Effects with Experimental Data
12
Randomized Experiments Basic Terminology

• The experimenter randomly divides the subjects
  into two groups, a treatment group and a control
  group.
• The treatment group receives the treatment (T
  1).
• The control group does not receive the treatment
  (T 0).
• Causal effect sometimes called treatment effect
• Randomization implies everyone has same
  probability of treatment

13
Why is Randomization Good?

• If T allocated at random then know that T is
  independent of all pre-treatment variables in
  whole wide world
• Implies there cannot be a problem of omitted
  variables, reverse causality etc
• On average, only reason for difference between
  treatment and control group is different receipt
  of treatment

14
An Experimental Design

• Bertrand/Mullainathan Are Emily and Greg More
  Employable Than Lakisha and Jamal, American
  Economic Review, 2004
• Create fake CVs and send replies to job adverts
• Allocate names at random to CVs some given
  black-sounding names, others white-sounding

15

• Outcome variable is call-back rates
• Interpretation not direct measure of racial
  discrimination, just effect of having a
  black-sounding name may have other
  connotations.
• But name uncorrelated by construction with other
  material on CV

16
Bertrand/MullainathanBasic Results
17
Bertrand/Mullainathan

• Different treatment effect for high and low
  quality CVs

18
Treatment EffectEstimating Means (cont.)

• We really want to know the causal effect of the
  treatment T.
• We want to know what would happen on average to a
  person randomly chosen from the population if we
  gave him/her the treatment, as opposed to NOT
  giving him/her the treatment.

19
Estimating Treatment Effects the Statistics
Course Approach

• Take mean of outcome variable in treatment group
• Take mean of outcome variable in control group
• Take difference between the two
• No problems but
• Does not generalize to where X is not binary
• Does not directly compute standard errors

20
Estimating Treatment Effects A Regression
Approach

• Run regression
• yiß0 ß1Tiei
• Class exercise asks you to prove that
• OLS estimate of intercept is consistent estimate
  of E(yT0)
• OLS estimate of intercept is consistent estimate
  of
• E(yT1) -E(yT0)
• Hence can read off estimate of treatment effect
  from coefficient on T
• Approach easily generalizes to where T is not
  binary
• Also gives estimate of standard error

21
The Uses of Other Regressors Check for
Randomization

• Randomization can go wrong
• Poor implementation of research design
• Bad luck
• If randomization done well then W should be
  independent of T this is testable
• Test for differences in W in treatment/control
  groups
• Probit model for T on W

22
The Uses of Other Regressors IIIImprove
Randomization

• Can also use W at stage of assigning treatment
• Can guarantee that in your sample T and W are
  independent instead of it being just
  probabiliistic
• This is what Bertrand/Mullainathan do when
  assigning names to CVs

23
Randomized Experiment (cont.)

• Biases in Randomized Experiments
• Non-Response Bias participants do not provide
  their data to the researchers
• Attrition Bias participants drop out of the
  study
• Sample Selection Bias individuals who agree to
  participate in a randomized study differ from the
  population of interest

24
Randomized Experiments (cont.)

• General Equilibrium Effects training programs
• If a few individuals randomly receive extra job
  training, their wages increase because (i) they
  are more productive and (ii) they have a
  competitive edge.
• If everyone receives the extra training, no one
  gains a competitive edge.

25
Natural Experiments

• In a laboratory experiment, experimental
  economists can exert great control over every
  aspect of the subjects environment.
• Greater control increases the artificiality of
  the experiment.
• At some point, economists wish to generalize from
  the subjects and environments studied to a real
  program in the population of interest.

26
Natural Experiments (cont.)

• Internal Validity the ability of the economist
  to attribute differences between the treatment
  and control groups to the treatment itself (X and
  ? are uncorrelated).
• External Validity the ability of the economist
  to generalize from the experiment to the setting
  and population of interest.

27
Natural Experiments (cont.)

• Often economists face a trade-off between
  internal and external validity.
• The more they break the natural connections
  between X and e, the more danger arises that the
  results will not generalize.
• Natural experiments often offer greater external
  validity (and are much cheaper!)

28
Differences-in-Differences
29
(No Transcript)
30
(No Transcript)
31
The Grand Experiment

• Water supplied to households by competing private
  companies
• Sometimes different companies supplied households
  in same street
• In south London two main companies
• Lambeth Company (water supply from Thames Ditton,
  22 miles upstream)
• Southwark and Vauxhall Company (water supply from
  Thames)

32
In 1853/54 cholera outbreak

• Death Rates per 10000 people by water company
• Lambeth 10
• Southwark and Vauxhall 150
• Might be water but perhaps other factors
• Snow compared death rates in 1849 epidemic
• Lambeth 150
• Southwark and Vauxhall 125
• In 1852 Lambeth Company had changed supply from
  Hungerford Bridge

33
What would be good estimate of effect of clean
water?
34
Natural/ Quasi Experiments

• Used to refer to situation that is not
  experimental but is as if it was
• Not a precise definition saying your data is a
  natural experiment makes it sound better
• Refers to case where variation in X is good
  variation (directly or indirectly via
  instrument)
• A Famous Example London, 1854

35
The Case of the Broad Street Pump

• Regular cholera epidemics in 19th century London
• Widely believed to be caused by bad air
• John Snow thought bad water was cause
• Experimental design would be to randomly give
  some people good water and some bad water
• Ethical Problems with this

36
Soho Outbreak August/September 1854

• People closest to Broad Street Pump most likely
  to die
• But breathe same air so does not resolve air vs.
  water hypothesis
• Nearby workhouse had own well and few deaths
• Nearby brewery had own well and no deaths
  (workers all drank beer)

37
Why is this a Natural experiment?

• Variation in water supply as if it had been
  randomly assigned other factors (air) held
  constant
• Can then estimate treatment effect using
  difference in means
• Or run regression of death on water source
  distance to pump, other factors
• Strongly suggests water the cause
• Woman died in Hampstead, niece in Islington

38
Whats that got to do with it?

• Aunt liked taste of water from Broad Street pump
• Had it delivered every day
• Niece had visited her
• Investigation of well found contamination by
  sewer
• This is non-experimental data but analysed in a
  way that makes a very powerful case no theory
  either

39
This is basic idea of Differences-in-Differences

• Have already seen idea of using differences to
  estimate causal effects
• Treatment/control groups in experimental data
• Twins data to deal with ability bias
• Often would like to find treatment and
  control group who can be assumed to be similar
  in every way except receipt of treatment
• This may be very difficult to do

40
A Weaker Assumption is..

• Assume that, in absence of treatment, difference
  between treatment and control group is
  constant over time
• With this assumption can use observations on
  treatment and control group pre- and
  post-treatment to estimate causal effect
• Idea
• Difference pre-treatment is normal difference
• Difference pre-treatment is normal difference
  causal effect
• Difference-in-difference is causal effect

41
A Graphical Representation
42
What is D-in-D estimate?

• Standard differences estimator is AB
• But normal difference estimated as CB
• Hence D-in-D estimate is AC
• Note assumes trends in outcome variables the
  same for treatment and control groups
• This is not testable with two periods but its
  testable with more

43
(No Transcript)
44
Differences-in-Differences (cont.)

• where DTreatment 1 if the observation is of a
  subject assigned to the treatment group, either
  before or after the treatment is received
• and DAfter 1 if the observation is of a subject
  in either group after the treatment has been
  received by the treatment group

45
Differences-in-Differences (cont.)

• a1 captures the underlying difference between the
  treatment and control groups.
• a2 captures the underlying difference between
  the two time periods.
• a3 captures the effect of the treatment.

46
Differences-in-Differences (cont.)
47
Differences-in-Differences (cont.)

• One method for estimating a3 is to calculate the
  means for each group at each time and subtract
  twice.

48

• The Great Weakness of Diffs-in-Diffs
• Some other difference may arise between the
  Treatment and Control groups at the same time
  that the Treatment occurs.
• Example suppose at the same time NJ increased
  the minimum wage, PA introduced a new food
  labeling law that reduced the demand for fast
  food.

49

• This is simply differences estimator applied to
  the difference
• To implement this need to have repeat
  observations on the same individuals
• May not have this individuals observed pre- and
  post-treatment may be different
• What can we do in this case?

50
Differential Trends in Treatment and Control
Groups

• Key assumption underlying validity of D-in-D
  estimate is that differences between treatment
  and control group are constant over time
• Cannot test this with only two periods
• But can test with more than two periods

51
An Example Project STAR

• Allocation of students to classes is random
  within schools
• But small number of classes per school
• This leads to following relationship between
  probability of treatment and number of kids in
  school

52
(No Transcript)
53
(No Transcript)
54
What treatment effect to estimate?

• Would like to estimate causal effect for everyone
  this is not possible
• Can only hope to estimate some average
• Average treatment effect

55
Units of Measurement

• Causal effect measured in units of experiment
  not very helpful
• Often want to convert causal effects to more
  meaningful units e.g. in Project STAR what is
  effect of reducing class size by one child

56
Simple estimator of this would be

• where S is class size
• Takes the treatment effect on outcome variable
  and divides by treatment effect on class size
• Not hard to compute but how to get standard
  error?

57
IV Can Do the Job

• Cant run regression of y on S S influenced by
  factors other than treatment status
• But X is
• Correlated with S
• Uncorrelated with unobserved stuff (because of
  randomization)
• Hence X can be used as an instrument for S
• IV estimator has form (just-identified case)

58
The Wald Estimator

• This will give estimate of standard error of
  treatment effect
• Where instrument is binary and no other
  regressors included the IV estimate of slope
  coefficient can be shown to be

59
Partial Compliance

• So far
• in control group implies no treatment
• In treatment group implies get treatment
• Often things are not as clean as this
• Treatment is an opportunity
• Close substitutes available to those in control
  group
• Implementation not perfect e.g. pushy parents

60
Some Terminology

• Z denotes whether in control or treatment group
  intention-to-treat
• X denotes whether actually get treatment
• With perfect compliance
• Pr(X1Z1)1
• Pr(X1Z0)0
• With imperfect compliance
• 1gtPr(X1Z1)gtPr(X1Z0)gt0

61
What Do We Want to Estimate?

• Intention-to-Treat
• ITTE(yZ1)-E(yZ0)
• This can be estimated in usual way
• Treatment Effect on Treated

62
Estimating TOT

• Cant use simple regression of y on Z
• But should recognize TOT as Wald estimator
• Can estimated by regressing y on X using Z as
  instrument
• Relationship between TOT and ITT

63
Angrist/Imbens Monotonicity Assumption

• Assume that everyone moved in same direction by
  treatment monotonicity assumption
• Then can show that IV is average of treatment
  effect for those whose behaviour changed by being
  in treatment group
• They call this the Local Average Treatment Effect
  (LATE)

64
Spill-overs/Externalities/General Equilibrium
Effects

• Have assumed that treatment only affects outcome
  for person for receives it
• Many situations in which this is not true
• E.g. externalities, spill-overs, effects on
  market prices

65
Problems with Experiments

• Expense
• Ethical Issues
• Threats to Internal Validity
• Failure to follow experiment
• Experimental effects (Hawthorne effects)
• Threats to External Validity
• Non-representative programme
• Non-representative sample
• Scale effects

66
Conclusions on Experiments

• Are gold standard of empirical research
• Are becoming more common
• Not enough of them to keep us busy
• Study of non-experimental data can deliver useful
  knowledge
• Some issues similar, others different

67
Summary

• Econometrics very easy if all data comes from
  randomized controlled experiment
• Just need to collect data on treatment/control
  and outcome variables
• Just need to compare means of outcomes of
  treatment and control groups
• Is data on other variables of any use at all?
• Not necessary but useful