Quasi-Experiments - PowerPoint PPT Presentation

Loading...

PPT – Quasi-Experiments PowerPoint presentation | free to view - id: e0d31-NDhiZ



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

Quasi-Experiments

Description:

Researcher didn't control assignment. Groups may be different. ... In a true experiment, the researcher performs the random assignment ... – PowerPoint PPT presentation

Number of Views:175
Avg rating:3.0/5.0
Slides: 47
Provided by: willia493
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Quasi-Experiments


1
Quasi-Experiments
2
The Basic Nonequivalent Groups Design (NEGD)
N O X O N O O
  • Key Feature Nonequivalent assignment

3
What Does Nonequivalent Mean?
  • Assignment is nonrandom.
  • Researcher didnt control assignment.
  • Groups may be different.
  • Group differences may affect outcomes.

4
Equivalence
  • Equivalent groups are not necessarily identical
    on any pre-test measure.
  • Merely implies that if the random assignment
    procedure was repeated, the groups would tend
    toward equivalence.

5
Non-Equivalence
  • Non-equivalent groups do not necessarily differ
    on any pre-test measure.
  • Merely implies that If the same non-random
    assignment procedure was repeated, the groups
    would tend to toward non-equivalence.
  • If assignment to groups was based partly on
    income, then groups would tend to have different
    expected mean levels of income but any two
    groups you picked might well be similar in income
    levels.

6
The Point
  • Equivalence or non-equivalence is defined by the
    selection procedure.
  • Even if the difference in pre-test means across
    groups is small, this does not imply that the
    groups are equivalent.
  • Small differences can introduce big threats.

7
Quasi- vs. Natural vs. Experiment
  • In a true experiment, the researcher performs the
    random assignment
  • Can be in a lab or the field
  • In a natural experiment, someone else assigns
    through a random process.
  • In a quasi-experiment, assignment is not random,
    introducing selection threats.
  • Much stronger if the selection is not done by the
    cases themselves (exogenous sorting).

8
What is a Natural Experiment
  • Strict Definition
  • Some truly natural process, such as rainfall or
    weather patterns, assigns IV.
  • Definition we all use in our own work
  • Some exogenous process, rather than our cases,
    ourselves, or a causal process relevant to our
    theory, assigns IV.

9
Genres of Natural Experiments
  • The natural border or natural disaster
  • The Rule Change
  • House seniority system (Crooks and Hibbing)
  • GAVEL amendment in Colorado
  • Connecticut speeding law
  • New Zealand electoral reform
  • Propositions
  • Relatively easy to spot, hard to defend
  • Jared Diamonds islands
  • Dan Posners rivers
  • Caroline Hoxbys streams
  • Settler mortality (Acemoglu, Johnson, and
    Robinson)
  • Hurricane Katrina
  • Strength is that nature doesnt care about your
    cases or IV

10
Genres of Natural Experiments
  • The Court Decision
  • The Lottery
  • Roe V. Wade for Levitt and Donohue
  • Iowa item veto decision
  • Strength is that court is not a blatant political
    actor responding to societal shifts or societal
    pressures
  • James Fowlers use of Canadian bill introduction
    privilege
  • US House Clerk conducts a randomization of the
    order in which members choose office
  • Strength is true randomness in first step, but
    human action in 2nd

11
Genres of Natural Experiments
  • Staged Implementation
  • The Threshold
  • Mail ballot assignment in precincts with lt250
    voters
  • Need to make the threshold unrelated to DV, or
    else use Trochim-style regression discontinuity
  • Two-step reapportionment revolution in the United
    States
  • Lots of program evaluations in development
  • Helps to rule out history and maturation threats

12
What Makes a Convincing Natural Experiment?
  • You can show that the process of selection was
    not related to characteristics of the cases that
    are relevant to your DV
  • In a cross-sectional experiment, demonstrate that
    the two groups are quite similar
  • In a time-series experiment, demonstrate that
    little else changed when the treatment took
    place.
  • In a word, show equivalence

13
Any purported causal test of needs to take into
consideration all of the two-group threats to
validity.
R X O R O
Can be a valid causal test.
N X O N O
Fully exposed to threats.
14
NEGD Design has Multiple Groups AND Multiple
Measures
N O X O N O O
This helps rule out (or at least recognize)
threats.
15
Pre-Tests v. Covariates
N O X O N O O
Pre- Post-Test Design Observations are tests you
administer.
N O1 X O2 N O1 O2
Proxy Pre-Test Design First observations are
covariates on which you collect data.
16
Problems of Internal Validity in NEGDs
17
Internal Validity
  • N O X O
  • N O O

All designs suffer from threats to validity. In
addition to all the single group
threats, quasi-experiments are particularly
likely to suffer from multi-group threats.

Selection-history Selection-maturation Selection-t
esting Selection-instrumentation Selection-regress
ion Selection-mortality
18
The Bivariate Distribution
19
The Bivariate Distribution
Program Group has a 5-point pretest advantage.
20
The Bivariate Distribution
Program group scores 15-points higher on Posttest.
Program group has a 5-point pretest advantage,
21
Graph of Means
22
Possible Outcome 1
  • Possible local event
  • Possible PG initially higher
  • Unlikely no change in CG
  • Possible scale effects
  • Unlikely expect change in CG
  • Possible PG loses low scorers

Selection-history Selection-maturation Selection-t
esting Selection-instrumentation Selection-regress
ion Selection-mortality
23
Possible Outcome 2
  • Likely PG initially higher
  • Likely PG initially higher
  • Possible
  • Possible
  • Unlikely expect change in CG
  • Possible both lose low scorers

Selection-history Selection-maturation Selection-t
esting Selection-instrumentation Selection-regress
ion Selection-mortality
24
Possible Outcome 3
Selection-history Selection-maturation Selection-t
esting Selection-instrumentation Selection-regress
ion Selection-mortality
  • Possible local event
  • Unlikely no change in CG
  • Unlikely no change in CG
  • Possible scale effects
  • Likely
  • Possible PG loses high scorers

25
Possible Outcome 4
  • Possible local event
  • Unlikely no change in CG
  • Unlikely no change in CG
  • Possible scale effects
  • Very Likely
  • Possible PG loses low scorers

Selection-history Selection-maturation Selection-t
esting Selection-instrumentation Selection-regress
ion Selection-mortality
26
Possible Outcome 5
  • And you should be so lucky

Selection-history Selection-maturation Selection-t
esting Selection-instrumentation Selection-regress
ion Selection-mortality
27
Analysis Requirements
N O X O N O O
  • Pre-post (or covariates)
  • Two-group
  • Treatment-control (dummy 0, 1)

28
Analysis of Covariance (ANCOVA)
yi ?0 ?1Xi ?2Zi ei
where
  • yi outcome score for the ith unit
  • ?0 coefficient for the intercept
  • ?1 pretest coefficient
  • ?2 mean difference for treatment
  • Xi covariate
  • Zi dummy variable for treatment(0 control, 1
    treatment)
  • ei residual for the ith unit

29
The Bivariate Distribution
Program group scores 15-points higher on Posttest.
Program group has a 5-point pretest Advantage.
30
The Bivariate Distribution
Slope is B1
Vertical Distance is Mean Treatment Effect, or B2
31
Why Add Covariates to Analysis?
  • ANCOVA can include more than one pretest or
    control variable.
  • Additional pretests further adjust for initial
    group differences.
  • Ideally, in the absence of any treatment effect,
    the covariates would perfectly predict the
    posttest.
  • Additional covariates will often improve the
    accuracy of the estimate of the treatment effect.

32
Irrelevant Covariates
  • Adding pretests that are completely unrelated to
    the posttest, however, actually decreases
    precision.
  • Irrelevant covariates contribute nothing to the
    analysis, but subtract a degree of freedom from
    the error term.
  • This reduces the efficiency of the estimate.

33
Omitted Covariates
  • Covariates that are related to the posttest but
    not to the treatment can be ignored without
    biasing the estimate of the treatment effect.
  • Covariates that are related to the posttest and
    the treatment but that are omitted will bias the
    estimate of the treatment effect.
  • We can safely omit control variables even if they
    are highly correlated with the posttest as long
    as they do not correlate with the treatment.

34
Omitted Variables Bias
  • Omitted (relevant) covariates that are positively
    correlated with the treatment will lead us to
    overestimate the treatment effect.
  • Omitted (relevant) covariates that are negatively
    correlated with the treatment will lead us to
    underestimate the treatment effect.

35
Bottom Line
  • We should always try to include omitted relevant
    covariates, except
  • When the omitted covariate is itself a
    consequence of the treatment.
  • If cannot include a relevant covariate, we can at
    least predict the direction if not magnitude of
    the likely bias.

36
ButWhat about measurement error?
  • With multiple covariates, measurement error does
    not always lead to a pseudo-effect.
  • As measurement error in any single variable
    increases, it becomes as if the variable is not
    included in the ANCOVA.
  • This then mimics an omitted variables problem,
    and the direction of bias depends upon the
    relationship between the noisy covariate and
    the treatment.

37
Other Quasi-Experimental Designs
38
Separate Pre-Post Samples
N1 O N1 X O N2 O N2 O
  • Groups with the same subscript come from the same
    context.
  • Here, N1 might be people who were in the program
    at Agency 1 last year, with those in N2 at Agency
    2 last year.
  • This is like having a proxy pretest on a
    different group.

39
Separate Pre-Post Samples
R1 O R1 X O R2 O R2 O
N
N
  • Take random samples at two times of people at two
    nonequivalent agencies.
  • Useful when you routinely measure with surveys.
  • You can assume that the pre and post samples are
    equivalent, but the two agencies may not be.

40
Double-Pretest Design
N O O X O N O O O
  • Strong in internal validity
  • Helps address selection-maturation

41
Switching Replications
N O X O O N O O X O
  • Strong design for both internal and external
    validity
  • Strong against social threats to internal
    validity
  • Strong ethically

42
Nonequivalent Dependent Variables Design (NEDV)
N O1 X O1 O2 O2
  • The variables have to be similar enough that they
    are affected the same way by all threats.
  • The program has to target one variable and not
    the other.
  • In simple form, weak internal validity.

43
NEDV Example
  • Only works if we can assume that geometry scores
    show what would have happened to algebra if
    untreated.
  • The variable is the control.
  • Note that there is no control group here.

44
NEDV Pattern Matching
  • Have many outcome variables.
  • Have theory that tells how affected (from most to
    least) each variable will be by the program.
  • Match observed gains with predicted ones.
  • With pattern, NEDV can be extremely powerful.

45
NEDV Pattern Matching
  • A ladder graph.

r .997
46
NEDV Lake and OMahony 2006
Hypothesis As territory declines in value in
20th century (measured by average state size),
wars fought over territory should decline
in frequency. There should be no pattern in
other Issues.
About PowerShow.com