# Quasi-Experiments - PowerPoint PPT Presentation

PPT – Quasi-Experiments PowerPoint presentation | free to view - id: e0d31-NDhiZ

The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
Title:

## Quasi-Experiments

Description:

### Researcher didn't control assignment. Groups may be different. ... In a true experiment, the researcher performs the random assignment ... – PowerPoint PPT presentation

Number of Views:175
Avg rating:3.0/5.0
Slides: 47
Provided by: willia493
Category:
Tags:
Transcript and Presenter's Notes

Title: Quasi-Experiments

1
Quasi-Experiments
2
The Basic Nonequivalent Groups Design (NEGD)
N O X O N O O
• Key Feature Nonequivalent assignment

3
What Does Nonequivalent Mean?
• Assignment is nonrandom.
• Researcher didnt control assignment.
• Groups may be different.
• Group differences may affect outcomes.

4
Equivalence
• Equivalent groups are not necessarily identical
on any pre-test measure.
• Merely implies that if the random assignment
procedure was repeated, the groups would tend
toward equivalence.

5
Non-Equivalence
• Non-equivalent groups do not necessarily differ
on any pre-test measure.
• Merely implies that If the same non-random
assignment procedure was repeated, the groups
would tend to toward non-equivalence.
• If assignment to groups was based partly on
income, then groups would tend to have different
expected mean levels of income but any two
groups you picked might well be similar in income
levels.

6
The Point
• Equivalence or non-equivalence is defined by the
selection procedure.
• Even if the difference in pre-test means across
groups is small, this does not imply that the
groups are equivalent.
• Small differences can introduce big threats.

7
Quasi- vs. Natural vs. Experiment
• In a true experiment, the researcher performs the
random assignment
• Can be in a lab or the field
• In a natural experiment, someone else assigns
through a random process.
• In a quasi-experiment, assignment is not random,
introducing selection threats.
• Much stronger if the selection is not done by the
cases themselves (exogenous sorting).

8
What is a Natural Experiment
• Strict Definition
• Some truly natural process, such as rainfall or
weather patterns, assigns IV.
• Definition we all use in our own work
• Some exogenous process, rather than our cases,
ourselves, or a causal process relevant to our
theory, assigns IV.

9
Genres of Natural Experiments
• The natural border or natural disaster
• The Rule Change
• House seniority system (Crooks and Hibbing)
• Connecticut speeding law
• New Zealand electoral reform
• Propositions
• Relatively easy to spot, hard to defend
• Jared Diamonds islands
• Dan Posners rivers
• Caroline Hoxbys streams
• Settler mortality (Acemoglu, Johnson, and
Robinson)
• Hurricane Katrina
• Strength is that nature doesnt care about your
cases or IV

10
Genres of Natural Experiments
• The Court Decision
• The Lottery
• Roe V. Wade for Levitt and Donohue
• Iowa item veto decision
• Strength is that court is not a blatant political
actor responding to societal shifts or societal
pressures
• James Fowlers use of Canadian bill introduction
privilege
• US House Clerk conducts a randomization of the
order in which members choose office
• Strength is true randomness in first step, but
human action in 2nd

11
Genres of Natural Experiments
• Staged Implementation
• The Threshold
• Mail ballot assignment in precincts with lt250
voters
• Need to make the threshold unrelated to DV, or
else use Trochim-style regression discontinuity
• Two-step reapportionment revolution in the United
States
• Lots of program evaluations in development
• Helps to rule out history and maturation threats

12
What Makes a Convincing Natural Experiment?
• You can show that the process of selection was
not related to characteristics of the cases that
• In a cross-sectional experiment, demonstrate that
the two groups are quite similar
• In a time-series experiment, demonstrate that
little else changed when the treatment took
place.
• In a word, show equivalence

13
Any purported causal test of needs to take into
consideration all of the two-group threats to
validity.
R X O R O
Can be a valid causal test.
N X O N O
Fully exposed to threats.
14
NEGD Design has Multiple Groups AND Multiple
Measures
N O X O N O O
This helps rule out (or at least recognize)
threats.
15
Pre-Tests v. Covariates
N O X O N O O
Pre- Post-Test Design Observations are tests you
N O1 X O2 N O1 O2
Proxy Pre-Test Design First observations are
covariates on which you collect data.
16
Problems of Internal Validity in NEGDs
17
Internal Validity
• N O X O
• N O O

All designs suffer from threats to validity. In
addition to all the single group
threats, quasi-experiments are particularly
likely to suffer from multi-group threats.

Selection-history Selection-maturation Selection-t
esting Selection-instrumentation Selection-regress
ion Selection-mortality
18
The Bivariate Distribution
19
The Bivariate Distribution
Program Group has a 5-point pretest advantage.
20
The Bivariate Distribution
Program group scores 15-points higher on Posttest.
Program group has a 5-point pretest advantage,
21
Graph of Means
22
Possible Outcome 1
• Possible local event
• Possible PG initially higher
• Unlikely no change in CG
• Possible scale effects
• Unlikely expect change in CG
• Possible PG loses low scorers

Selection-history Selection-maturation Selection-t
esting Selection-instrumentation Selection-regress
ion Selection-mortality
23
Possible Outcome 2
• Likely PG initially higher
• Likely PG initially higher
• Possible
• Possible
• Unlikely expect change in CG
• Possible both lose low scorers

Selection-history Selection-maturation Selection-t
esting Selection-instrumentation Selection-regress
ion Selection-mortality
24
Possible Outcome 3
Selection-history Selection-maturation Selection-t
esting Selection-instrumentation Selection-regress
ion Selection-mortality
• Possible local event
• Unlikely no change in CG
• Unlikely no change in CG
• Possible scale effects
• Likely
• Possible PG loses high scorers

25
Possible Outcome 4
• Possible local event
• Unlikely no change in CG
• Unlikely no change in CG
• Possible scale effects
• Very Likely
• Possible PG loses low scorers

Selection-history Selection-maturation Selection-t
esting Selection-instrumentation Selection-regress
ion Selection-mortality
26
Possible Outcome 5
• And you should be so lucky

Selection-history Selection-maturation Selection-t
esting Selection-instrumentation Selection-regress
ion Selection-mortality
27
Analysis Requirements
N O X O N O O
• Pre-post (or covariates)
• Two-group
• Treatment-control (dummy 0, 1)

28
Analysis of Covariance (ANCOVA)
yi ?0 ?1Xi ?2Zi ei
where
• yi outcome score for the ith unit
• ?0 coefficient for the intercept
• ?1 pretest coefficient
• ?2 mean difference for treatment
• Xi covariate
• Zi dummy variable for treatment(0 control, 1
treatment)
• ei residual for the ith unit

29
The Bivariate Distribution
Program group scores 15-points higher on Posttest.
Program group has a 5-point pretest Advantage.
30
The Bivariate Distribution
Slope is B1
Vertical Distance is Mean Treatment Effect, or B2
31
• ANCOVA can include more than one pretest or
control variable.
group differences.
• Ideally, in the absence of any treatment effect,
the covariates would perfectly predict the
posttest.
• Additional covariates will often improve the
accuracy of the estimate of the treatment effect.

32
Irrelevant Covariates
• Adding pretests that are completely unrelated to
the posttest, however, actually decreases
precision.
• Irrelevant covariates contribute nothing to the
analysis, but subtract a degree of freedom from
the error term.
• This reduces the efficiency of the estimate.

33
Omitted Covariates
• Covariates that are related to the posttest but
not to the treatment can be ignored without
biasing the estimate of the treatment effect.
• Covariates that are related to the posttest and
the treatment but that are omitted will bias the
estimate of the treatment effect.
• We can safely omit control variables even if they
are highly correlated with the posttest as long
as they do not correlate with the treatment.

34
Omitted Variables Bias
• Omitted (relevant) covariates that are positively
correlated with the treatment will lead us to
overestimate the treatment effect.
• Omitted (relevant) covariates that are negatively
correlated with the treatment will lead us to
underestimate the treatment effect.

35
Bottom Line
• We should always try to include omitted relevant
covariates, except
• When the omitted covariate is itself a
consequence of the treatment.
• If cannot include a relevant covariate, we can at
least predict the direction if not magnitude of
the likely bias.

36
• With multiple covariates, measurement error does
not always lead to a pseudo-effect.
• As measurement error in any single variable
increases, it becomes as if the variable is not
included in the ANCOVA.
• This then mimics an omitted variables problem,
and the direction of bias depends upon the
relationship between the noisy covariate and
the treatment.

37
Other Quasi-Experimental Designs
38
Separate Pre-Post Samples
N1 O N1 X O N2 O N2 O
• Groups with the same subscript come from the same
context.
• Here, N1 might be people who were in the program
at Agency 1 last year, with those in N2 at Agency
2 last year.
• This is like having a proxy pretest on a
different group.

39
Separate Pre-Post Samples
R1 O R1 X O R2 O R2 O
N
N
• Take random samples at two times of people at two
nonequivalent agencies.
• Useful when you routinely measure with surveys.
• You can assume that the pre and post samples are
equivalent, but the two agencies may not be.

40
Double-Pretest Design
N O O X O N O O O
• Strong in internal validity

41
Switching Replications
N O X O O N O O X O
• Strong design for both internal and external
validity
• Strong against social threats to internal
validity
• Strong ethically

42
Nonequivalent Dependent Variables Design (NEDV)
N O1 X O1 O2 O2
• The variables have to be similar enough that they
are affected the same way by all threats.
• The program has to target one variable and not
the other.
• In simple form, weak internal validity.

43
NEDV Example
• Only works if we can assume that geometry scores
show what would have happened to algebra if
untreated.
• The variable is the control.
• Note that there is no control group here.

44
NEDV Pattern Matching
• Have many outcome variables.
• Have theory that tells how affected (from most to
least) each variable will be by the program.
• Match observed gains with predicted ones.
• With pattern, NEDV can be extremely powerful.

45
NEDV Pattern Matching