Bias in Political Communication Experiments

1
Bias in Political Communication Experiments

• Jamie Druckman
• Thomas Leeper
• Dept. of Political Science Northwestern University

2
Mass Communication Effects

• Political communication research is one of the
  most notable embarrassments of modern social
  science (Bartels 1993).
• Compelling concepts that have had a major impact
  in political science and communications
  scholarship (Iyengar 2010).
• Key method behind progress ? experiments (survey
  lab).
• Randomly expose some respondents to one message
  (e.g., hate group rally as a free speech issue).
• others to another message (e.g. rally as a
  public safety issue), and
• measure the effect (e.g., support for the
  rally).

3
Mass Communication Effects

• But what about time? What happens before and
  after the experiment? Does it matter?
• Claim Ignoring what happens prior to the
  experiment (i.e., pre-treatment) has produced
  a bias portrait of communication effects that
  may
• overstate the malleability of the public (see
  Barabas and Jerit 2010).
• miss the identification of potentially two groups
  of citizens malleably reactive and dogmatists.
• contradict much macro opinion formation research.

4
Outline

• Psychology of pre-treatment effects
• Framing effects and defining pre-treatment
• Attitude strength
• Hypotheses ? conditions when pre-treatment
  effects occur
• Laboratory experiment
• Manipulates pre-treatment environment and
  attitude strength (via processing mode)
• Survey experiment
• Election exit poll experimental survey
• Measures pre-treatment environment and attitude
  strength correlate

5
Framing

• Framing effects ? In the course of describing an
  issue, a speakers emphasis on a subset of
  relevant considerations causes individuals to
  focus on these considerations when constructing
  their opinions.
• Example Politicians, media frame hate group
  rally request as a
• free speech issue ? citizens focus on speech
  considerations ? citizens support right to rally
• public safety issue ? citizens focus on safety
  considerations ? citizens oppose right to rally
• Experiments ? random exposure to framed
  communication as a news article or in question
  wording (e.g., on a survey).

6
Framing

• Many other examples
• campaign finance (free speech or corruption?)
• abortion (rights of mother or rights of unborn
  child?)
• gun control (right to bear arms or public
  safety?)
• affirmative action (reverse discrimin. or
  remedial action?)
• welfare policy (humanitarianism or overspending?)
• social security (individualism or shared
  security?)
• elections (economy or foreign affairs?)
• A Central Means of Elite Influence on Public
  Opinion.
• These effects ? valence framing effects (Tversky
  and Kahneman)

7
Pre-treatment

• Pre-treatment environment ? context prior to
  exposing experimental participants to the
  stimulus (frame).
• Pre-treatment effect ? aspect of the prior
  context affects responses to the stimulus
  (frame).
• Example
• Prior to the hate group rally experiment,
  respondents view news coverage using the free
  speech frame and become more supportive.
• another free speech exposure in the experiment
  does not further move opinion.
• Mistaken conclusion of no effect.
• Possibly dont/cant publish the experiment!

8
Pre-treatment

• Experimenters implicitly assumethat respondents
  enter the survey as clean slates despite the
  fact that there is inevitably some possibility
  that respondents enter the experiment having
  already participated in a similar experiment,
  albeit one occurring in the real world (Gaines
  et al. 2007).

9
Conditions for Pre-treatment Effect

• Exposure/Attention ? Absent exposure and
  attention to information, there is no
  pre-treatment.
• Durable Influence ?
• Absent initial influence, there is no effect.
• Absent durability, the initial influence will not
  impact experimental reactions (i.e., the effect
  must sustain until the time of the experiment).
• Depends on time between pre-treat and experiment.
• Depends on nature of the attitude formed in
  response to the effect
• Attitude strength

10
Attitude Strength

• Attitude Strength ? attitudes that persist and
  resist change.
• Stems from attitude features (e.g., extremity)
  and the formation/updating process.
• Two dynamics that affect attitude strength
• 1. Processing Mode
• 2. Need to Evaluate

11
Attitude Processing Mode

• Individuals form/update attitudes in relatively
  more memory-based or online fashion (Haste
  Parke 1986, Lodge et al.).
• On-line (OL)
• immediately integrate information (i.e., frames
  at time t in the pre-treatment environment) into
  an overall evaluative summary,
• store it, and
• recall it when needed (i.e., at t1, in an
  experiment).
• Thus, impact of earlier information sustains ?
  attitudes formed at time t pre-treatment are
  stronger (i.e., durable, resistant change in time
  t experiment).

12
Attitude Processing Mode

• Memory-based (MB) store information (frames) in
  memory, do not evaluate it until asked for an
  attitude at which point retrieve what can be
  recalled and integrate.
• May not recall items from distant past.
• Thus, impact of earlier information does not
  sustain ? attitudes formed at time t
  pre-treatment are weak and information may be
  forgotten.
• Pre-treatment effects are more likely to occur
  among OL processors since earlier effects sustain
  and generate resistance to later influence (i.e.,
  in the experiment).

13
Need to Evaluate

• Need to Evaluate (NE) ? individual propensity to
  engage in evaluation.
• More likely to assess and retain information
  (once exposed). Generate stronger attitudes,
  similar to OL processing.
• Thus, impact of earlier information sustains ?
  attitudes formed at time t pre-treatment are
  stronger (i.e., durable, resistant to change in
  time t experiment).
• Pre-treatment effects are more likely to occur
  among high NE individuals since earlier effects
  sustain and generate resistance to later
  influence (i.e., in the experiment).

14
Hypothesis

• Pre-treatment effects (e.g., leading to no
  experimental stimulus effect) will be more likely
  to occur when individuals are
• exposed and attentive to earlier communications
  similar to the experimental stimuli and
• form/update their attitudes in ways that promote
  strength. This occurs among on-line processors
  and high NEs.
• Motivated Reasoning Corollary
• Motivated reasoning ? reject information that is
  inconsistent with prior opinions.
• Those with stronger attitudes are more likely to
  engage in motivated reasoning.
• Those with strong attitudes (i.e., OL, high NE)
  not only will not be influenced by a repeated
  communication but may reject contrary
  communication.

15
Experiment 1

• Laboratory experiment with 744 participants
  (mostly students), Spring 2010.
• Two Issues (both salient but not currently
  intensely debated)
• Support for the Patriot Act (increases law
  enforcement power to combat terrorism) (measured
  on 7-point scale).
• Support for a state owned gambling casino
  (measured on 7-point scale).

16
Experiment 1

• Pro/Con Strong Frames (in news articles)
• Patriot Act
• Pro ? Protection from terrorism
• Con ? Violation of Civil Liberties
• Casino
• Pro ? Economic benefits (e.g., tax relief)
• Con ? Social costs (e.g., addiction, debt)

17
Experiment 1

• Manipulated OL or MB processing mode (using
  conventional psychological approach)
• OL ? respondents instructed to evaluate articles
  for their impact in increasing or decreasing
  support. Told they will later report opinions.
• MB ? respondents instructed to evaluate articles
  for their dynamic nature (i.e., use of
  action-oriented words).

18
Experiment 1

• Procedure ? Four waves, 5 days apart.
• Background survey, assigned to condition that
  varied (a) pre-treatment environment, (b)
  processing mode, (c) survey frame, AND
• --received two relevant pre-treatment frames,
  along with processing manipulation.
• Received pre-treatment frame articles, along with
  processing manipulation.
• Same as 2.
• Received survey question using no, con, or pro
  frame.
• ? virtually no attrition because compensation
  contingent on full completion.
• ? virtually no coverage of these issues during
  the experiment or several months prior.

19
Conditions

Expect survey frames will have scant effects in
the OL conditions, but will impact MB processors
(and the no-pre-treated individuals).
20
Patriot Act Survey Frames

• Control ? The Patriot Act was enacted in the
  weeks after September 11, 2001, to strengthen law
  enforcement powers and technology. What do you
  thinkdo you oppose or support the Patriot Act?
  Choose one number on the following 7-point
  scale.
•  
• Con ? technology. Under the Patriot Act, the
  government has access to citizens confidential
  information from telephone and e-mail
  communications. As a result, it has sparked
  numerous controversies and been criticized for
  weakening the protection of citizens civil
  liberties
•  
• Pro ?technology. Under the Patriot Act, the
  government has more resources for
  counterterrorism, surveillance, border
  protection, and other security policies. As a
  result, it enables security to identify terrorist
  plots on American soil and to prevent attacks
  before they occur

21
Casino Survey Frames

• Control ? A proposal is being considered for the
  Illinois state government to operate a land-based
  gambling casino. What do you thinkdo you oppose
  or support the proposal for a state-run gambling
  casino? Choose one number on the following
  7-point scale.
•  
• Con ? Some say that a state-run casino will
  have severe social costs, such as addiction and
  debt..
•  
• Pro ? Some say that the revenue from the casino
  would provide tax relief and help to fund
  education

22
Results

• Simple mean comparisons, robust to controls.
• Similar results for two issues here present
  Patriot Act results.
• Gradual disaggregation of conditions

23
Patriot Act Results

• Strong survey framing effect.

24
Patriot Act Results

Non-manipulated

• Strong survey framing effects for
  non-manipulated and MB.
• NO SURVEY FRAMING effects for OL.
• Effects apparent in merged data stem entirely
  from non-manipulated and MBs.

25
Patriot Act Results

Non-manipulated

• Non-manipulated MB.
• Aggregate results may stem from particular
  sub-groups.
• Aggregate effect sizes underestimate impact on
  affected groups (e.g., MB moved .10 more than
  merged data from pro frame).

26
Patriot Act Results

• Non-effect among OL is illusionary!

27
Patriot Act Results

• Pre-treatment effects
• Pro pre-treatment frames significantly increased
  support, regardless of survey frame.
• Con pre-treatment frames significantly decreased
  support, regardless of survey frame.
• Repeated frame in survey had minimal impact, and
  contrary frame was rejected (motivated
  reasoning).

28
Patriot Act Results

• No evidence of pre-treatment effects for MB
  processors.
• Significant survey framing effect in each case,
  regardless of the pre-treatment environment.

29
Patriot Act Results Belief Importance
Survey Frame / Consideration OL Processors in contrary pre-treat environment and received survey frame All Other Respondents OL Processors in contrary pre-treat environment but did not receive survey frame
Patriot Act Con Civil Liberties 4.49 (1.71 37) 5.55 (1.46 612) 4.94 (1.56 98)
Patriot Act Con Terrorism 3.97 (1.87 39) 5.15 (1.51 609) 4.55 (1.76 85)
p.01 p.05 p.10 for one-tailed tests,
relative to OL Processors in contrary pre-treat
environment and received survey frame condition.

• OL processors who received a survey frame
  contrary to their pre-treatment environment,
  viewed that argument as significantly less
  important ? evidence of motivated reasoning
  (rejection of contrary information).
• Are these types of effects evident outside of
  manufactured lab setting?

30
Experiment 2

• Exit poll experimental survey that measures
  pre-treatment environment and attitude strength
  correlate, with 338 respondents.
• Issue Support for a proposed state owned
  gambling casino during the 2006 IL Gubernatorial
  campaign (measured on a 7-point scale).
• Tracked media (i.e., pre-treatment) environment
• Pro-casino frames economic benefits,
  (entertainment).
• Con-casino frames social costs (corruption,
  morality).
• Exit poll survey experiment.
• Measure attention and likelihood of enduring
  opinion (via Need to Evaluate item).

31
The Campaign(pre-treatment environment)

• 2006 IL Gubernatorial pitting Blagojevich (D) vs.
  Topinka (R).
• Aug. 23 Topinka proposes state owned casino to
  raise revenue (t 1).
• Aug. 24 Chicago Tribune Comment Topinka is
  framing the contest just as it needs to be
  framed How can a grossly overcommitted state
  gov. bend financial trend lines that point
  inexorably toward ruin.
• Sept. 9 Corruption accusation against
  Blagojevich of taking a personal payoff for a
  state job.
• Others follow!
• Nov. 7 Election Day (t n).
• Coded campaign coverage in the Chicago Tribune
  from t 1 (Aug. 23rd) to t n (Election Day).
• Most prominent issues economy (budget)
  corruption.

32
The Campaign

• Early campaign discussions of the casino were
  framed in terms of economic benefits gt 75 of the
  time.

33
Predictions

• Attentive high NEs will be significantly more
  likely to form and maintain casino opinions upon
  exposure to the early campaign information (e.g.,
  with economic frame focus).
• ?
• Less susceptible to the economic frame later
  (already influenced) and the social costs frame
  (reject it).
• More supportive of the casino proposal.

34
Exit Poll

• Election Day exit poll (t n).
• Random sample of polling stations in north Cook
  County. (5 for participation).
• Measured NE (i.e., 1 item measure) Campaign
  Attention (i.e., newspaper reading during
  campaign period), etc.
• Attentive voters ? greater knowledge, discussion,
  interest in politics.
• Attentive / High NE voters ? above median on NE
  and attention (n 111).
• Four relevant experimental conditions
• No frame asked extent of casino support.
• Economic frame ? pro casino.
• Social costs of gambling ? con casino.

35
NE Measures

• Some people have opinions about almost
  everything other people have opinions about just
  some things and still other people have very few
  opinions. What about you? Would you say you have
  opinions about almost everything, about many
  things, about some things, or about very few
  things?
• almost many some very few
• everything things things things
• Compared to the average person, do you have a lot
  fewer opinions about whether things are good or
  bad, somewhat fewer opinions, about the same
  number of opinions, somewhat more opinions, or a
  lot more opinions?
• a lot fewer somewhat about the somewhat a lot
  more
• opinions fewer opinions same more
  opinions opinions
• Some people say that it is important to have
  definite opinions about lots of things, while
  other people think that it is better to remain
  neutral on most issues. What about you? Do you
  think it is better to have definite opinions
  about lots of things or to remain neutral on most
  issues?
• definite remain
• opinions neutral

36
Results

• All Large survey framing effect.
• Non-attentive/Low NE ? large survey framing
  effect.

37
Results

• Attentive/High NE ? NO survey framing effect.
  Significantly higher support across conditions.
• Pre-treatment effects ? No experimental effects
  but was earlier influence.
• Other evidence suggests motivated reasoning for
  those receiving social costs frame.

38
Summary

• The existence of an experimental effect can be
  misleading as it
• may stem from a subgroup that formed weak
  attitudes (e.g., MB) on the issue (Gaines et al.
  2007, Barabas and Jerit 2010), and
• may understate the effect size among those
  individuals.
• The non-existence of an experimental effect can
  be misleading as it
• may stem from a large number of individuals
  forming strong attitudes (e.g., OL) in response
  to communications prior to the experiment.
• Such individuals were limited in our studies, but
  may be pervasive in other contexts.
• Hillygus and Jackman (2003) ? conventional
  effects gt debate effects.

39
Summary

• Given publication biases (e.g., Gerber et al.
  2010), experimental studies may over-state the
  existence of effects, and thus
• The mass public, on average, is less malleable
  and holds more stable opinions than would be
  suggested by the aggregation of experimental
  results.
• The mass public may be bi-modal ? malleably
  reactive and dogmatically invulnerable.
• Caveat ? those who formed strong attitudes were
  affected by earlier stimuli.

40
Summary

• Varying levels of stability in macro and
  micro-level studies of opinion may stem, in part,
  from different issue foci
• Macro studies ? longstanding salient issues that
  generate stronger attitudes (e.g., Gallups most
  important problem surveys)
• Micro over-time studies ? relatively novel and
  specific issues (e.g., ballot proposition, new
  candidate, regulation of hog farms, campaign
  finance).

41
Conclusions

• Opinions are not fixed in time. Time dynamics
  need study ? priors, pre-treatment, durability.
  and post-treatment effects.
• Failure to account for these dynamics ?
  inferential errors.
• Precise effects depend on attitude strength.
• What should public opinion researchers do?
• Define time period of study (as a unit of
  analysis).
• If goal is to evaluate impact of an argument,
  test for pre-treatment effects.
• Identify prior rhetorical context.
• Test with distinct populations or times.
• Develop theories of over-time effects

42

• END

43
Casino Results
44
Casino Results
45
Casino Results
46
Casino Results