Experiments Exploring the Relationship (or Lack Thereof) Between Nonresponse Error and Measurement Error

1
Experiments Exploring the Relationship (or Lack
Thereof) Between Nonresponse Error and
Measurement Error

• Roger Tourangeau, SRC, JPSM
• Robert M. Groves, SRC, JPSM
• Stanley Presser, JPSM
• Chris Toppe, Points of Light Foundation
• Courtney Kennedy, MPSM
• Ting Yan, NORC

2
Acknowledgement

• The work reported here was supported by a grant
  from the National Science Foundation (SES 550385)
  to Roger Tourangeau and Robert Groves. We are
  grateful to NSF for its support and especially to
  Cheryl Eavey.
• The National Science Foundation is not
  responsible for the conclusions presented here.
• We thank Duston Pope for his contributions to the
  design, John LaFrance, and Mirta Galesic for
  their contributions to the studies discussed here
  study design.

3
Outline of Talk

• Models of the relationship between nonresponse
  and measurement error
• Experiment 1 A study of volunteers and
  volunteering
• Experiment 2 Topic and sponsorship in a two
  phase web surveys
• Two additional studies
• Gain-loss framing and response propensities
• An experiment with voters and nonvoters
• Conclusions

4
Models of the Nonresponse-Measurement Error
Relationship

• Underlying model

5
Model 1 Independence Model

• Nonresponse essentially motivational (interest in
  topic, altruism, low opportunity costs)
• Error largely cognitive (poor comprehensive of
  the items, memory difficulties, poor estimation
  strategies)

M
C
p
Up
Ue
e
6
Model 2 Reluctance

• Difficult to persuade respondents answer sloppily
  (Bollinger and David, 2001 Fricker 2006)Common
  cause model

M
p
Up
Ue
e
7
Model 3 Topic Sensitivity

• Drug users and reports about drug use

X
p
e
Up
Ue
8
Model 4 Topic, Sponsor Effects

• Presentation of the survey request affects both
  who responds and how they respond

Topic Interest
Topic
p
e
Up
Ue
9
Framing Effects

• Why should the description of the topic or
  sponsor effect the answers Rs give?
• Three mechanisms
• Construal Description of the survey may affect
  how they interpret the questions (National Crime
  Survey vs. National Survey of Violence Against
  Womenthe latter may suggest a broader
  interpretation of sexual assault than the
  former)
• Priming Topic, sponsor may serve as a retrieval
  cue, making some memories more accessible
• Perception of what is relevant or
  usefulNorenzayan and Schwarz (1999) Institute
  for Social Research and Institute for Personality
  Research
• Evidence for all three (Galesic and Tourangeau,
  2007)

10
Experiment 1 The March of Dimes Study

• Survey topics
• Survey of Giving and Volunteering
• Survey of Jobs and Labor Force Activity
• Target populations
• March of Dimes mailing list
• Donors, volunteers, other nominated by donors
• General adult sample from vendor list
• Sponsor
• March of Dimes
• University of Michigan

11
Recruitment Protocol

• Mailed paper questionnaire
• Advance letter
• Initial questionnaire packet
• Reminder postcard
• Followup questionnaire packet
• All materials emphasized sponsor and topic

10
12
Response Rate by Topic and Population
Concern about sugging?
11
13
Commitment to MoD Predicts Preference for the MoD
Survey
Predictor Comparison Categories Odds-Ratio of Participating in the Survey Odds-Ratio of Participating in the Survey Odds-Ratio of Participating in the Survey
Predictor Comparison Categories March of Dimes Volunteering UM Volunteering UM Labor Force
Volunteer Last Year Yes vs. No 7.9 (4.0, 15.5) 2.5 (1.1, 5.4) 1.9 (0.9, 3.8)
Events Last Year Any Event vs. No Event 5.0 (2.6, 9.8) 2.2 (1.1, 4.4) 2.3 (1.2, 4.7)
Recency of Donation w/in last 12 mos. Vs. longer ago 2.4 (1.6, 3.4) 1.8 (1.4, 2.5) 1.6 (1.2, 2.2)
Last Year Donation Some Money vs. None 1.8 (1.2, 2.8) 1.4 (1.0,1.9) 1.4 (1.0, 1.9)
Mean 4.3 2.0 1.8
12
14
Mixed Results on Commitment and Poor Match to
Records
Predictor Comparison Categories Odds-Ratios Odds-Ratios
Predictor Comparison Categories Participating Matching Record Report
Volunteer Last Year Yes vs. No 7.9 (4.0, 15.5) 0.8 (0.4, 1.5)
Events Last Year Any Event vs. No Event 5.0 (2.6, 9.8) 0.006 (.002, .020)
Recency of Donation w/in last 12 mos. Vs. longer ago 2.4 (1.6, 3.4) 7.1 (3.4, 14.9)
Last Year Donation Some Money vs. None 1.8 (1.2, 2.8) 2.5 (1.206, 5.303)
Mean 4.3 2.6
13
15
Conclusions from Preliminary Analysis

• Volunteers and donors to March of Dimes tend to
  participate in all survey topics/sponsor
  combinations
• Those MoD donors with greater commitment more
  positive to the MoD sponorship than those with
  lesser commitment
• Mixed results on whether those with greater
  commitment tend to misreport their activities in
  support of MoD

14
16
Experiment 2 Two-Wave Web Survey

• Collected data from members of two web panels in
  on-line survey
• Wave 1 collected predictors of response
  propensities and answers to second wave
• Wave 1 respondents (n3000) invited to take part
  in second survey about a month later
• Ostensible topic and sponsorship of second survey
  varied
• Examined how response rates varied as a function
  of attitudes toward topic and sponsor
• Also looked at effects on answers

17
Experiment 2 First Wave

• n47,078 asked to complete University of Michigan
  survey on attitudes and lifestyles
• Members of two web panels SSI (4.1 responded),
  e-rewards (14.8) 3,000 respondents in all
• Wave 1 items included
• Questions on how often the respondent had
  responded to web survey requests in the past
• Items assessing the respondents views about the
  importance of a number of social issues,
  including issues related to the topic of the
  second questionnaire (crime and violence, gun
  control)
• Items assessing support for or opposition to gun
  control policies
• Relevant factual items (such as the presence of a
  gun in the household and experiences using guns)
• Rating scales on the prestige of various
  organizations, including the purported sponsors
  of the second

18
Experiment 2 Second Wave

• n2,020 (67.3 response rate conditional on wave
  1)
• Varied three factorssponsor, topic, and
  purported length
• Sponsor
• National Coalition of Gun Owners (pro-Gun)
• National Coalition for Victims of Gun Violence
  (anti-gun)
• National Center for the Study of Crime
• Topic
• Crime prevention
• Gun control policies
• Length (20 minutes, 5 minutes, no mention)

19
Our Hypotheses

• Attitudes toward the sponsor and topic to affect
  both the likelihood of cooperation and response
  behavior on questions relevant to the sponsors
  likely point of view.
• Similarly, we expected topic interest to affect
  response propensities and answers to questions in
  the survey
• Expected burden to lower response rates

20
ResultsResponse Rates

• The experimental variables seem to have little
  impact on either the response rates or
  nonresponse bias in estimates in the second
  survey.

21
Results Answers

• Created three composites
• Personal safety (4 items) no effects
• Gun control attitudes (11 items)
• Main effect for topic More support for gun
  control when the topic was described as crime
  prevention than when it was described as gun
  controlcontext effect?
• Purported sponsor interacted with wave 1 atts
  (next slide)
• Involvement in issue of gun control (3
  items)topic effect among pro-gun types those
  below the first survey median in their support
  for gun control measures reported greater
  involvement in the issue when the topic was
  described as crime prevention (mean of 2.47) than
  when it was described as gun control (mean of
  2.36)

22
Results Gun Control Attitudes
23
Results Liking for the Sponsor and Sloppiness
24
Two New Studies Gain-Loss Framing

• Gain-Loss Framing Two-wave telephone study
  request to do second interview at end of first
  interview
• We really appreciate the help youve given us
  today. We are interested in how peoples views
  about the issues we discussed today change over
  time so it is important we talk to you again in a
  couple of weeks.
• Then get either
• The information youve already provided to us
  will be a lot more valuable if you complete the
  second interview (77.9)
• Or
• Unfortunately, the information youve already
  provided to us will be much less valuable unless
  you complete the second interview (87.5)

25
Voting Study

• Obtained sample of Maryland voters and non-voters
• Vary mode of data collection (mail vs.
  telephone), incentive (5 vs. none), and topic
  (Health and Lifestyles vs. Politics, Elections,
  and Voting)
• Attempt to test topic sensitivity model (Model
  3)
• Reasons to think that dosage could be higher or
  lower in mail

26
Voting StudyMail Questionnaire
27
Voting StudyMail Questionnaires
28
Conclusions

• Every study is framed somehow
• Framing can effect response rates, though not
  necessarily nonresponse error
• Topic effects on response rates found in Groves,
  Presser, and Dipko (2004), Experiments 1 and 2
  here, Birding study in Groves et al. (2006)
• But not alwaysthree other studies in Groves et
  al. (2006) dont find topic effects
• Sponsor can effect responses, but again not
  always
• Framing can effect answers, though effects may be
  subtle as in Experiment 2 here and in earlier
  experiment by Galesic and Tourangeau (2007)
• Only weak evidence that reluctant Rs, or those
  who dislike sponsor or topic more prone to
  measurement error