Survey research techniques: Matching the method with the research question

1
Survey research techniques Matching the method
with the research question

• Robert M. Stein
• Department of Political Science
• Rice University
• March 2015

2
Popular survey techniques

• Face-to-face interviews
• Mail surveys
• Self-administered surveys
• Telephone interviews
• Live interviewer
• Interactive voice recorded
• Web based surveys (with/without telephone
  recruitment. e.g., Mechanical Turk, Knowledge
  networks, Goggle Consumer Surveys
• Focus groups

3
Major research issues with surveys

• Sampling
• Cost
• Reactivity
• Psychology of the survey response
• The dont know response
• Response time
• Social desirability
• Question wording and placement
• Meaning of survey responses

4
Survey methods for studying who votes

• Web based surveys
• Strengths Cost, sample size, timely data
  collection, survey panels
• Weaknesses Selection bias e.g., digital divide
• Telephone, face-to-face surveys
• Strengths Sample size, limited to registered
  voters
• Weaknesses Social desirability, reactivity,
  recall problems, sample selection bias (unlisted
  phone number
• Exit poll
• Strengths Validated voters, limited number of
  questions
• Weaknesses Reactivity, unable to study
  non-voter
• IVR poll
• Strengths Large sample, less invasive, lower
  cost
• Weaknesses Low response rate, limited number of
  questions, skewed sample
• Surveyless Survey (e.g., annotated voter
  histories)
• Strengths Unambiguous, valid and reliable
  information
• Weaknesses Limited sample (i.e. registered
  voters) and bias time series

5
Overreporting voting

• Robert Bernstein, Anita Chadha and Robert
  Montjoy, Overreporting Voting Why it Happens
  and Why it Matters. Public Opinion Quarterly
  65(2002)22-44
• Allyson L. Holbrook and Jon A. Krosnick,
  Measuring voter turnout by using the randomized
  response technique Evidence calling into
  question the methods validity, Public Opinion
  Quarterly 74(2010)328-343.
• Brian D. Silver, Paul R. Abramson and Barbara A.
  Anderson, The presence of others and
  overreporting of voting in American National
  Elections, Public Opinion Quarterly
  50(1986)229-239
• Robert F. Belli, Sean E. Moore and John
  VanHoewyk, An experimental comparison of
  question forms used to reduced vote
  overreporting, Electoral Studies 25(2006)751-759

6
Social expectation and over reporting of voting

• Treatment 1
• In talking to people about elections, we often
  find that a lot of people were not able to vote
  because they weren't registered, they were sick,
  or they just didn't have time. How about you--did
  you vote in the elections this November?
• Treatment 2
• In talking to people about elections, we often
  find that a lot of people were not able to vote
  because they weren't registered, they were sick,
  or they just didn't have time. Which of the
  following statements best describes you One, I
  did not vote (in the election this November)
  Two, I thought about voting this time - but
  didn't Three, I usually vote, but didn't this
  time or Four, I am sure I voted?

7
Live telephone and IVR surveys of 2009 Houston
mayoral election voters

• Live telephone interview
• 500 registered voters interviewed 11/15/09
  -11/18/09
• Error rate /- 4.5
• Interview length 7.5-12 minutes (95)
• Response rate 32
• IVR interview
• 586 registered voters interviewed 11/15/09
  -11/18/09
• Error rate /- 3.3
• Interview length 1- 3.5 minutes (95)
• Response rate 8.5

8
(No Transcript)
9
(No Transcript)
10
Response time (latency) measures of contextual
effects on voting behavior

• Problem Studies of contextual processes have
  always involved the possibility that if
  individuals aggregation into geographic units is
  not exogenous to their values on the dependent
  variable, then what appear to be contextual
  processes may be due solely to selection
  effects.
• Solution measure contextual variables separately
  from the individual-level variables, and later to
  connect them to the survey data by means of
  linkage variables. Latency measures of response
  time to survey generated questions.
• Methodology Cati (computer assisted telephone
  interviewing) technology used to generate latency
  measures to query to respondents assessment of
  the partisanship of their neighbors

• M. Johnson, W. Phillips Shively and R.M. Stein,
  Contextual data and the study of elections and
  voting behavior connecting individuals to
  environments. Electoral Studies
  21(2002)219-233.

11
Hypotheses

1. When perception of partisanship of ones
   neighbors (i.e., (generally speaking, do you
   usually think of your neighborhood as Republican,
   Democratic or Independent?) is latent (i.e.,
   faster response time) context sould have a
   significant and positive effect on vote choice.
2. Republicans (Democrats) residing in neighborhoods
   that perceive to be Democratic (Republican) will
   be more likely to vote for Democrats
   (Republicans) when their perception of the
   partisan make-up of their neighborhood is readily
   accessible.
3. The accessibility of ones context will be
   greater when that context is congruent with a
   respondents personal preference.

12
Research design

• Live telephone interview with 750 registered
  voters in Harris County, Texas, conducted
  September 23-29, 1999
• Retrospective vote choice in the 1996
  Presidential election, partisanship and
  perception of partisanship of neighborhood
• Response times collected on Cati system similar
  to Bassli and Fletcher (1991).

13
(No Transcript)
14
(No Transcript)
15
(No Transcript)
16
Analyzing surveys

• When are answers to survey questions revealing of
  individual perceptions, preferences and behavior?
• How might multiple survey responses reveal more
  information than single responses?
• An Answer Risk perceptions and evacuations from
  hurricanes.

17
Google consumer surveys(GCS)

• GCS is a new tool developed by Google that
  surveys a sample of Internet users as they
  attempt to view content on the websites of online
  publishers who are part of the program.
• These sites require users to complete a survey
  created by researchers in order to access premium
  content provided by the publishers.
• Currently, GCS is available in four countries and
  it takes about 48 hours to field a survey of any
  size. The 4 countries are USA, Canada, UK, and
  Australia.
• The current cost-structure of GCS makes very
  short surveys (even as short as one question) the
  most economically attractive.
• Since the respondents are not part of an online
  panel, GCS has not collected this information at
  an earlier date (as is the case in many panels).
  Instead, GCS provides researchers with a set of
  respondents inferred demographics (location,
  income, age, gender) that are derived from their
  browsing histories and IP addresses .

18
Example of Survey Wall
19
Cost structure of GCS
20
Limitations of GCS

• GCS recommends the question length to be 125
  characters and sets the maximum limit to 175
  characters.
• Censored questions and populations GCS places
  restrictions on sensitive demographics
  information by prohibiting researchers from
  asking respondents for their age, gender,
  ethnicity, religion, and immigration status.
  Researchers can only ask these questions if the
  response choices include "I prefer not to say" as
  the opt out answer.
• GCS also requires an option for the respondents
  to not answer the question by either clicking an
  I dont know, show me another question link.
• Observational studies that require statistical
  models including a long (or even small list of
  control variables can likely be accomplished as
  efficiently using other survey tools.

21
Strengths of GCS

• First, if these experiments can be implemented in
  one question, then the considerable cost
  advantages of GCS over its competitors can be
  realized.
• Second, If true randomization of treatments can
  be achieved (and we argue that GCS methodology
  does so - and demonstrate resulting balance in a
  set of measured covariates then no additional
  control variables are necessary to make strong
  inferences.
• Finally, given the fact that most experimenters
  are much more concerned with the internal
  validity of their inferences than with tier
  external validity, unresolved questions about the
  representativeness of the GCS sample may be of
  secondary concern and certainly the sample is
  better  than that of most lab experiments.