Writing questions

1
Writing questions

• Survey Research and Design
• Spring 2006
• Class 8

2
Todays objectives

• To answer questions you have
• To understand the process of answering questions
  and the problems associated with the process
• To explore principles of survey question writing
• To understand how to build scales
• To distribute and discuss the mid-term exam

3
Quality perspective (Groves, et al., p. 48)
Measurement
Representation
_ Y
Target Population
m1
Construct
Coverage Error
Validity
_ yc
Measurement
Sampling Frame
Yi
Sampling Error
Measurement Error
_ ys
Sample
Response
yi
Non-response Error
Processing Error
_ yr
Respondents
Edited Response
Adjustment Error
yip
Postsurvey Adjustments
_ yrw
_ yprw
Survey Statistic
4
Survey questions

• Designing survey questions, evaluating them, and
  constructing the questionnaire are all
  interrelated weve just broken them up over
  three weeks
• How do respondents answer questions?
• Optimize respondents try to understand and
  accurately answer the question
• Comprehension, retrieval, estimation and
  judgment, reporting
• Satisfice try to get through questions quickly
• Three strategies
• Dont know response
• Nondifferentiation
• Acquiesence
• Holbrook et al. (2003). Vol. 67, pp. 79-125,
  Public Opinion Quarterly

5
Cognitive Processes in Answering Questions
Comprehension of the question
Retrieval of information
Judgment and estimation
Reporting an answer
6
Problems in answering survey questions

• Encoding problems
• Misinterpreting the questions
• Forgetting and memory problems
• Flawed estimation processes
• Problems in formatting the answer
• Deliberate misreporting
• Instruction/navigational errors

7
Types of response options

• Open-ended
• For self-administered surveys, best used for
  specific questions
• Closed-ended
• Ordered
• Useful for well-defined one-dimensional concepts
• E.g., satisfaction, agreement, intensity
• Unordered
• Cognitively more difficult
• E.g., please check which reasons you chose this
  college

8
Principles for writing survey questions

• Use words that all respondents will understand
• This means you will have to think carefully about
  your population
• College students/graduates are easiest to write
  for general population and children more
  difficult
• Choose as few words as possible
• People tend to be efficient and skip words
• Use complete sentences to ask questions
• Avoids misinterpretation
• Avoid vague quantifiers when more precise
  estimates can be obtained, and avoid specificity
  that exceeds the respondents potential for
  having an accurate, ready-made response
• Never, rarely, etc. vs. once a month, once a
  week, etc.
• Think carefully about categories that ask for
  amount intervals

9
Principles for writing survey questions

• Use equal numbers of positive and negative
  categories for scalar questions
• This means you want to balance your response
  categories
• Distinguish undecided from neutral by placing it
  at the end of the scale
• Undecided, no opinion, dont know, not applicable
• Avoid bias from unequal comparisons and state
  both sides of attitude scales in question stems.
• Eliminate check-all-that-apply questions.
• Use a yes/no response for the list

10
Principles for writing survey questions

• Develop mutually exclusive response categories
• Use cognitive design techniques to improve recall
• Nice recommendation, but difficult in practice
• Provide appropriate time referents
• Time questions are notoriously difficult to ask
• You may not be able to ask the question you want
• Be sure each question is technically accurate
• Choose wordings that allow essential comparisons
  to be made with previously collected data
• Not sure how applicable this is to us

11
Principles for writing survey questions

• Avoid asking respondents to say yes in order to
  mean no
• Avoid double-barreled questions
• Sounds obvious but can be hard to catch
• Soften the impact of potentially objectionable
  question
• Dillman has some excellent examples
• Avoid having respondents make unnecessary
  calculations

12
Response formats

• Likert scale
• Semantic differential
• Visual analog
• Numerical
• Binary

13
How many response categories should you have?

• Consider variability
• Increase response categories or increase number
  of questions related to a construct
• Can respondents discriminate meaningfully between
  a large number of response categories?
• Be careful not to be too ambiguous in the pursuit
  of more response categories
• Odd or even? Omit middle category? No opinion
  option?

14
Determine the format for measurement

• Thurstone scaling
• http//www.socialresearchmethods.net/kb/scalthur.h
  tm
• Guttman scaling
• http//www.socialresearchmethods.net/kb/scalgutt.h
  tm
• Scales with equally weighted items

15
Developing scales

• Some fields emphasize scales others are fine
  with single items
• Why use a scale?
• All of our survey data contain error for
  example, a respondent might choose the wrong
  response category due to distractions
• For a single item, this means error in the
  measure
• Suppose we combine several related items by
  adding them together
• Random measurement errors for a persons
  responses should average out
• Theoretical approach is that there is a latent
  construct that we can imperfectly measure with
  several survey items

16
Developing scales

• For the survey
• Figure out what you want to measure
• Develop a set of survey items that you think
  measure your construct
• Guidelines are similar to regular questions
• One difference reversal in item polarity
• Determine response format
• Likert scale is generally the best choice
• Have your item pool reviewed by experts
• Best to pretest so you have data to analyze
  before the survey administration
• With data
• Examine correlation matrix correlations should
  be positive
• If not, check item polarity and recode

17
Developing scales

• Calculate Cronbachs alpha
• SPSS Analyze, Scale, Reliability analysis
  choose Statistics and check Scale if item deleted
  and Inter-item correlations
• SAS Proc Corr, include Alpha option
• Drop items to increase alpha not necessary to
  use all items
• Alpha increases as
• Inter-item correlations increase i.e., items are
  similar
• Number of items increase
• Rule of thumb is alpha at least .70 or higher
  .90 or higher is best
• If you have several scales, okay to have one or
  two less than .70
• But be aware that this indicates your scale has
  quite a bit of noise

18
Generate a large pool of items

• Choose items that reflect the scales purpose.
• Redundancy can be okay, particularly when
  generating an item pool, if items express a
  similar idea in somewhat different ways.
• Start with a large item pool if possible.
• Look to other instruments for help.
• Begin writing questions.

19
knumber of items in the scale
mean interitem correlation
20
Group Projects Begin writing questions

• What questions you will ask?
• Are there survey instruments that will inform
  your work?
• Do you plan to build scales?

21
For next time Evaluating questions

• Readings
• Groves et al. - Chapter 8 (o. 241-266)
• Sudman, S., Bradburn, N., Schwarz, N. (Chapter
  2, p. 15-54)
• DROP Willis, G. B. In Presser, S., Rothgeb, J.
  M., Couper, M. P. , Lessler, J. T., Martin, E.,
  Martin, J., Singer, E. (Eds.). (Chapter 2, p.
  23-41)
• ADD Willis, Cognitive Interviewing A How to
  Guide (on course website)
• Due Take-home exam