Plus ca change: barriers to innovation in survey research

1
Plus ca change barriers to innovation in
survey research

• Patrick Sturgis, University of Southampton
• and
• National Centre for Research Methods (NCRM)

Paper presented at Government Statistical Service
Conference, London, July 6 2011
2
NCRM

• Network of gt10 universities offering an
  integrated programme of methodological research
  and training
• Quantitative, qualitative and mixed methods
• Online archive of papers, videos, presentations,
  toolkits, etc.
• Bursary scheme to support training attendance
• Find out more at www.ncrm.ac.uk

3
Key changes in survey practice

• Telephone interviewing
• RDD
• CAPI
• Electronic sample management
• Web surveys
• Mobile digital devices
• Attitude measurement?

4
5-point Likert agree/disagree format

• Probably the most widely used attitude question
  format in the UK and throughout the world
• BSA, ESS, BHPS, BCS, Taking Part, cohort studies,
  WVS, EVS, etc. etc.
• But it has long been recognised that there are
  some major problems with this approach

5

• Working in government statistics is enjoyable
• Strongly agree/agree/neither-nor/disagree/strongly
  disagree

6
But there is another problem
7
Neither/nor mid-points

• Perhaps the most common form of middle
  alternative is the neither/nor in Likert items
• Usually treated analytically as implying
  neutrality, or being in the middle on the
  issue
• But it is also logically consistent with having
  no opinion
• If I do not have an opinion on an issue, it is
  true that I am neither in favour nor opposed to
  the statement

8
Research Design

• Face-to-face CAPI interviews on the Ipsos-MORI
  omnibus quota survey
• May 2008
• n3113
• Respondents randomly allocated to one of three
  5-point uni-polar Likert items

9
Overall, how satisfied or dissatisfied are you
with the performance of the European Commission?

• 1. Very satisfied
• 2. Fairly satisfied
• 3. Neither satisfied nor dissatisfied
• 4. Fairly dissatisfied
• 5. Very dissatisfied

10
How important or unimportant do you think it is
it for Britain to be at the forefront of
developments in nanotechnology?

• 1. Very important
• 2. Fairly important
• 3. Neither important nor unimportant
• 4. Not very important
• 5. Not at all important

11
On balance, the advantages of GM foods outweigh
any dangers

• Strongly agree
• Agree
• Neither agree nor disagree
• Disagree
• Strongly disagree

12
For those selecting neither/nor

• Which of the following statements best describes
  why you are neither satisfied nor dissatisfied
  with the performance of the European Commission?
• I dont really have an opinion on this issue
• I have an opinion which is right in the middle on
  this issue
• Neither of the above
• If (3) Could you tell me what led you to say you
  are neither satisfied nor dissatisfied?
• PROBE FULLY. For what other reasons?
• RECORD VERBATIM

13

• Responses can now be categorised as either
• Substantive (directional) response
• Initial Dont Know
• Hidden Dont Know
• Neutral

14
Table 1 Distribution of middle alternative
responses after probing
15
Reference category initial DK
16
Why are survey question formats so resistant to
change?
17

• However, you will have to balance this against
  the policy needs of your funders changing the
  question and/or response categories will mean
  that the data you collect will not be comparable
  to other surveys against which you may be
  benchmarking your data.

18
Some predictions for 2020

• General population surveys will routinely be
  conducted using web data collection
• Interviewing conducted by avatars
• 5-point agree/disagree questions with neither/nor
  mid-point still widely used.

19
Some previous predicitons

• Frank Dobson for Mayor of London
• West Ham premier league champions 2010