Telephone Polls: Opportunities and Pitfalls Martin Boon, ICM Research

1
Telephone Polls Opportunities and
PitfallsMartin Boon, ICM Research
2
The catalyst 1992

• The 1992 election was a defining point in
  political opinion polling in the UK.
• 1992 five main pollsters, one data collection
  methodology, one wrong outcome.
• The scene is set for the pollsters
  methodological debate.

3
Telephone polling a short history method

• The most visible aspect of methodological change
  the switch to telephone.
• The basic method RDD sampling using
  approximately 12,000 telephone numbers.

2005 Evening Standard prediction poll by
telephone
4
Telephone polling the outcome

• ICM final prediction telephone poll average error
  in 1997 1.2 average error. Peter Kellner, 2nd
  May 1997 Congratulations. The Evening Standard
  is on the streets.
• It the only prediction poll to understate the
  Labour lead in any General Election since 1983.
• In 2001, the best telephone poll had an average
  error of 1.6.
• In 2005, we saw the great success of NOP in
  producing a statistical bulls-eye (average error
  of 0.25).
• Other types of telephone polling have also
  produced accurate predictions. Polls across
  marginal seats, constituency polling and at least
  one poll in Scotland did well in estimating the
  actual result.

5
But is it about the data collection method?

• The introduction of a telephone methodology is
  but one aspect of the evolution of political
  opinion polling.
• Over time, data collection change was reinforced
  and refined with wording change, turnout filters,
  past vote weighting and adjustment of partial
  refusers.
• Most of these changes had a greater impact on
  vote intentions than data collection switchover.
  The switch to telephone should not be seen as the
  single most important contribution to the
  development of polling methods.
• By the 1997 General Election, Gallup had also
  switched to random telephone methods but had not
  innovated beyond that. Their campaign polls
  looked more like face-to-face polls rather than
  ICMs telephone equivalent

6
But is it about the data collection method?
Average 1997 campaign poll ratings, by company
Con Lab LD Other Lead
Harris (6) 30 51 13 6 21
MORI (8) 30 51 13 6 21
NOP (6) 29 49 15 7 20
Gallup (15) 31 50 13 6 19
ICM (6) 33 45 16 6 12
Result 31 44 17 7 13
7
The BIG issues that confront us today

• Non-Internet opinion polls in Britain suffer from
  a persistent, long standing Labour bias. The
  great challenge for pollsters is to remove or at
  least account for pro-Labour bias.
• On a standardised measure of deviation across
  these polls we find that Labour recall is the
  least volatile (1.13), followed by the Liberal
  Democrats (1.2) and the Conservatives most
  accurate but most volatile (1.27).
• i.e recall for Labour may be hopelessly wrong,
  but at least its wrong in the most consistent
  way.

Average faulty recall by party, ICM polls Nov 08
Dec 09

8.7 1.8 6.6
8
Why do telephone polls overstate Labour?

• Suggestions include
• Inaccurate demographic profiles - but
  demographics have a negligible impact on voting
  demographically representative surveys do not
  mean they are politically representative.
• We get through to too many Labour voters for
  systematic reasons
• low response rates
• differential refusal
• public sector workers
• spiral of silence

9
The practical difficulties

• There can be little doubt that telephone polling
  is becoming harder, and will get harder still.
• An increasingly unresponsive public. N 1,000
  depends on c.12,000 RDD records. ICM telephone
  omnibus alone conducts 100,000 interviews per
  year, implying that on a 121 ratio our pool of
  the general public is being depleted by over a
  million households per year on a worst case
  scenario.
• Sales, marketing and research drive more and more
  to TPS and tele-screening gadgets.
• 21 million UK households, 12 (2.5m) are mobile
  only. Cardiff (29) is the mobile capital of the
  UK (August 2009). How do we contend with mobile
  only households and future possible
  non-geographic landline numbers?
• Non-response among specific groups. We typically
  achieve 64 of the required interviews with 18-24
  year olds.

10
Is it a case of papering over the cracks?
11
Telephone polls are still up to the job!

• When all is said and done we have to be rational
  and make decisions on what we think is the best
  methodology over the medium to long term. We
  still believe that telephone trumps online.
• The record of telephone polls does stand up by
  and large when tested in electoral conditions.
• RDD sampling combined with quota methods is still
  able to draw a representative sample and
  demographic weighting is no big deal.
• Past vote weighting, or other strategies can and
  do contend with the political balance issue.
• On the practical level our telephone opinion
  research continues to be successful and doesnt
  currently suffer from unsustainable refusal
  rates.
• Telephone polls can be undertaken quickly and
  cost effectively.

12
Innovation
13
Crystal ball gazing.

• Innovation was the key when it came to dealing
  with 1992 issues. ICM and others innovated within
  the telephone interviewing framework YouGov and
  others innovated with the online solution.
• Telephone polling will have to change in order to
  meet the conditions the great British public
  impose on us. In 20 years time what might a
  telephone poll look like?
• We can speculate that any of the following might
  contribute to orthodox opinion research
• SMS polling (its here now)
• Mixed method polling
• Mobile-only random samples
• Telephone panels
• We reserve the right to make any changes that we
  think will help to improve accuracy so long as
  we are transparent about what we do, then others
  can judge both in terms of suitability and
  effectiveness.