A poor prognosis for survey methods Assessing the health of survey methods in social research

1
A poor prognosis for survey methods? Assessing
the health of survey methods in social research
McCaughey CentreSurvey Methods SeminarMay 2008
2
The Topic
Survey methods have been a stalwart tool for
social science research, however, traditional
approaches to survey methods are becoming
increasingly difficult to undertake. Letterboxes
are overflowing with junk mail, households are
fatigued by unsolicited telephone calls and
electronic home protection systems are turning
homes into fortresses. With low response rates -
what does this mean for ideals of
representativeness and what does
representativeness mean in contexts of
increasing diversity within populations? For
survey methods to adapt to these changing
contexts for social science research and to
continue to provide reliable and valid insights
into peoples lives there is a need for
innovative approaches.
Sample frames Response Rates Methodological
Innovations
3
Credentials About the Social Research Centre

• Established in 2000 as a specialist provider of
  survey research services to government and
  academia
• Operates as a social research supplier under the
  umbrella of the market and social research
  industry
• 25 permanent staff
• 180 telephone interviewers
• 110 seat Call Centre
• Our aim is to enhance knowledge and contribute to
  informed decision-making by undertaking high
  quality social and health research

4
About the Social
Research Centre
Services provided

• large-scale survey design, conduct and management
• longitudinal surveys
• population health research
• community attitudes research
• campaign evaluation research
• research amongst persons of culturally and
  linguistically diverse backgrounds
• Programme evaluation and policy development
  research

5
About the Social
Research Centre
Research Areas
6

• Random Digit Dialling (RDD)

7
RDD Background
The rise of RDD and the demise of the Electronic
White Pages (EWP)

• Up until quite recently the Electronic White
  Pages was considered complete and accurate for
  most survey research purposes
• The final version of the Desktop Marketing
  Systems (DtMS) Electronic White Pages (EWP)
  product was released in July 2004 and is still
  widely used by market research companies but is
  no longer used for high quality social research
• Random Digit Dialing (RDD) methodologies are now
  the norm rather than the exception
• One of the main barriers to the efficient use of
  RDD has been overcome by the development of
  automatic dialing technologies

8
EWP versus RDD Methods of RDD generation

• Generation process involves
• Undertaking a random selection of seed numbers
  from the 2004 Electronic White Pages
• Retaining the eight digit stem and deleting the
  last two digits
• Randomly generating two numbers to append to the
  eight digit stem
• Washing the resultant randomly generated ten
  digit numbers against business listings
• Send those records for which we get a confirmed
  address a primary approach letter

Launched in 2003
Governments 5 year plan for Queensland women
Initiatives to address areas where women
experience inequity,under-representation and need
9
EWP versus RDD Selected characteristics
Selected demographic characteristics of persons
with listed and unlisted phone numbers
10
EWP versus RDD Selected characteristics
Selected survey measures for persons with listed
and unlisted phone numbers
All of the results are statistically significant
at the 95 confidence interval.
11
RDD Improved representation

• RDD samples can access different persons in terms
  of
• Age distribution
• Demographic characteristics
• Key usage, attitudinal or behavioural variables
• Inclusion of persons with unlisted numbers in the
  sample universe by using RDD techniques can
  achieve a more representative sample of the
  general population
• BUT IN THE SPACE OF A DECADE THE
  REPRESENTATIVENESS OF SAMPLES OBTAINED FROM RDD
  FRAMES IS STARTING TO BE ERODED

Launched in 2003
Governments 5 year plan for Queensland women
Initiatives to address areas where women
experience inequity,under-representation and need
12
RDD Limitations of RDD frames

• Little or no coverage of areas poorly represented
  in source listing e.g. emerging growth areas
• The emergence of mobile only households. Non
  coverage of households without a landline (used
  to be 1-2 but now probably around 13)1,2
• Decreasing ability to obtain address matches and
  send out primary approach letters
• 12.8 of households in the US in 2006. National
  Health Interview, Public Opinion Quarterly, Vol.
  71
• One in four adults aged 18 to 24 years, Ibid.

Launched in 2003
Governments 5 year plan for Queensland women
Initiatives to address areas where women
experience inequity,under-representation and need
13
Response Rates Are response rates declining?

• Generally, probably yes but not inevitably so
• In market research it has been predicted that
• Reach is increasing but co-operation is
  declining and non-contact is increasing, leading
  to a 3 decline in response rate per annum
  (Bednall, 2000).

Launched in 2003
Governments 5 year plan for Queensland women
Initiatives to address areas where women
experience inequity,under-representation and need
Is the same true in social research Possible yes
but not inevitably so and certainly not to the
same extent . It depends on how hard you work
14
Response Rates Selected findings
Governments 5 year plan for Queensland women
15
Response Rates The current state of play
Overall only marginally lower than they were
several years ago or at least not substantially
lower. Certainly no precipitous decline. This
has been achieved by
Launched in 2003
Governments 5 year plan for Queensland women

• Extended call regimes
• Longer fieldwork periods
• Increased use of Primary Approach Letters
• Increased reliance on bilingual interviewing
• More sophisticated interviewer training
• Leaving messages on answering machines
• Increased reliance on survey sponsors to promote
  their surveys
• Creating an internet presence for surveys
• More calls to mobiles to obtain interviews with
  selected respondents

16
Population Research The future?

• Access to the Integrated Public Number Data Base
  is needed to overcome these deficiencies
• Mixed / multi modes of data collection
• On line access panels1
• Access to mobile phone directories

Governments 5 year plan for Queensland women

• Advocacy is needed
• Further research into advances in survey
  methodology is needed
• Survey methods are certain to adapt to the
  changed environment as the alternative is sub
  standard population research

1. 2006 Census, 58 of households had an internet
connection, 37 had broadband.
17
Population Research The future?
A poor prognosis for survey methods?
No, just continued innovation
18
CATI Computer Assisted Telephone
Interviewing
?
?
?? 
?
??
?? 
 __  
?? 
19
RDD Random Digit Dialing

• The proceedings from the first CATI Population
  Health Forum in 1998 record the following
  advantages of CATI and RDD
• CATI
• quick
• economical
• good response rates
• direct data entry
• computerised survey management system (SMS)
• Random Digit Dialling (RDD)
• reaches silent numbers and recent connections
• more disconnected, non-household, fax
• not completely random