A Tale of Two Methods: Comparing mail and RDD data collection for the Minnesota Adult Tobacco Survey III

1
A Tale of Two Methods Comparing mail and RDD
data collection for the Minnesota Adult Tobacco
Survey III

• Wendy Hicks and David Cantor
• Westat
• Ann St. Claire,
• ClearWay Minnesotasm
• Rebecca Fee,
• Blue Cross and Blue Shield of Minnesota
• Peter Rhode,
• Minnesota Department of Health

2
Overview of Presentation

• Report results of a feasibility study comparing
  mail and telephone data collection methods
• Seeming declines in RDD response rates (Curtin et
  al, 2005)
• Changes in coverage of the telephone population
  (Blumberg et al, 2006, 2007)
• Increasing costs associated with RDD surveys
• Findings mostly replicate those reported by Link
  et al. (2006)
• Some suggestion that an all adult selection
  method in a mail survey results in biased
  estimates for young adults (18-24 year olds)

3
Methodological Approach

• Mail survey has small sample and large confidence
  intervals
• Contrast results found in a parallel mail vs.
  telephone study for which mail study selected
  individuals rather than addresses
• Bring in similar findings from another pilot
  study (Health Information National Trends Survey,
  HINTS)

4
MATS Overview

• Minnesota Adult Tobacco Survey (MATS)
• ClearWay Minnesotasm
• Blue Cross and Blue Shield of Minnesota
• Minnesota Department of Health
• Major objective of MATS collect a diverse set of
  public health data about adult pop in MN,
  focusing on tobacco and cigarette use
• MATS samples from two frames
• RDD
• Blue Cross member list frame

5
Overview of MATS 2007 samples by mode and frame
Telephone (Feb June, 2007) Mail (March June, 2007)
General population samples targeted 7500 completes oversampled young adults African Americans selected one individual within household sampled 1200 addresses no oversampling all adults selected to respond
BCBS member list targeted 5000 completes oversampled young adults selected individual BC member sampled 1200 members no oversampling selected individual BC member
6
USPS Mail Data Collection

• All adult respondent selection mechanism
• Mailing package included 4 identical
  questionnaires and return envelopes
• Each adult living at this address should
  complete one of the enclosed questionnaires.
  Please give one of the enclosed questionnaires
  and an envelope to each person 18 years old or
  older living at this address.
• Some differences from the CATI instrument to make
  it more suitable to paper, self-administration

7
Response rates by frame and mode
MAIL TELEPHONE
General Population Frame (USPS and RDD-only) 31 - 33 41
BCBS Member List 56 48
8
Nonresponse and coverage error by mode

• 9.3 weighted estimate of cell-phone only
  households in the USPS survey
• Generally, the demographic distributions for mail
  and telephone respondents parallel one another
• Overall, more missing data in mail than telephone
• USPS vs. RDD-only respondents (differences of 3
  plus)
• USPS has fewer HS grads, larger underestimate
  relative to CPS
• USPS has fewer 65, smaller overestimate relative
  to CPS
• USPS has fewer 1-adult households and more 4
  adult households, closer distribution to CPS

9
Nonresponse and coverage error by mode

• Similar story for the BC member list, generally
  the distributions for mail and telephone parallel
• BC-M had fewer high school grads and some
  college
• BC-M had more married than phone respondents

10
Comparing survey estimates by mode

• Socially sensitive items (risk behaviors)
• Smoking prevalence,
• Binge drinking
• Factual items
• Exposure to media messages regarding tobacco use
• Workplace smoking policies
• Items assessing beliefs
• Harm in smoking an occasional cigarette
• Smoking increases comfort in social situations

11
Estimates of risk behaviors
Hypothesis mail gt reporting of risk behaviors
USPS RDD BC mail BC phone
Smoking prevalence (current smokers) 14.32 4.66 17.01 1.39 11.90 2.85 10.86 0.92
Binge drinking 19.80 4.78 15.44 1.21 21.68 3.78 14.56 1.03
12
Percent current smokers, by age, mode General pop


p gt0.03
13
Percent binge drinkers, by age, mode General pop.
14
Percent current smokers, by age, mode Member
frame
15
Percent binge drinkers, by age, mode Member frame
16
More suggestive evidence . . .

• Health Information National Trend Survey (HINTS)
• Small pilot
• USPS frame
• National sample
• all adults selection method
• Replicated the smoking result Estimates for
  18-24 year old mail respondents significantly
  lower

17
What about other types of items?

• Factual items
• Reported exposure to media messages
• Work place policy regarding smoking
• No differences by mode, for either frame
• No differences by mode for 18-24 years old
• Items assessing beliefs and attitudes
• Some support for age effect for one of the two
  items

18
Any harm in smoking an occasional cigarette? Gen.
pop.
19
Any harm in smoking an occasional cigarette?
Member list
20
Summary and Discussion

• Response rate roughly comparable
• Mail mode includes coverage of cell-only
  population
• Demographic comparison by mode-
• USPS (all adult) to RDD (next birthday) very
  similar distributions, some differences in educ,
  adults in HH,
• Member list also very similar, differences as
  anticipated
• Some suggestion that 18-24 year olds not
  representative for USPS all adult method
• Lower reports of risk behaviors in mail mode

21
Household type for 18-24 year olds, by mode

• In USPS sample
• Almost 80 live in family-type households
• About 6 live alone
• About 3 live in un-related households
• Only 13 of 18-24 year olds were cell-only
  households
• In the RDD sample,
• Almost 65 live in family-type households
• About 4 live alone
• Almost 25 live in un-related households

22
Next steps

• Incorporate methods to broaden type of young
  adult respondents
• Advance letter
• Incentive
• Fed-ex the second package
• Explore other respondent selection mechanism
• Next birthday method
• Other methods