Evaluating Internet Based Cessation Programs II: Minnesotas QUITPLAN'COMsm Intensive Followup Study

1
Evaluating Internet Based Cessation Programs II
Minnesotas QUITPLAN.COMsmIntensive Follow-up
Study (IFUS)

• Sharrilyn Evered, Ph.D., Jessie Saul, Ph.D.,
  Annette Kavanaugh, MS, Michael Luxenberg, Ph.D.,
  Nathan Cobb, MD, Randi Lachter, MPH, Lawrence An,
  MD, Barbara Schillo, Ph.D., and Ann Wendling, MD
  MPH

2
Evaluation Team

• Minnesota Partnership for Action Against Tobacco
  (MPAAT)
• Jessie Saul, PhD, Barbara A. Schillo, PhD, Ann
  Wendling, MD, Randi Lachter, MPH
• Professional Data Analysts, Inc (PDA)
• Sharrilyn Evered, PhD, Michael Luxenberg, PhD,
  Annette Kavanaugh, MS
• QuitNet.com
• Nathan Cobb, MD Pat Milner, Dave Atkins, JD
• Consultants
• Lawrence An, MD, University of Minnesota

3
Background

• MPAAT launched quitplan.com (fall 03)
• Online six-month follow-up survey obtained low
  response rate (10)
• Quitters are more likely to respond, producing an
  inflated cessation rate
• Intention-to-treat assumes all non-respondents
  still smoking - underestimates cessation rate

4
Low Response Rates Produce Big Gaps between
Completer and ITT
Standard Follow-Up Survey
5
Evaluation Context

• Contracted with PDA to design an evaluation that
  would produce a valid cessation rate (i.e., high
  response rate)
• Targets 70 consent rate and 70 response rate
• Barriers distrust, reconnecting 6 months
  post-registration
• Adopted cafeteria approach

6
Evaluation Design

• Mixed mode survey with incentive
• Obtained consent at registration
• Mailed pre-notification letter, requesting
  updated contact information
• Emailed live link to take survey (two reminders
  3 7 days)
• After 12 days, phoned non-respondents (up to 25
  attempts)
• Respondents mailed 10 thank you check

7
Pre-IFUS (8/1/03 - 2/1/04) (N3,743)
Post-IFUS (4/14/04 - 8/28/04) (N1,319)
IFUS (2/2/04 - 4/13/04) (N1,294)
Invited (N1,139)
Consented (N685)
Declined (N454)
60.1
39.9
Responded (N535)
Did Not Respond (N150)
78.1
21.9
Online (N283)
Phone (N252)
52.8
47.1
Excludes 9 people who did not permit us to use
their responses for research purposes and 7
people who were ineligible for the study (under
18 or non-MN resident)
8
High Response Rate Closes the Gap between
Completer and ITT
IFUS
9
Design Tips

• Contact info update card 253 in postage 19
  people updated their info 18 completed the
  survey (3.4)
• Calling 25 times 90 in 10, 99 in 15
• SurveyMonkey.com low cost, control

10
Key to Our Success

• Getting commitment to participate in a scientific
  study at the outset
• Provide an incentive
• Pre-notification letter
• Highly trained and invested interviewers
• Careful wording of messages is crucial

11
How Well do the IFUS Respondents Generalize to
Other quitplan.com Registrants?

• Two Sets of Comparisons to Assess Bias

12
Groups Compared for Response Bias IFUS-era
Invited (N1,139)
Consented (N685)
Declined (N454)
60.1
39.9
Responded (N535)
Did Not Respond (N150)
78.1
21.9
Online (N283)
Phone (N252)
52.8
47.1
13
First Comparison IFUS-era

• Logistic regression to predict response status
  (respondent vs. non-respondent)
• sex, education level, race/ethnicity
• smoking intensity, time to first cigarette, stage
  of readiness, quit attempt past year, use of quit
  aids, where heard about website
• website features visited, post-registration login

14
Statistically Significant Relationships
IFUS-Era

• The following groups were more likely to be an
  IFUS respondent
• women
• those with some education post HS
• used the nicotine lozenge in the year prior to
  registration
• used more of the sites features (i.e, set a quit
  date, posted to a forum)
• logged in after registration

15
Groups Compared for Response Bias Pre/Post IFUS
Pre-IFUS (7/31/03 - 2/1/04) (N3,743)
Post-IFUS (4/14/04 - 8/28/04) (N1,319)
IFUS (2/2/04 - 4/13/04) (N1,294)
Invited (N1,139)
Consented (N685)
Declined (N454)
60.1
39.9
Responded (N535)
Did Not Respond (N150)
78.1
21.9
Online (N283)
Phone (N252)
52.8
47.1
16
Significant Relationships Pre/Post IFUS

• Overrepresented in IFUS
• women
• more educated
• used lozenge past year
• heard about the site on TV or from healthcare
  provider
• Logged in after registration

• Underrepresented
• tried to quit at least once in the past year
• heard about the site from newspaper or magazine

17
Summary of Response Bias Analysis

• Bivariate analyses revealed seasonal variation
  can account for some relationships
• Differences between IFUS respondents and others
  (not attributable to seasonal variation)
  education, gender, site usage

18
Great Response Rate and a Representative Sample!

• But Did We Sacrifice Internal Validity?

19
Groups Compared for Mode Bias Online vs. Phone
Pre-IFUS (8/1/03 - 2/1/04) (N3,743)
Post-IFUS (4/14/04 - 8/28/04) (N1,319)
IFUS (2/2/04 - 4/13/04) (N1,294)
Invited (N1,139)
Consented (N685)
Declined (N454)
60.1
39.9
Responded (N535)
Did Not Respond (N150)
78.1
21.9
Online (N283)
Phone (N252)
52.8
47.1
20
Analysis

• Compared respondents on
• missing data
• sex, education level, race/ethnicity
• smoking intensity, time to first cigarette, stage
  of readiness, quit attempt past year, use of quit
  aids, where heard about website
• website features used, post-registration login
• cessation and satisfaction outcomes

21
Results Demographics, Clinical Characteristics,
and Site Use

• Percent of cases with complete data on every
  variable did not vary by mode
• These groups were overrepresented among online
  respondents
• women
• those in action or maintenance stage at
  registration
• those who used the site more

22
Results Cessation Outcomes
23
Results Attitude Items
24
Mode Bias Summary

• Cessation outcomes do not vary by mode
• Some attitude items vary, but not in a consistent
  direction
• Mode confounded with an early response, so
  results are difficult to interpret

25
Conclusions

• Mode bias suggests attitude questions may be less
  reliable with this design
• Mixed mode design with incentive can produce a
  high response rate
• High response rate generates a more
  representative sample and closes the gap between
  completer and ITT rates
• Web-based cessation programs can be meaningfully
  evaluated

26
For more information

• About the design, contact me at
• sharrilyn_r_evered_at_bluecrossmn.com
• 651-662-9383
• About the results, contact Jessie Saul at
• jsaul_at_mpaat.org
• 952-767-1415