Increasing Survey Cooperation: Motivating Chronic Late Responders to an Annual Survey

1
Increasing Survey Cooperation Motivating Chronic
Late Responders to an Annual Survey
National Science Foundation
Division of Science Resources Statistics
Ronda Britt and Fran Featherston ICES III June
19, 2007 National Science Foundation Division
of Science Resources Statistics www.nsf.gov/statis
tics
2
NSF Academic RD Survey
National Science Foundation
Division of Science Resources Statistics

• Voluntary survey of 700 research development -
  performing universities and colleges
• Conducted annually since 1972
• Web only since 2001
• Requests expenditures for RD performed during
  previous year
• Survey data based on respondents fiscal year

3
Historical Response Pattern
National Science Foundation
Division of Science Resources Statistics

• Early November email launches survey sets
  January 31 deadline
• 50-60 respond by deadline, despite multiple
  reminders
• Several months after deadline to reach response
  rate of 95
• Respondents were allowed to extend deadline,
  creating cycle of late response

4
Goals of Experiments FY 2004 - FY 2005
National Science Foundation
Division of Science Resources Statistics

• Increase response rate at survey deadline
• -Speed up first interaction with web survey
• -Increase number of reminders
• -Vary mode of reminders
• Decrease number of weeks to reach 95 response
  rate

5
Experiment 1- FY 2004Increase Response Rate at
Survey Deadline, Part 1
National Science Foundation
Division of Science Resources Statistics

• Hypothesis Respondents who visit website soon
  after launch (acknowledge survey) are more likely
  to respond by the survey deadline
• 3 study groups received varying modes of
  reminders
• Group 1 received one e-mail reminder in December
  (control group)
• Group 2 received two e-mail reminders in December
• Group 3 received 2 phone calls in December
• All groups received final e-mail in January
  requesting acknowledgement

6
Experiment 1 Design
National Science Foundation
Division of Science Resources Statistics
Key Dates Study Groups Study Groups Study Groups
Key Dates Group 1 (control) Group 2 Group 3
November 23 Survey Launch Survey Launch Survey Launch
December 14 Acknowledgement e-mail Acknowledgement phone calls
December 30 Acknowledgement e-mail Acknowledgement e-mail Acknowledgement phone calls
January 11 Acknowledgement e-mail Acknowledgement e-mail Acknowledgement e-mail
January 31 Survey Due Date Survey Due Date Survey Due Date
7
Experiment 1 Results
National Science Foundation
Division of Science Resources Statistics

• You can lead a respondent to the website, but you
  cant encourage an earlier survey completion
• Multiple reminders to acknowledge succeeded in
  decreasing elapsed days until acknowledgement
• Multiple reminders to acknowledge had no effect
  on decreasing elapsed days to survey completion

8
Experiment 1 Results
National Science Foundation
Division of Science Resources Statistics
Control Group 1 E-mail reminders Group 2 Phone Reminders Group 3 All Groups
Mean days from survey launch to acknowledgement 37 days (n177) 29 days (n179) 30 days (n179) 32 days (n535)
Response rate at survey deadline 46.3 49.7 49.2 48.4
9
Experiment 2 FY 2004 Increase Response Rate at
Survey Deadline, Part 2
National Science Foundation
Division of Science Resources Statistics

• Hypotheses
• Respondents who receive more reminders are more
  likely to respond by the survey deadline
• Respondents who receive reminders that vary in
  mode are more likely to respond by the survey
  deadline
• During month prior to survey deadline
• Group 1 One e-mail reminder (control group)
• Group 2 One e-mail, one mail reminder
• Group 3 Two e-mail reminders

10
Experiment 2 Design
National Science Foundation
Division of Science Resources Statistics
Key Dates Study Groups Study Groups Study Groups
Key Dates Group 1 Group 2 Group 3
January 18-20 (expected delivery) Reminder Mailing
January 19 Reminder E-mail 1 Reminder E-mail 1
January 26 Reminder E-mail 2 Reminder E-mail 2
January 31 Survey Due Date Survey Due Date Survey Due Date
11
Experiment 2 Results
National Science Foundation
Division of Science Resources Statistics

• Variety and intensity will spice up the response
  rate.
• Group who received both mail and e-mail responded
  at a higher rate by deadline than group receiving
  only e-mail
• Experiment 4 (FY 2005) validated the effect of
  two contacts vs. one, but no significant
  difference between two e-mails vs. e-mail and
  mailing

12
Experiment 2 Results Single mode, 1 contact
vs. mixed mode, 2 contacts
National Science Foundation
Division of Science Resources Statistics
Group 1 1 e-mail Group 2 1 e-mail and 1 mailing
Responded by deadline 35.4 45.9
51 62
Total 100 100
144 135
13
Experiment 3 FY 2004Increase Timeliness after
Survey Deadline
National Science Foundation
Division of Science Resources Statistics

• Hypothesis Respondents will respond sooner after
  the deadline if follow-up messages are
  individualized and less frequent than weekly
• During 24 weeks after survey deadline
• Group 1 Low tailoring standard weekly messages
  (control group)
• Group 2 Medium tailoring less frequent
  messages with different content each time
• Group 3 High tailoring adjusted to individual
  respondent history (more frequent and tailored
  contacts for previous non-responders)

14
Experiment 3 Design
National Science Foundation
Division of Science Resources Statistics
Follow-up Period Group 1 Low Tailoring Group 2 Medium Tailoring Group 3 High Tailoring Group 3 High Tailoring
Likely non-responders All others
0-11 weeks 10 e-mail or phone contacts Same messages 3 e-mails Varied messages Dependent on past behavior Tailored messages 2 e-mails Same messages
12-24 weeks 12 contacts alternating e-mail or phone weekly 4 contacts spaced 3 to 4 weeks e-mail, phone, letter, and final phone call 6 contacts alternating e-mail and phone every two or three weeks 6 contacts alternating e-mail and phone every two or three weeks
Same messages Varied messages and senders Same messages Same messages
15
Experiment 3 Results
National Science Foundation
Division of Science Resources Statistics

• It is more important to stay in contact than it
  is to use a particular contacting strategy
• Little variation of final response rate across 3
  groups
• Weekly reminders were not found to perform better
  than tailored reminders sent every 3 to 4 weeks

16
Experiment 3 Results
National Science Foundation
Division of Science Resources Statistics
Group 1 Low Tailoring Group 2 Medium Tailoring Group 3 High Tailoring All Groups
Number of respondents 105 95 102 302
Response rate on January 31 43.9 49.2 45.5 46.2
Final response rate 90.5 94.7 90.2 91.7
Response added between January 31 and August 10 50.8 48.1 49.2 49.3
17
Changing Procedures
National Science Foundation
Division of Science Resources Statistics

• Used experimental results as starting point for
  making changes
• Consulted with Dr. Don Dillman on message
  crafting and changing survey procedures
• Determined four areas for change
• Acknowledgement process
• Mode/frequency of contacts
• Extension policy
• Survey close-out policy (letters to institution
  presidents)

18
Acknowledgement Process
National Science Foundation
Division of Science Resources Statistics

• Simplified acknowledgement process in FY 2006
  less burdensome for respondents
• Prior to FY 2006, asked respondents to log-on to
  survey website to acknowledge continuation as
  survey respondent
• Now, respondent replies to e-mail
• Prior to FY 2006, 23 acknowledged in first week
  of survey, 72.5 month before due date
• Now, increased to 63 and 98.7, respectively
• Results Earlier identification of changes in
  respondents

19
Mode/frequency of Respondent contacts
National Science Foundation
Division of Science Resources Statistics

• Changed mode and frequency, varied content in FY
  2006
• Pre-due date reminders increased in frequency
• Early December additional e-mail reminder
• January - 2 reminders (1 mail and 1 e-mail)
• Post-due date follow-ups decreased in frequency
• First 6 weeks - 3 e-mails (every 2 weeks)
• 7th week - phone call
• 8th week - letter to presidents office
• 9th week phone call
• 11th week - final e-mail announcing survey
  closing date
• Results Increased response rate at deadline from
  56 in FY 2005 to 68 in FY 2006, shortened
  post-deadline follow-up period

20
Extension Policy
National Science Foundation
Division of Science Resources Statistics

• Eliminated unlimited extensions
• Prior to FY 2005 Extension requests up to 36
  weeks after survey due date
• FY 2005 and FY 2006
• Requests for extensions restricted to shorter
  time frame (only granted for dates before
  Presidents letter)
• After presidents letter mail-out, respondents
  were told survey closes when target response
  rate reached and 2 week final warning would be
  given
• Results Shortened post-deadline follow-up period

21
Survey Close-out Policy
National Science Foundation
Division of Science Resources Statistics

• Presidents letter was added in FY 2004 as final
  call for response
• Goal set to mail Presidents letter to
  nonrespondents 1 month earlier each year
• FY 2004 - May 25 (87 response rate at mailing)
• FY 2005 - April 21 (88 response rate at mailing)
• FY 2006 - March 23 (92 response rate at mailing)
• Results Earlier survey closeout on May 21, 2007
  v. August 10, 2005

22
Response Rate History, FY 1997- FY 2006
National Science Foundation
Division of Science Resources Statistics
Weeks to reach final response rate
Weeks to reach 88
Weeks to reach 80
23
Final Response Rates, FY 1997- FY 2006
National Science Foundation
Division of Science Resources Statistics
24
Questions?For more information
contactRonda Britt, Academic RD Survey Manager
703-292-7765rbritt_at_nsf.gov
National Science Foundation
Division of Science Resources Statistics