What happens if you offer a mobile option to your web panel? Results from a pilot study comparing a desktop and mobile device survey

1
What happens if you offer a mobile option to
your web panel?Results from a pilot study
comparing a desktop and mobile device survey

• Vera Toepoel
• (Utrecht University, V.Toepoel_at_uu.nl)
• Peter Lugtig
• (Utrecht University/University of Essex,
  p.lugtig_at_uu.nl)

2
Vast increase in mobile phone use

• According to Statistics Netherlands, 50 of all
  12 to 72 year olds with Internet access do this
  via a mobile device in 2011
• Smartphone penetration rate explosion
• In 2011 1 out of 5 people opened their email on a
  smartphone (source Emerce)

http//www.emerce.nl/research/mobiel-emails-lezen-
2011-verdubbeld
3
Smartphone as research tool?

• Online probability based panel of Market Response
• RDD, Internet population
• Panel members ask if they can complete the survey
  on their mobile phone
• Young panel members (age 18-34) have the lowest
  response probabilities in the panel and are
  continuously underrepresented in the panel
  (reasons for non response dont feel like it, no
  time)
• Current software not capable of using mobile
  friendly design
• Confirmit software offers responsive design
  (survey automatically optimized for the device
  chosen)

4
Research Design Two parallel surveys

• June 2012 two surveys with similar sampling frame
  
• (NL, 18, Internet users, mobile registration)
• Standard QSL (current online method)
• Confirmit with response design (respondent could
  chose prefered device)
• Soft check for mobile completion

Invitation QSL
Confirmit
5
Research Questions What happens if you offer a
mobile option to your web panel?

• Amount of people using mobile
• Substantial?
• Socio-demographic characteristics
• Coverage bias age, edu, gender, urbanization, hh
  sizecomposition
• Response (break offs and item non-response)
• GPS
• Evaluation
• Response quality (measurement error)
• Questionnaire already adapted to mobile (no
  experiment)
• Open-ended questions, check-all-that-apply,
  primacy

6
Research Questions Amount of people using mobile

• 57 used mobile phone throughout the survey
  (N443)
• 252 used mobile (57)
• 33 people switched from desktop to mobile during
  the survey (7, 12 of mobile starters)
• 4 people switched from desktop to mobile (1)
• 154 people used desktop (35)
• Note we define mobile as smart phone only.
  Additional analyses with mobileall with
  touchscreen (incl. ipads etc.) produces similar
  results.

7
Research Questions Socio-demographics

• Age, hh size and hh composition (kids)
  significant predictors of mobile use
• Older people
• Larger households
• No kids
• No effect of urbanization, gender, social class,
  education, household position, family income

8
Hard-to-reach-groupYoung mobile increases
response probability

9
Research Questions Response

• Break offs limited
• Short questionnaire
• 4 out of 285 break off on mobile
• 3 out of 153 on desktop
• No evidence that mobile results in higher break
  offs
• Item non-response
• 7 over 3 questions
• No significant difference mobile versus desktop
• No significant difference in non-substantive
  answers (dont know)

10
Research Questions time to complete
11
Research Questions Day questionnaire completed
12
Research Questions GPS
Mobile Desktop Total
GPS ok 38 8 25
N 253 190 443
Mobile
Home 67
Work 14
On the road 11
outside 6
abroad 2
 N110  
13
Research Questions Evaluation
Mobile Desktop Total confirmit survey qsl
Positive (8-10) 42 44 43 38
Neutral (6-7) 50 47 49 55
Negative (1-5) 8 9 8 7

• n(mobile)281, n(desktop)151, N(QSL)498
• Chi-square test across mobile and desktop not
  significant
• Chi-square test across all three groups not
  significant

14
EvaluationHigher evaluation young people
7.7
18-24
7.1
25-34
7.2
7.5
35-44
7.2
7.1
45-54
7.2
6.9
55-64
7.2
6.4
65
15
Research Questions Response Quality open-ended
questions
Mobile Desktop Total (Confirmit)
No answer 35 33 33
0-50 characters 52 49 51
51-100 charcters 10 16 13
100 characters 3 3 3

• Notes N(mobile)285, N(desktop)154
• No significant difference

16
Research Questions Response Quality
check-all-that-apply
Mobile Desktop Total (Confirmit)
0 24.2 21.4 23.2
1 9.1 9.7 9.3
2 15.4 18.2 16.4
3 14.7 12.3 13.9
4 14.4 15.6 14.8
5 8.8 8.4 8.7
6 7.0 6.5 6.8
7 4.6 4.5 4.6
8 1.8 3.2 2.3
Mean number of shops visited 2.7 2.8 2.8
Number of shops visited in past year Notes
N(mobile)285, N(desktop)154 No significant
differences
17
Research Questions Response Quality primacy in
radio buttons
Mobile Desktop Total Confirmit
One a week or more often 3.9 0.6 2.7
About once a month 23.9 27.7 25.2
About every 3 months 37.2 32.1 35.4
About once every 6 month 21.4 26.4 23.2
Once a year o less often 10.2 9.4 9.9
Dont know 3.5 3.8 3.6
Notes N(mobile)285, N(desktop)154 Frequency of
buying clothes in a shop nearby No sigificant
difference
18
Feedback mobile optionPerfect format, excellent
speed! Super!
The scrollbar was not working properly. Did not
like it. it is easier on desktop(evaluation 3
woman, 58)
I am glad you figured out that desktop pcs are
getting used less often and that questionnaires
should be made mobile friendly. I dont need to
get my laptop (evaluation 8 woman, 29)
Looovvedddd it(evaluation 9 woman, 18)
I like this very much. I do not need to get my
laptop (evaluation 8 man, 20)
Good. But it should not take me too long,
otherwise Ill break off(evaluation 8 man, 51)
Good! I often complete surveys on my mobile phone
but this is the first one that reads well,
perfect format, excellent speed! Super!
(evaluation 9 woman 24)
Dont like it. I have a pc which I use for all my
survey activities. Next time I wont be doing the
survey(evaluation 1 man, 71)
Very good!!! I will definetely get higher
participation rates if this will be implemented
(evaluation 9 man, 32)
Handy! Better then the desktop since I tend to
forget the survey invitations (evaluation 8
woman, 38)
19
Conclusion

• Considerable amount (57) uses mobile
• No effect on non-response
• No effect on response quality
• Similar evaluation
• GPS gives additional insights
• No reason to believe that mixed-device is a
  problem
• Able to attract hard-to-reach group such as young
  people

20
Discussion

• What about difficult questionnaires?
• Long
• grids
• What about privacy?
• How to offer additional information to
  researchers?
• Coverage bias?

Thank you!