The Survey "Burden" Factor How Important is Respondent's Perception of Survey Length

1
The Survey "Burden" Factor - How Important is
Respondent's Perception of Survey Length?
CASRO PANEL CONFERENCE February 2-3, 2009 New
Orleans, Louisiana
2
Agenda

• Introductions
• Study Design
• Traditional vs. Engaging Question Types
• Respondent Experience
• Effects of Survey Burden on the Data
• Conclusions
• Questions and Answers

3
Questions to Answer

• Do the types of questions (traditional vs.
  engaging) in a 30-minute survey have an effect
  on
• Survey Results
• Respondent Experience
• Respondent Behavior (The Survey Burden Factor)

4
Survey Design

• 30-minute survey among 1132 eRewards panelists,13
  to 64 years old, with age and gender quotas to
  ensure balanced representation.
• Survey consisted of screener, demographics and
  two question blocks with similar questions, one
  about Quick Service Restaurants (QSR) and the
  other about Casual Dining Restaurants (CDR).
• QSR and CDR blocks were rotated (approximately
  half received QSR questions first and half
  received CDR questions first).
• Approximately half of the respondents completed a
  traditional version of the survey and half
  completed the engaging version.
• The survey ended with survey experience questions
  and 107 completed online chats with iModerate.

5
Unaided Awareness Static vs. Triggered Boxes

• Traditional Engaging

6
Unaided Awareness - Results
Unaided Restaurant Awareness
Ease of Use
Enjoyment
Top 2
Top 2
Bottom 2
Bottom 2
of Restaurants Aware
7
Usage - Check Box Grid vs. Logo Card Sort

• Traditional Engaging

8
Usage - Results
Most Recent Dinner Visit
Ease of Use
Enjoyment
Top 2
Top 2
Bottom 2
Bottom 2
9
Logo Card Sort - Comments
I enjoyed the way that each section had a
different method of input. I get bored selecting
boxes all of the time. I especially liked
dragging to pictures into the boxes. It felt like
I was playing a game of solitaire instead of
answering a survey.

• I especially liked the questions where you moved
  the picture logo into the correct box.

I really liked being able to drag and drop the
answers into the appropriate buckets instead of
having to click in the circles - I always miss
some! I could change my mind on an answer without
a problem, and the survey didn't yell at me for
missing a box.
10
Attribute Grid Long vs. Short

• Traditional Engaging

11
Attribute Grid - Results
Combined Attributes
Ease of Use
Enjoyment
Top 2
Top 2
Bottom 2
Bottom 2
Straight-lining Speed (min)
12
Attribute Grids - Comments

• The large grid formatted questions are just not
  very easy to answer. More likely to click
  through it than other questions.

The grid like responses are not at all
interesting, though easy to answer.  It can be
very easy to simply run down the line with
responses without actually reading the questions.
I don't mind the bubbles...but there were
seriously like 30 different questions in a
line...I think if they were broken up into
groups...it wouldn't have felt nearly as
overwhelming...It reminds me of a bad scan-tron
test.
Well, it becomes tedious to look at a webpage
filled with grids of products and boxes or
experiences and boxes, in my opinion, by the time
you reach the bottom of the page, you tend to
care less about how accurate your response is.
13
Comparative Ratings Number Entry Grid vs. Logo
Slider

• Traditional Engaging

14
Comparative Ratings - Results

Combined Ratings
Ease of Use
Enjoyment
Top 2
Top 2
Bottom 2
Bottom 2
Straight-lining Speed (min)
15
Logo Slider - Comments

• What parts of the survey were most interesting?

When I was comparing, I believe,
KFC-McDonald's-Subway and I had to drag the logo
on a scale from 1 to 5 that answered the question
given.
I liked moving the sliders and sorting the
boxes. It was fun, easy and something different
It made the survey more like a game.
I believe the drag and drop inputs and brightly
colored brand logos with the companies name made
it more enjoyable to take this survey.
16
Importance Grid vs. Card Sort

• Traditional Engaging

17
Importance - Results
Combined Importance
Ease of Use
Enjoyment
Top 2
Top 2
Bottom 2
Bottom 2
Straight-lining Speed (min)
18
Survey Design Elements
Engaging Cell Discussion focused heavily on the
drag-and-drop and slider elements, which
respondents felt were interesting, different, and
easy-to-use.
Mod In the survey you mentioned you thought
this survey was better than most. What aspects
made you feel this way? Guest The
way they had the move the card sections instead
of just clicking bubbles It's more
interesting, I just enjoyed it better than a
drab bubble clicking marathon.
Cell 2 Engaging, Easy, Very long, Better than
most, Takes the average number per year, Female,
18 to 24 Guest The questions were easy with
pictures because I didn't have to read that much
I don't know.. maybe because I'm
lazy but its easier to just grab the pictures It
made this survey the most
enjoyable I've ever taken. Cell 2
Engaging, Easy, About right, Better than most,
Takes less than the average per year, Male, 13 to
17

• Traditional Cell Several respondents in this
  cell specifically remarked that the survey could
  be made more interesting if it offered some
  visual interest or interactivity.
• Guest Make the page visually pop. This one was
  just white. Add some graphical effects around the
  
• page. Curving lines, different
  colors, shapes. The most visually popping feature
  of this survey
• was the different colored text for
  the key words
• Cell 1 Traditional, Boring, Very
  long, Worse than some, Takes the average number
  per year, Male, 13 to 17

19
Respondent Experience

Length
Compared to All Other Online Surveys
20
Respondent Experience - Survey Length

• Traditional Cell While respondents in this cell
  often thought the survey was long, it seems that
  their impressions of length were based as much on
  their sense of tedium as on the actual time spent
  completing the survey.
• Guest It seemed to go on and on. . .I was
  irritated by similar type of questions over and
  over.
• Guest It was too long and I feel if it was
  broken up into fast food and another one on
  causal dining it
• would be less tedious. Fast food is
  not the kind of dining I do too often, only when
  I am in a
• time crunch.
• - Traditional, Tedious, Very long,
  Worse than some, Takes more than average number
  per year, Female,
• 50 to 64 years old

Engaging Cell Despite the diversion of the
interactive question format, respondents in this
cell still often felt that the survey was long.
However, because they werent bored by it, the
length was not particularly bothersome.
Guest It was a little longer than I expected,
but the variety seemed to keep it moving, so I
didn't notice the length as
much. Guest In any activity if your interest is
kept to a high level times seems to move more
quickly. By making the ways of
answering the questions so varied it kept my
interest level high and didn't give
me as much of a chance to become bored.
- Engaging, Interesting, About right, Better
than most, Takes the average number per year,
Female, 50 to 64 years old
21
Respondent Behavior (The Survey Burden Factor)
Average Length
Traditional
Engaging
22
Respondent Behavior (The Survey Burden Factor)
Straight-lining
of Unaided Brands
Traditional
Engaging
Straight-lined one or more of eleven rating
questions
23
Overall Respondent Feedback - Traditional
Experience

• Respondents who completed the traditional survey
  generally felt that the experience fell in line
  with what they expect from surveys . A
  significant portion of our discussions with them
  focused on the fact that the survey seemed
  repetitious and redundant, however the survey
  topic was interesting.
• Guest The sections which asked me to write
  every aspect of multiple restaurants were
  tedious it
• was incredibly repetitive, and it
  was boring having to answer the same question for
  every
• single restaurant.
• Traditional, Tedious, Very long,
  About the same, Takes the average number per
  year, Female, 13 to 17
• Guest I can tell you that I hate when questions
  seem to be repeated. To me it's not worth it
  when the
• survey asks the same thing over and
  over.
• Traditional, Easy, About right,
  About the same, Takes more than average number
  per year, Female, 25 to 34
• Guest Well, it becomes tedious to look at a
  webpage filled with grids of products and boxes
  or
• experiences and boxes, in my
  opinion, by the time you reach the bottom of the
  page, you tend
• to care less about how accurate your
  response is. When I am asked to type in a
  response, I
• tend to become more engaged.
• Traditional, Tedious, About right,
  About the same, Takes more than average number
  per year, Male, 25 to 34

24
Overall Respondent Feedback - Engaging Experience

• Those in the engaging survey believed that the
  interactive elements kept them focused despite
  the length and therefore allowed them to put more
  thought into their answers rather than completing
  the survey on autopilot.
• Guest Nice to have graphics and pictures to
  move instead of just check boxesMore interesting
  
• and kept my attention More
  interesting to be able to move boxes and logos
  around instead of
• just horizontal rows of check boxes.
• Guest I think it had a significant impact...it
  was a pretty long survey and if I wasn't kept
  interested I
• would have quit in the middle of it
  The varying formats kept me interested...a
  variety of ways
• to answer etc.
• Engaging, Enjoyable, Very long,
  Better than some, Takes less than average per
  year, Female, 35 to 49
• Mod Why do you think the variety of answer
  choices helped it from becoming boring?
• Guest Because of the different options - it
  wasn't repetitive More enjoyable than other
  similar
• surveys I have done.
• Engaging, Interesting, About
  right, Better than some, Takes more than average
  number per year,
• Female, 50 to 64
• Guest It wasn't the typical radio button check
  box and the additional ways of answering got me
  to think
• more about my answer. Which pile
  did I want to put those restaurants type of
  thoughts
• Engaging, Interesting, Very long, Better
  than most, Takes the average number per year,
  Male, 35 to 49

25
Conclusions

• The use of interactive questions can
• Change the way that some people answer
  questions.
• Decrease the number of inattentive respondents.
• Improve the respondent experience.
• Some respondents will become inattentive during
  long surveys, regardless of the survey experience
  you provide.

26
Recommendations

• When gathering unaided awareness, place separate
  boxes on the screen in order to gather the most
  responses.
• If survey is over 15 minutes, place tedious and
  repetitive tasks early in the questionnaire.
• Insert engaging question types throughout the
  survey to keep the respondents interested.
• Limit the use of grids, especially long ones that
  require respondents to scroll.
• Use brand logos for rating questions when
  possible helps the respondents connect the
  rating with the brand.

27
Thank You

• Questions?