Scale perception bias in Likert scale questions: Improved validity through anchoring vignettes?

1

• Scale perception bias in Likert scale questions
  Improved validity through anchoring vignettes?

• Klaus Gebel
• Centre for Chronic Disease Prevention
• James Cook University

2
German tourists in Spain
3
Political efficacy in China and Mexico
4
Health inequalities between Indigenous and
non-Indigenous Australians
5
Anchoring vignettesExample Self-rated health
SF-36 In general, would you say your health
is
Excellent
Very good
Good
Fair
Poor

• Most common measure of health
• Often negative association with actual health

6
Anchoring vignettes

• Differential Item Functioning (DIF)

High
High
Low
Low
7
Recoding depending on self-rating and rating of
vignettes
Self vignettes Recoding
xgtV1gtV2gtV3 7
xV1gtV2gtV3 6
V1gtxgtV2gtV3 5
V1gtxV2gtV3 4
V1gtV2gtxgtV3 3
V1gtV2gtxV3 2
V1gtV2gtV3gtx 1
xgtV1gtV2V3 7
xgtV1V2V3 7
8
Next steps

• Conversion from 7-point scale back to 5-point
  scale
• Paired t-test to compare DIF-adjusted vs
  unadjusted scores
• Walds test to compare associations of
  DIF-adjusted vs unadjusted scores with continuous
  or binary outcome measures

9
Design of vignettes

• Ideally equally spaced through distribution of
  self-assessments
• Avoid extremes
• Gender specific
• Ideally 2-3 vignettes
• Design vignettes in focus group
• Test them with sub sample

10
Example Safety from traffic and cycling / walking

• Safety and accessibility are the two most
  important environmental factors affecting
  activity participation across the lifespan.
• DiPietro 2012, Human Kinetics

11
Background
12
Background
13
Results

• Predictive validity of perceived safety from
  traffic

Very unsafe A little unsafe Neither safe nor unsafe A little safe Very safe
How safe do you think it is to ride a bicycle in your local area? 1 2 3 4 5
14
Methods

• Baseline data collected 2013
• Cycling and walking measured with travel diary
• Outcome measures
• Cycle to work/study y/n (Probit)
• Frequency of cycling (ordered Probit)
• Mins/week of cycling and walking (Poisson)

15
Results

• n871
• 58 women
• 18-55 (3711.1 years)
• 14 use bicycle as main way travel mode to
  work/study
• 29 cycle at least 1-2 days/week

16
Results

• Perceived safety from traffic

17
Results

• Ratings of vignettes

18
Results

• DIF adjusted perceived safety from traffic

19
Results

• Predictive validity of perceived safety from
  traffic

Unadjusted Adjusted for DIF Walds test p-value
Cycle to work/study y/n 0.63 0.79 0.03
Frequency of cycling 0.71 0.78 0.06
Mins/week of cycling 0.53 0.78 gt0.01
Mins/week of walking 0.48 0.62 0.01
20
Conclusion

• Significant scale perception bias
• Anchoring vignettes ? ? predictive validity
• Anchoring vignettes might be a powerful tool to
  improve validity of Likert-scale items

21

• Opportunistic evaluation of new bicycle paths in
  Cairns

22
Bike path evaluation in Cairns
23
Thank you for your attention! Questions? klaus.geb
el_at_jcu.edu.au