Highfidelity or Lowfidelity, Paper or Computer Choosing attributes when testing web prototypes

1
High-fidelity or Low-fidelity, Paper or
Computer?Choosing attributes when testing web
prototypes
University of California Berkeley
G r o u p f o r User Interface Research

• Miriam Walker
• Leila Takayama
• Professor James Landay

2
Outline

• Motivation
• Experimental testing of prototypes
• Results and conclusions
• You can fix it now on the drafting board
• with an eraser, or you can fix it later
• with a sledgehammer
• Frank Lloyd Wright

3
Practical prototyping

• Prototypes as tools for design
• Track changes is easier on computer than paper
• Designers invest less time and ego in
  low-fidelity prototypes
• Prototypes as tools for usability testing
• Computer prototypes allow remote testing
• Computer prototypes have more realistic
  interactions

4
Website prototyping tools

• Prototype functionality depends on tools
• Prototyping tools can be
• Low-fidelity or high-fidelity
• Paper medium or computer medium
• Research and practical considerations
• should drive the selection of prototyping tools

5
Dreamweaver Familiar high-fidelity, computer
prototyping tool

6
Paper Familiar low-fidelity, paper prototyping
tool
Post-its Scissors Pens Tape Transparencies Rulers
Cardboard Foam-core
(e.g. Rettig, 1994)
7
DENIM A low-fidelity, computer prototyping tool
(Lin, Newman, Hong, Landay, 2000)
8
Why would fidelity and medium affect user
testing?

• Fidelity and medium change interaction
• Example text-entry is handwritten or typed
• Colors in high-fidelity direct attention
• Fidelity and medium may alter the users views
  on
• Functionality of prototype
• Causes of and solutions for usability problems
• Ability of users to have an impact on design
  (Hong et al, 2001)

9
Outline

• Motivation
• Experimental testing of prototypes
• Results and conclusions

10
Making early stage prototypes
Sketched with paper and pens
Scanned in paper pages
Printed screens
Coded in HTML
11
Low-Fidelity Prototype
12
High-Fidelity Prototype
13
Experimental design

• Participants were unaware of the experimental
  hypotheses
• Each participant saw either low-fidelity or
  high-fidelity websites on both paper and computer

14
Testing Methods

• Faked prototype functionality by constraining
  tasks
• Sign-up for online banking services
• Email a checking account statement
• Calculate value of foreign currency
• Asked participants to think aloud
• Recorded their comments, and took copious notes
• Gave participants minimal assistance
• Followed up user tests with more questions

15
Outline

• Motivation
• Experimental testing of prototypes
• Results and conclusions

16
Analysis methodeffective usability testing

• More problems - six comments on one issue vs. one
  comment on each of six issues
• Most severe problems
• All types of problems e.g. consistency, feedback
• Level of detail - information architecture
  problems, widget problem

17
Quantitative analysis process

• Ratings
• Issue severity
• Issue heuristic category (Nielsen, 1994)
• Comment scope
• (widget, page, website)
• Counts
• Issue
• Comments

issue
issue
issue
Quantitative statistical analysis
18
Issues
Issue confusion between scheduled, single, and
recurring on bill payer

• Comments
• I would like recurrent paymentsno scheduled. I
  dont see the point, I dont see the difference
  between these two but um..
• Payment. Oh, actually, it would be recurring.
  Im trying to decide if its a scheduled payment
  or recurring payment.
• Oops. Recurring. Then the single payment would
  be like a scheduled? How would Im just trying
  to figure out what the difference would be
  between the two.

19
Quantitative Analysis of Results

• 1270 comments and 169 issues
• Low-fidelity vs High-fidelity
• No significant differences in number of comments
  or issues
• Paper vs. computer
• Average of 5 more comments about computer
  prototype (Wilcoxon signed ranks test, p 0.015)
• Issues no significant difference

20
Results

• No differences in severity of issues found
• No differences in scope of issues
• Differences between fidelities but not media
  categorizing issues by Nielsens heuristics
• (Chi Squared, p
• Only 10 of comments mentioned aesthetics
• Classifying issues using Nielsens Heuristics is
  difficult

21
Conclusion

• Fidelity and medium do not seem to affect
  quantity of problems found by user testing
• Prototyping techniques should be chosen by
  considering
• Need for remote testing
• Importance of recording design process
• Keeping designs at a level of detail appropriate
  to the stage of design

22
G r o u p f o rUser Interface Research
University of California Berkeley

• Special thanks to
• Corey Chandler
• Jason Hong, James Lin, and Francis Li
• Sarah Waterson
• Professor Rashmi Sinha
• Ten anonymous expert raters
• Miriam Walker - mwalker_at_cs.berkeley.edu
• Leila Takayama - leila_at_cs.berkeley.edu
  http//guir.berkeley.edu/projects/fidelity

23
Question Time!