Comparative Usability Evaluation of Usability Test Sample Size

1
Comparative Usability Evaluation of Usability
Test Sample Size

• Ray Nosko
• IDIA 750.185
• Humans, Computers, and Cognition
• December 10, 2002

2
Purpose

• The goal of this study is to apply comparative
  usability testing techniques to the question of
  usability test sample size.
• The study seeks to add to the common, shared
  knowledge of the usability community.

3
Background

• What is a comparative usability study?
• There are two basic types
• One lab conducts several studies
• Multiple labs conduct independent studies of the
  same application.

4
CUE-1

• Comparative Evaluation of Usability Tests
  (CUE-1) was conducted by Rolf Molich, et al. in
  1998.
• CUE-1 was the first well-known comparative study
  of usability testing.

5
CUE-1

• Four independent usability labs tested a Windows
  calendar program.
• In total, 141 usability problems were identified.
  Of these, only 1 problem was found by all four
  labs. 128 problems were only identified by one
  lab.

6
CUE-2

• A second comparative usability test was conducted
  in 1999 by Molich, et al.
• This study showed dramatic differences in the
  practices of professional usability labs.

7
CUE-2

• Seven professional labs and one student team
  tested hotmail.com.
• In total, over 300 problems were identified. No
  single lab found more than 75 of the problems.
  Most labs found less than 25.
• Report formats varied from 5 to 55 pages.

8
Why You Only Need 5 Users

• In 2000, Jakob Nielsen published an article
  asserting that you only need to test with 5 users
  to find 85 of the usability problems of a web
  site.
• Nielsen based this on a formula that he and
  Landauer did the the early 90s.

9
Why You Only Need 5 Users

• N ( 1 - ( 1 - L )n )
• Given 100 errors, 84.43 are discovered after
  five users.

10
Why You Only Need 5 Users
11
Why You Only Need 5 Users

• Spool Schroeder, 2001 Woolrych Cockton, 2001
  conducted studies that question the validity of
  Nielsens assertion.
• Nielsen does not take into account the level of
  severity of problems, the type of site, the size
  of the site, or the practices of individual labs

12
The Proposed Study

• The proposed study will use the same techniques
  as the CUE-1 and CUE-2 studies.
• Multiple labs will run tests to related to the
  number of users required and the number of
  problems uncovered.

13
So What?

• This study will add to the common knowledge base
  of the usability community.
• This study will serve as a benchmark for further
  studies in the future and will serve as a test of
  the studies presented here.