Usability Testing

1
Usability Testing

• Chris North
• cs3724 HCI

2
Presentations

• karen molye,
• steve kovalak
• Vote UI Hall of Fame/Shame?

3
Next

• Thurs
• Proj 2, final implementation
• McBryde 102
• Email code to Purvi, bring report to demo
• Presentations UI critique or HW2 results
• Next Tues adam hahn, hugh hockett
• Next Thurs matthew jaswa, jason bower

4
Review

• What are independent variables?
• What you vary
• What are dependent variables?
• What you measure perf time
• How do you analyze the results?
• T-test, anova

5
UI Evaluation

• Early evaluation
• Wizard of Oz
• Role playing and scenarios
• Mid evaluation
• Expert reviews
• Heuristic evaluation
• Usability testing - formative design
• Controlled experiments - summative measuring
• Late evaluation
• Data logging
• Online surveys

6
Usability Testing

• Data
• Objective observe users
• Subjective user opinion
• Qualitative non-numeric data
• Quantitative numeric data
• Steps
• Design experiment
• Run experiment
• Analyze data
• Back to UI design

7
1. Design Experiment

• Users
• Representative users from User Analysis
• 3-5 users quality not quantity (80 rule)
• Tasks
• Benchmark tasks has metrics, usability
  specification
• Informal tasks no metrics, more exploratory

8
Setup

• User instructions
• We are evaluating the system, not you!
• Legal consent forms
• Pilot test rehearsal

9
2. Run Experiment

• Lab (McBryde 102)
• Mobile lab

10
Process

• Roles
• Subject/User
• Facilitator instructs user
• Observers collect data
• Executor run the prototype, e.g. if faked
• Process
• Give user a task
• Observe
• Avoid interfering, hint if completely stuck
• 1 hour / subject

11
Data Collection

• Video tape
• Users screen
• Users keyboard and mouse
• Users face audio
• Verbal protocol think aloud
• Note taking, real time
• Critical incidents
• Time stamp
• Quantitative
• HCI metrics, like controlled experiment
• Post-session interviews
• Eye tracking, biometrics

12
3. Analyze Data

• Funny to watch
• stupid user!, thats developer Xs fault!,
  this sucks
• how can we redesign UI to solve that usability
  problem?
• Compare measures to usability specifications
• Identify problems
• Solve problems in order of importance

13
Cost-Importance Analysis

• Spreadsheet
• Importance 1-5 (task effect, frequency)
• 5 critical, major impact on user, frequent
  occurance
• 3 user can complete task, but with difficulty
• 1 minor problem, small speed bump, infrequent
• Ratio importance / cost
• Sort by this
• 3 categories Must fix, next version, ignored

Problem Importance Solutions Cost Ratio

14
Solutions

• Design principles and guidelines
• Brainstorming
• Study other similar designs
• Solutions suggested by users, experts
• NOT more training, documentation
• Small UI change vs. major redesign

15
The big picture

• Iterative re-design, re-test,
• Goal achieve the usability spec (like soft.
  eng.)
• Cost effective
• Find problems early before the architecture is
  finalized

16
Example Toms hardware

• Compare, buy computer hardware
• Find a p4 board max 150
• How to fix
• Tweak systems
• Improve sys perf?
• How to overclock celeron?