User Testing and Modeling

1
User Testing and Modeling

• ICS 205
• Chris Wesson
• Christina Wuerth
• November 14, 2003

2
User Testing

• What is User Testing
• Measuring performance of typical users doing
  typical tasks
• Goal
• Obtain objective performance data showing how
  usable a system is in terms of usability goals
  (ease of use, learnability)

3
User Testing

• Applied form of experimentation
• Tests whether product is usable by intended user
  group(s)
• Measures/Records
• Time to complete typical tasks
• Number and type of errors
• Routes users take through tasks (Web)
• Typically 6-12 users

4
User Testing

• Is part of Usability Testing
• Obsevations
• Questionaires
• Interviews
• User Testing

5
User Testing and Modeling

• Doing User Testing
• Experiments
• Predictive Models

6
Keys to Doing User Testing

• Control testing conditions
• Careful planning
• Same conditions for each participant
• Measurements correspond to test
• Explicit assumptions
• DECIDE framework

7
Q How Long Should User Testing Process
Take? A As little time as possible while still
yielding useful information.
On average most companies take more than 8 weeks
for user testing
Testing of small piece of well developed
procedures can be done in less than 1 week if
needed
8
Things to Consider in Planning a User Test

• What aspects of the product need to be improved?
• Do the participants in the study represent actual
  users of the product?
• What tasks should the participants perform?
• What information will you observe during the
  tests?
• How will you analyze the data you collect?
• What will you do with this information once it is
  analyzed?

9
User Test Planning Steps

1. Define goals and concerns
2. Decide who will participate
3. Recruit participants
4. Select and organize tasks to be tested
5. Create task scenarios
6. Decide how to measure usability
7. Prepare other materials for the tests
   (questionnaires, etc.)
8. Prepare the testing environment
9. Prepare the test team
10. Conduct pilot tests making changes where needed

10
The Testing Team

• The Roles
• Usability Specialists
• The usability specialists know what can be
  accomplished in a user tests and how a test
  should be planned. They should also understand
  the interface and potential problems users may
  have with it.
• Designers/Developers
• The designers and developers understand the
  product and what the user should be able to do
  with it.
• Technical Communicators
• Technical communicators know what aspects of
  the documentation should be tested and also see
  potential problems in how the product
  communicates to the users.

11
More User Test Team Roles

• Trainers
• The trainers understand the problems that users
  will have when learning to use the product.
• Marketing Specialists
• Marketing specialists know who the users will
  be.
• Helpdesk/Customer Service Reps
• Helpdesk and customer service reps know what
  problems users have had with earlier versions of
  the software.

12
Preparing for the Test

• 1. Schedule activities
• 2. Assign roles and responsibilities
• 3. Train team
• 4. Write out a test plan
• 5. Practice by running pilot tests
• Throughout the entire process remember to
• - track progress
• - document all decisions

13
DECIDEUser Testing Guideline

• D etermine goals
• E xplore questions
• C hoose paradigm and techniques
• I dentify practical issues
• D eal with ethical issues
• E valuate, analyze, present data

14
Determine Goals, Explore Questions

• User testing is best used for testing prototypes
  and working systems
• Goals can be broad (how usable?)
• Specific questions required to focus study (can
  task X be done in time Y?)

15
Choose Paradigm and Techniques

• User testing is part of Usability testing
  paradigm
• Data can be recorded using
• Video
• Interaction log
• User satisfaction questionnaires
• Interviews

16
Identify Practical IssuesDesign Typical Tasks

• Create set of completion tasks (finding a
  website)
• Choosing which tasks to test is critical
• Tasks generally last 5-20 minutes
• Tasks are often straightforward
• Complex tasks (create design, solve problem) are
  okay
• Start simple to build user confidence

17
Identify Practical IssuesSelect Typical Users

• Must know users characteristics
• Most important previous experience
• Use short questionnaire to identify users
• Equal number of males and females

18
Identify Practical IssuesPrepare Testing
Conditions

• Control testing environment
• Minimize outside influences/noises that could
  distort results

19
Identify Practical IssuesPlan How to Run Tests

• Create schedule and scripts for running tests
• Start with familiarization task
• Contingency plan for spending too much time on a
  given task
• Avoid long tasks and long testing procedure
• Session lt 1 hour

20
Deal with Ethical Issues

• Informed consent form
• Point out any
• One-way mirrors
• Video cameras
• Interaction logging

21
Evaluate, Analyze,and Present Data

• Performance measures recorded from video,
  interaction logs
• Only use simple statistics
• Maximum Minimum
• Mean Standard Deviation
• Allows evaluators to compare performance between
  systems and across tasks

22
Establishing Goals and Concerns

• Step 1 of the planning phase is to define goals
  and concerns.
• Q What is a goal?
• A A quantitative usability aim that can be
  stated as a declarative sentence
• example A user should be able to locate the
  print menu item in less than 5 seconds with less
  than 3 errors during first attempt.
• Q What is a concern?
• A A concerns are questions raised about the
  usability of the product these are often raised
  while planning for the tests.
• example While a user find the new error
  messages confusing?

23
Establish Which Goals and Concerns will be tested.

• Not all goals and concerns can be tested
  simultaneously.
• Example of 2 conflicting concerns
• 1. Will the user use feature X?
• 2. Will the user find X easy to use?
• These two questions cant be answered at the same
  time instead you must prioritize.

24
Using Concerns to Plan the User Test

• Use General concerns to decide who your
  participants should be
• Example
• Will new users find this product easy to learn?
• Make sure you test with some beginning users.
• Use specific concerns to decide which tasks
  should be performed.
• Example
• Will users be able to find the print command
  quickly? In less than 5 seconds?
• Have the users perform a task of printing.

25
Sources of Goals and Concerns

• 1. Task Analysis and Quantitative usability goals
• Example
• General Menus should be easy to navigate
• Quantitative Goal Users should find menu
  choice in less than 2 minutes with no more than
  2 wrong choices when first need it. Task
  should be done with no errors in less than 1
  minute after first attempt.

26
Sources of Goals and Concerns

• 2. Timely Issues Certain concerns arrive at
  different stages of development and should be
  resolved at that time.
• Example
• 1) Two different design philosophies that you
  cant decide betweenplan a user test at start of
  the project to test prototypes of each
• 2) There has been previous testing earlier in
  the project, now you want to test a new feature
  before continuing development.

27
Sources of Goals and Concerns

• 3. Heuristic Analysis and Expert Reviews
• a) Problems that have been predicted by
  heuristic analysis and expert reviews should be
  candidates for user tests.
• b) Problems predicted by designers, planners,
  developers, human factor specialists, technical
  communicators should be tested.
• c) Concerns found by reflecting on the product
  itself should be tested.

28
Sources of Goals and Concerns

• 4. Previous Tests
• Example
• iterative testing with rapid prototypes
• run several user tests with the same concerns
• Example
• large scale user tests that raise future
  concerns
• retest with same concerns to make sure changes
  improved the problem

29
Who Should be Participants?

• Participants must be like the people who will
  actually use the product.
• Develop a user profile then choose participants
  that fit the profile.

30
Developing a User Profile

• Base profile on
• General market research
• Analysis of customers of competitors products
• Focus group sessions
• Observing and interviewing prospective users

31
Developing User Profile

• 1. Think about relevant characteristics
• a) those that all users share
• b) those that might make a difference among
  users
• example
• shared characteristic
• users will be undergraduate university
  students
• differences
• users will have lots of computer experience
• users will have very little computer
  experience
• users will have registered for classes before
• users will have never registered for a class
  before

32
Developing User Profile

• 2. Decide which characteristics matter most in
  establishing usability
• experience and motivation contribute more towards
  and individuals understanding than education,
  income, age, etc.
• Good Factors to Consider
• 1. Work experience
• 2. General computer experience
• 3. Specific computer experience
• 4. Experience with this product
• 5. Experience with similar products

33
Developing User Profile

• 3. Think broadly about who the users will be
• Sample questions to reflect on
• 1. Not just those currently working, but what
  about new hires? What is the rate of turnover
  that the software will need to handle?
• 2. Dont limit yourself to the current
  marketwhat about growth, who might want to use
  this product in the future?
• 3. What other areas of the corporation may what
  to adopt this software?
• 4. Reflect on differences within a category of
  users. Older users versus younger users? Users
  who have done the job longer versus users that
  more adaptable to computer environments?

34
Developing a User Profile

• 1. Product Name Class Registration Software
• 2. General Characteristics of User Population
• University Students
• 3. Characteristics of Users Relevant to the test
• Year in School
• Computer Experience
• Previous Experience Registering for Classes

35
Developing a User Profile

• 4. Which Characteristics listed in 3 should all
  users in the test have in common how will you
  define them?
• All users should be students currently enrolled
  in a university
• There will undergraduate and graduate students

36
Developing a User Profile

• 5. Which characteristics listed in 3 will vary in
  the test and how will you define them?
• Computer Experience will vary.
• Novice lt 1 year experience
• Medium gt 1 year experience used lt 5 days a
  week regularly
• Expert gt 1 year experience used gt 5 days a
  week regularly

37
Select Subgroups for a User Test

• Subgroups are people who share specific
  characteristics important to the user profile
• Divide groups by one characteristic at a time.
• Example
• University Students
• Subgroup 1 Subgroup 2
• Undergraduates Graduates
• Subgroup 1a Subgroup 2a
• Undergraduates Graduates
• Little computer experience Little computer
  experience
• Subgroup 1b Subgroup 2b
• Undergraduates Graduates
• Lots of computer experience Lots of computer
  experience

38
Define Characteristics for Each Subgroup

• Define what you mean by characteristics
• Example
• novice less than 1 year experience with this
  product
• medium 1 year to 3 years experience with this
  product
• expert more than 3 years experience with this
  product

39
Qualify Characteristics of the Subgroups

• 1. The middle group is often omitted from the
  subgroups in user studies since most problems
  found here will be found in novice and expert
  studies
• 2. Select a range of participants in each
  subgroup.
• Example Subgroup of novices with lt 1 year
  experience
• Find users with 1 month, 6 months, and 11 months
  experience
• 3. Establish range of each subgroup
• set min and max for subgroup participants
• Example Subgroup of experts with gt 3 year
  experience
• min would be 3 years but perhaps you want max
  experience to be 5 yearsyou dont want someone
  whose a hacker even though they may have lots of
  experience

40
How Many Participants?

• Depends on
• 1. How many subgroups you need
• 2. Time and money constraints
• 3. How important it is to compute statistically
  significant results
• Most of the time all you will care about is
  inferential statistics

41
How Many Participants?

• 6 12 participants is the typical choice
• With a minimum of 3 individuals in each subgroup.
• Problems will show up across subgroups

42
Making the Most of a limited Number of
Participants

• 1. Decide which characteristics are the most
  important so that your subgroups will be most
  useful
• 2. Collect relevant information from participants
  via interviews or questionnaires to help account
  for differences that show up in results.
• Example one novice user performs faster than
  all the others, but you find out that he has used
  a similar product before
• 3. Select people in subgroups representing the
  full range of the group

43
Experiments

• User testing based on scientific experimentation
• Aim test hypothesis that predicts a relationship
  between variables

44
Variables

• Independent manipulated by researcher
• Often multiple independent variables
• System version, User experience
• Dependent affected by independent variable
• Common dependent variable
• Time to complete task, Number of errors

45
Assigning Participants to Conditions

• Three Experimental Designs
• Different participants for all conditions
• Same participants for all conditions
• Matched pairs of participants

46
Matched Participants

• Participants matched in pairs
• Based on specific characteristics (expertise,
  gender, etc.)
• Used when participants cannot perform in all
  conditions

47
Design Comparison
Design Advantages Disadvantages
Different Participants No order effects. Many participants needed. Individual differences can be a problem.
Same Participants Eliminates individual differences. Must counterbalance ordering effects.
Matched Participants No ordering effects. Reduces effects of individual differences. No guarantee subjects are match across all variables.
48
Data Collection

• Data should measure users performance
• Typical measures
• Response times
• Number of errors
• Time to complete task

49
Data Analysis

• Questions to ask
• Do data sets from two conditions look similar or
  different?
• Any extreme atypical values?
• If so, what does that mean?
• Graph data to display differences

50
Predictive Models

• Provide measures of user performance without
  testing users
• Useful when it is difficult to do user testing
• Estimate efficiency of different systems on
  different tasks
• Most common models GOMS family
• GOMS Model, Keystroke Level Model

51
GOMS Model

• Models cognitive processes used when interacting
  with systems
• G oals state the user wants to reach
• O perators cognitive processes, physical
  actions necessary to achieve goal
• M ethods learned procedures for accomplishing
  goals
• S election rules used to choose between
  available methods

52
Keystroke Level Model

• Variation of GOMS model
• Provides numerical predictions
• Uses a standard set of times for main types of
  operators used during a task
• Average time to perform certain tasks
• Compares times for task using different startegies

53
GOMS Pros and Cons

• Pros
• Allows comparative analysis of different systems
  easily
• Useful estimates to compare efficiency of
  different systems
• Works well for short, well-defined tasks

• Cons
• Only models tasks that involve routine tasks
• Designed to predict experts performance
• Difficult to predict for average users
• Only predictions about predictable behavior

54
Fitts Law

• Useful for planning button locations, size and
  proximity
• Predicts time to click on objects on a screen
• T k log2(D/S .5), k 100 msec
• T time to move hand to target
• D distance between hand and target
• S size of target
• Bigger target easier to reach

55
Summary

• User Testing is the core of Usability Testing
• Controlled laboratory-like conditions
• Control independent variable(s) to predict
  dependent variable(s)
• Experimental designs different participants,
  same participants, matched participants
• GOMS, Keystroke, Fitts can be used to predict
  expert performance