Evaluation II Cognitive Walkthroughs Catching up

1
Evaluation II / Cognitive Walkthroughs / Catching
up

• Loren Terveen
• CS 5115, Fall 2007
• October 17

2
Agenda

• Evaluation without users
• GOMS
• Cognitive walkthrough
• Catching up
• User types and personas
• Methods for gathering data from users

3
Hall of Fame/Shame

• Today
• Ben Pickhardt Brian Kuyper
• Weijia Deng Fernando Torre
• Next Monday
• Dan Rubin James Schwarzmeier
• Ryan Napier Ryan Ward

4
Not all mice are equal

• Ben Pickhardt
• Brian Kuyper
• 5115 Hall of Fame/Shame Fall 2007

5
Apple Mouse

• Provides feedback through sound/feel
• Poor forcing function
• Not obvious how to operate
• Clean look
• No wires to get in the way
• Only 1 button
• Lacks mapping at all

6
Logitech MX700

• Good constraints to force correct mouse operation
• Provides feedback through sound/feel
• Provides easy navigation
• Scroll Wheel
• Back/Forward Buttons
• Switch Application Button
• Conforms to cultural standards

7
Result

• Logitech
• Clearly a leader in the industry for good reason
• Provide for different users requirements
• Choices

• Apple
• One mouse does not fit all
• Usability more important then look and feel of
  mouse
• Too Simple

8
Is Logitech moving in the right direction?andIs
Apple Getting any better?
9
Hall of Fame/Shame Presentation

• Vending Machines
• Fernando Torre Weijia Deng

10
Vending Machine 1
11
Hall of Shame

• Conceptual model Simple. Enter the code for
  desired item. Problem People are used to
  combining numbers one by one. e.g. 1 and 0
  instead of 10.
• Visibility The codes of the products are small
  and hard to see.
• Mappings It is not clear sometimes if the code
  is for the above product or the lower product.
• Feedback You can see the chosen number on the
  screen and the chosen product dropping.
• Constraints This interface cant prevent you
  from choosing D1 instead of D10 if you want to
  get D10.
• Design for error Once you choose a product, you
  cant undo it.

12
Vending Machine 2
13
Hall of Less Shame

• Conceptual model Better. People are used to
  combining decimal numbers one by one like in
  telephones and ATMs. e.g. 1 and 0 instead of
  10.
• Visibility and affordance The labels are still
  small and hard to see.
• Mappings The same as vending machine 1.
• Feedback The same as vending machine 1.
• Constraints You will never confuse 1-1-0
  with of 1-10.
• Design for error There is a clear button you
  can press if you press a wrong number.

14
Vending Machine 3
15
Hall of Fame

• Conceptual model People buy exactly the same
  product as the choosing button shows. There is no
  gulf of execution.
• Visibility and affordance The pictures are easy
  to see and it is hard to make a mistake when
  choosing.
• Mappings Good mapping between the product and
  the icon button.
• Feedback You can see the chosen product after it
  drops.
• Design for error Once you choose a product, you
  cant undo it. Recommendation the vending
  machine could have a redo or confirm button.

16
Evaluation methods
17
Evaluation may be done with or without users

• Users arent perfect
• User time is valuable find/solve obvious
  problems so you dont waste users time
• Some users provide great information, others
  little
• But users are necessary
• cannot be simulated perfectly
• So test both ways

18
Evaluation Without Users

• Quantitative Methods
• GOMS/keystroke analysis
• back-of-the-envelope action analysis
• Qualitative Methods
• expert evaluation
• cognitive walkthrough
• heuristic evaluation

19
GOMS

• Goals Operators Methods Selection
• Based on information processing model of human
  cognition

20
GOMS

• Formal action analysis
• accurately predicts task completion time for
  skilled users
• Break task into tiny steps
• keystroke, mouse movement, refocus gaze
• retrieve item from long-term memory
• Look up average step times
• tables from large experiments

21
Examples

• http//www.cs.umd.edu/class/fall2002/cmsc838s/tich
  i/printer/goms.html

22
GOMS

• Primary utility repetitive tasks
• Success story telephone operators
• Gray, W. D., John, B. E., Atwood, M. E. (1993).
  Project Ernestine Validating GOMS for predicting
  and explaining real-world task performance. Human
  Computer Interaction, 8(3), 237-309.
• Benefit can be very accurate (within 20)
• May identify bottlenecks
• Difficulties
• challenging to decompose accurately
• long/laborious process
• not useful with non-experts

23
Back-of-the-Envelope Action Analysis

• Coarse-grain
• list basic actions, e.g., at the level of a
  scenario
• each action is at least 2-3 seconds
• what must be learned/remembered?
• what can be done easily?
• documentation/training?
• Goal is to find major problems
• Example 1950s 35mm camera

24
Expert Evaluation

• Usability specialists are very valuable
• double-specialists are even better
• An inexpensive way to get a lot of feedback
• Be sure the expert is qualified in your area

25
Cognitive Walkthroughs
26
Cognitive Walkthroughs - I

• A task-oriented method of evaluating an interface
  without users
• A formalized way of imagining peoples thoughts
  and actions when they use an interface for the
  first time.
• Benefits of evaluation before user meetings
• Helps get rid of obvious problems that would
  waste your users time
• May catch problems that testing with a few users
  will miss

27
Cognitive Walkthroughs - II

• Goals
• evaluate choice-points in the interface
• detect confusing labels, icons, images or options
• detect likely user navigation errors
• Remember improvement, not defense
• Start with a complete TCUID scenario
• never try to wing it on a walkthrough

28
Best Approach

• Tell a Believable Story
• How does the user accomplish the task,
  action-by-action?
• Based on user knowledge and system interface
• Recall DOET principles (Is this visible? Is
  feedback clear? Is there a gulf of execution? )
• Work as a group
• dont partition the task
• Be highly skeptical
• remember the goal!
• Every gap is an interface problem

29
A Quick Example PGP

• Users do not understand the key metaphor
• Users do not understand the implications of
  signing
• Signature verification is not represented visually

From Why Johnny Cant Encrypt A Usability
Evaluation of PGP 5.0, Alma Whitten J.D. Tygar
30
Theory

• The user sets a goal to be accomplished with the
  system (for example, "check spelling of this
  document").
• The user searches the interface for currently
  available actions (menu items, buttons,
  command-line inputs, etc.).
• The user selects the action that seems likely to
  make progress toward the goal.
• The user performs the selected action and
  evaluates the system's feedback for evidence that
  progress is being made toward the current goal.

31
Empirical Support

• Subjects will try label-guided actions first
  before they experiment with direct manipulations
  of unlabeled objects.
• Providing few actions in the search set can help
  to narrow the search if labeling cannot be
  provided, or if criteria for a "good" label are
  difficult to establish.
• Users are reluctant to try atypical actions
• Users are reluctant to extend their search beyond
  the readily available menus and controls.

32
Benefits of a Cognitive Walkthrough

• Focus most on first experiences - learnability
• Easy to learn
• Can do early in the software cycle
• Surfaces and examines assumptions about what
  users might be thinking.
• Can identify controls that are obvious to the
  designer but not to the user
• It can suggest difficulties with labels and
  prompts
• It can help find inadequate feedback
• Can help find inadequacies in the spec

33
Shortcomings of Cognitive Walkthrough

• Is diagnostic, not prescriptive
• Focuses mostly on novice users (someone who has
  to figure it out, rather than someone who already
  knows)
• Relies on the ability of designers to put
  themselves in the users shoes

34
When to do a Cognitive Walkthrough

• Before you do a formal evaluation with your users
• Can be done on your own for small pieces of the
  whole
• Can do a walkthrough of a complete task as the
  interface develops

35
Cognitive Walkthrough How To - I

• Prototype interface (preferably LoFi)
• Task description
• Scenario written list of the actions to
  complete the task in the interface
• An idea of who the users will be and their
  characteristic (so you can tell believable
  stories)
• Personas may be useful

36
Cognitive Walkthrough How To - II

• For each action in the sequence
• tell the story of why the user will do it
• ask critical questions (recall 7 Stages of
  Action)
• Will users be trying to produce the effect?
  I.e., will they form the goal designers wanted
  them to?
• Will users see the correct control?
• Will users recognize that this is the control
  theyre after, i.e., that it will advance them
  toward their goal?
• Or will they select a different control instead?
• Will users understand the feedback? That is,
  will be they be able to tell that they achieved
  their intended goal or at least made progress
  toward it?

37
Looking Ahead

• Heuristic evaluation another non-user
  evaluation method however, this one isnt task
  centered

38
Catching up more on working with users
39
Deepening our understanding of users and their
role

• Who are the users?
• Primary users those who interact with an
  interface to do a task
• Broader definition anyone affected by primary
  users ability to perform their tasks or who
  influences requirements
• Managers
• Product testers
• Purchasing
• Designers
• Customer Reps
• Union Reps

40
Users vs. Stakeholders

• For your project, youll deal with primary users
• In the real world, youll work with the entire
  range of stakeholders
• But dont let organizational politics keep you
  away from the primary users
• Managers or marketers cant define requirements
  or tasks at least not successfully
• Involve primary users in the entire process
• Early input needed usability delayed is
  usability denied

41
Different types of users

• Characteristics ability, background, attitude
  towards computers
• System use
• Novices
• First-time users
• Knowledgeable but infrequent
• Experts
• Job role e.g., nurse, physician, medical-record
  maintainer, database administrator

42
Novices / First-timers

• Novices
• Little task or interface knowledge
• First-time users
• Knowledgeable about the task, but not the
  interface
• Goal get the job done
• Design approach
• Step-by-step prompting
• Constrained action
• Clear procedures
• Error recovery
• Feedback is crucial

43
Knowledgeable but infrequent

• They know the task and interface concepts in
  general, but may find it difficult to remember
  interface details
• Design approach
• Well-designed menus
• Consistency, e.g. of terminology
• Recognition over recall

44
Experts

• Power users
• Design approach
• Speed is a key quick responses
• Shortcuts
• Feedback should be brief and non-distracting
• Support for user-defined macros

45
Work contexts matter, too

• Physical dusty? noisy? vibration? light? heat?
  humidity? hands free?
• Social sharing of files, of displays, in paper,
  across great distances, work individually,
  privacy for clients
• Organizational hierarchy, IT departments
  attitude, user support, communications structure
  and infrastructure, availability of training

46
Personas

• Newer technique
• Research-driven
• Narrative
• Descriptions of model users
• Basis
• Cluster users by relevant attributes
• Identify clusters
• Create realistic representatives
• Force you to consider whether your design is
  appropriate

47
Example Persona

• Sara is a graduate student living in Minneapolis.
  She travels by plane about three times a
  yearabout half of that time for conference trips
  for her University research group. When she
  travels on her own dollar, she is very
  price-conscious, and wants to be sure to get the
  lowest price, even if that involves obscure
  routes or indirect trips. When she travels for
  the University, she is happy to let their staff
  make the arrangements.
• As a computer scientist, Sara knows all about
  search engines and other computer systems.
  Sometimes this makes her suspect that the system
  may be hiding the best fares.

48
Another Persona

• Patricia is a 31 year old accountant for a
  technical publisher who has used Windows for six
  years at the office. She is fairly competent and
  technical. She installs her own software she
  reads PC Magazine she has programmed some Word
  macros. She has a cable modem for her home PC.
  Shes never used a Macintosh. Theyre too
  expensive, shell say, you can get a 2.5 MHz PC
  with a gig of RAM for the price of

49
Yet Another Persona

• Nelson has been an English professor at Carleton
  College since 1975. Hes written several books
  of poetry and has been using computer word
  processors since 1980, but has only used two
  programs, WordPerfect and Microsoft Word. He
  doesnt care how computers work he stores all
  his documents in whatever directory they get put
  in if you dont know about directories.

50
Exercise in project teams, briefly discuss

• Key properties of your users
• Any issues with their context
• Challenges in working with your users
• Summarize by writing two personas