1 Action Analysis 2 Automated Evaluation

1
(1) Action Analysis (2) Automated Evaluation

• CS 160, Spring 2002
• James Landay
• April 15, 2002

2
Hall of Fame or Hall of Shame?

• java.sun.com

3
Hall of Fame

• Good branding
• java logo
• value prop
• Inverse pyramid writing style
• Fresh content
• changing first read
• news in sidebar

4
Hall of Fame or Hall of Shame?

• Bryce 2
• for building 3D models

5
Hall of Shame!

• Icons all look similar
• what do they do????
• How do you exit?
• Note
• nice visuals, but must be usable
• What if purely for entertainment?

6
(1) Action Analysis (2) Automated Evaluation

• CS 160, Spring 2002
• James Landay
• April 15, 2002

7
Outline

• Review
• Action analysis
• GOMS? Whats that?
• The G, O, M, S of GOMS
• How to do the analysis
• Announcements
• Automated evaluation tools

8
Review

• What is a Serif what is it for?
• the curly feet at the top bottom of fonts
• give the eye a line to read by A Serif Font
• Sans Serif means without Serif
• What are some signals used w/ text?
• type size
• type weight
• word spacing
• line length
• leading (line spacing)
• type shifts

9
Action Analysis Predicts Performance

• Cognitive model
• model some aspect of human understanding,
  knowledge, intentions, or processing
• two types
• competence
• predict behavior sequences
• performance
• predict performance, but limited to routine
  behavior
• Action analysis uses performance model to analyze
  goals tasks
• generally done hierarchically (similar to TA)

10
GOMS Most Popular Action Analysis

• Family of UI modeling techniques
• based on Model Human Processor
• GOMS stands for (?)
• Goals
• Operators
• Methods
• Selection rules
• Input detailed description of UI/task(s)
• Output qualitative quantitative measures

11
Quick Example

• Goal (the big picture)
• go from hotel to the airport
• Methods (or subgoals)?
• walk, take bus, take taxi, rent car, take train
• Operators (or specific actions)
• locate bus stop wait for bus get on the bus...
• Selection rules (choosing among methods)?
• Example Walking is cheaper, but tiring and slow
• Example Taking a bus is complicated abroad

12
Goals

• Something the user wants to achieve
• Examples?
• go to airport
• delete File
• create directory
• Hierarchical structure
• may require many subgoals

13
Methods

• Sequence of steps to accomplish a goal
• goal decomposition
• can include other goals
• Assumes method is learned routine
• Examples
• drag file to trash
• retrieve from long-term memory command

14
Operators

• Specific actions (small scale or atomic)
• Lowest level of analysis
• can associate with times
• Examples
• Locate icon for item on screen
• Move cursor to item
• Hold mouse button down
• Locate destination icon
• User reads the dialog box

15
Selection Rules

• If gt 1 method to accomplish a goal, Selection
  rules pick method to use
• Examples
• IF ltconditiongt THEN accomplish ltGOALgt
• IF ltcar has automatic transmissiongt THEN ltselect
  drivegt
• IF ltcar has manual transmissiongt THEN ltfind car
  with automatic transmissiongt

16
GOMS Output

• Execution time
• add up times from operators
• assumes experts (mastered the tasks)
• error free behavior
• very good rank ordering
• absolute accuracy 10-20
• Procedure learning time (NGOMSL only)
• accurate for relative comparison only
• doesnt include time for learning domain knowledge

17
GOMS Output

• Ensure frequent goals achieved quickly
• Making hierarchy is often the value
• functionality coverage consistency
• does UI contain needed functions?
• consistency are similar tasks performed
  similarly?
• operator sequence
• in what order are individual operations done?

18
How to do GOMS Analysis

• Generate task description
• pick high-level user Goal
• write Method for accomplishing Goal - may invoke
  subgoals
• write Methods for subgoals
• this is recursive
• stops when Operators are reached
• Evaluate description of task
• Apply results to UI
• Iterate!

19
Comparative Example - DOS

• Goal Delete a File
• Method for accomplishing goal of deleting a file
• retrieve from Long term memory that command verb
  is del
• think of directory name file name and make it
  the first listed parameter
• accomplish goal of entering executing command
• return with goal accomplished

20
Comparative Example - Mac

• Goal Delete a File
• Method for accomplishing goal of deleting a file
• find file icon
• accomplish goal of dragging file to trash
• Return with goal accomplished

21
Comparative Example - DOS

• Goal Remove a directory
• Method for accomplishing goal of removing a
  directory -
• ?????

22
Comparative Example - DOS

• Goal Remove a directory
• Method for accomplishing goal of removing a
  directory
• Accomplish goal of making sure directory is empty
• Retrieve from long term memory that command verb
  is RMDIR
• Think of directory name and make it the first
  listed parameter
• Accomplish goal of entering and executing a
  command
• Return with goal accomplished

23
Comparative Example - MAC

• Goal Remove a directory
• Method for accomplishing goal of removing a
  directory
• ????

24
Comparative Example - MAC

• Goal Remove a directory
• Method for accomplishing goal of removing a
  directory
• Find folder
• Accomplish goal of dragging folder to trash
• Return with goal accomplished

25
Applications of GOMS

• Compare different UI designs
• Profiling (time)
• Building a help system? Why?
• modeling makes user tasks goals explicit
• can suggest questions users will ask the answers

26
What GOMS can model

• Task must be goal-directed
• some activities are more goal-directed
• creative activities may not be as goal-directed
• Task must be a routine cognitive skill
• as opposed to problem solving
• good for things like machine operators
• Serial parallel tasks (CPM-GOMS)

27
Real-world GOMS Applications

• Keystroke Level Model (KLM)
• Mouse-based text editor
• Mechanical CAD system
• NGOMSL
• TV control system
• Nuclear power plant operators associate
• CPM-GOMS
• Telephone operator workstation

28
Advantages of GOMS

• Gives qualitative quantitative measures
• Model explains the results
• Less work than user study no users!
• Easy to modify when UI is revised
• Research tools to aid modeling process since it
  can still be tedious

29
Disadvantages of GOMS

• Not as easy as HE, guidelines, etc.
• Takes lots of time, skill, effort
• Only works for goal-directed tasks
• Assumes tasks performed by experts without error
• Does not address several UI issues,
• readability, memorizability of icons, commands

30
Announcements

• Make sure your web sites are up to date
• I scanned last night and saw lots of material
  missing
• PowerPoint slides, all assignments, mailto link
  for team!
• We will start grading these soon
• I will schedule team meetings for Fri. Mon. to
  demo your system (turn in write-up on Fri.)
• I will email a midterm survey today please fill
  out so we can adjust
• PocketPC teams come to my office after lecture to
  pickup H/W THANK YOU MICROSOFT!!!!!
• Questions????

31
Rapid Iterative Design is the Best Practice for
Creating Good UIs
We have seen how computer-based tools can
improve the Design (e.g., Denim) Prototyping
(e.g., VB) phases
Design
Prototyping
Evaluation
32
Automated GOMS Tools

• Can save, modify and re-use the model
• Automation of goal hierarchy, method, selection
  rule creation

33
QGOMS tool
34
CRITIQUEHudson et al (1999)

• 1. Prototype system
• in this case with the SubArctic toolkit
• 2. Demonstrate a procedure (task)
• record events
• apply rules
• 3. Automatically generate KLMs
• 4. Semi-automatically generate classic GOMS
  models

35
Automated Web Evaluation

• Motivation
• Approaches
• GOMS inspired models
• remote usability testing

36
Factors Driving Repeat Visits Should Drive
Evaluation

• High quality content 75
• Ease of use 66
• Quick to download 58
• (Source Forrester Research, 1/99)

37
Max WebCriterias GOMS Model

• Predicts how long information seeking tasks would
  take on a particular web site
• Automated procedure
• seed with start page and goal page
• procedure
• reads page
• model predicts how long to find click proper
  link
• load time, scan time, and mouse movement time
• repeat until find goal page
• Claim time is directly related to usability

38
Sample of Maxs Reports
39
Advantages of Max-style Model

• Inexpensive (no users needed)
• Fast (robot runs then computes model)
• Can run on many sites compare -gt benchmarks

40
Disadvantages of Max-style Model

• Focus on time (much of it download time)
• only 3rd in important factors driving repeat
  visits
• cant tell you anything about your content
• doesnt say anything directly about usability
  problems
• Robots arent humans
• doesnt make mistakes
• remember, GOMS assumes expert behavior!
• doesnt account for understanding text
• only tries the best path users will use many
• Major flaw is the lack of real users in the
  process

41

• Warning
• I am a founder of the following company watch
  for bias!
• ?

42
The Trouble With Current Site Analysis Tools

• Unknowns
• Who?
• What?
• Why?
• Did they find it?
• Satisfied?

43
NetRaker Provides User-centric Remote Evaluation
Using Key Metrics

• NetRaker Index
• short pop-up survey shown to 1 in n visitors
• on-going tracking evaluation data
• Market Research Usability Templates
• surveys task testing
• invitation delivered through email, links, or
  pop-ups

44
NetRaker Index On-going customer intelligence
gathering

• Small number of rotated questions increases
  response rate

45
NetRaker Index On-going customer intelligence
gathering

• Small number of rotated questions increases
  response rate

46
NetRaker Index On-going customer intelligence
gathering

• Small number of rotated questions increases
  response rate

47
NetRaker Index On-going customer intelligence
gathering

• Small number of rotated questions increases
  response rate

48
NetRaker Index On-going customer intelligence
gathering

• Increasing these indices (e.g., retention)
  moderately (5) leads to a large increase in
  revenue growth

49
NetRaker Usability ResearchSee how customers
accomplish real tasks on site
50
NetRaker Usability ResearchSee how customers
accomplish real tasks on site
51
NetRaker Usability ResearchSee how customers
accomplish real tasks on site
52
NetRaker Usability ResearchSee how customers
accomplish real tasks on site
53
WebQuilt Visual Analysis

• Goals
• link page elements to user actions
• identify behavior/nav. patterns
• highlight potential problems areas
• Solution
• interactive graph based on web content
• nodes represent web pages
• edges represent aggregate traffic between pages
• designers can indicate expected paths
• color code common usability interests
• filtering to show only target particpants
• use zooming for analyzing data at varying
  granularity

54
(No Transcript)
55
(No Transcript)
56
(No Transcript)
57
Advantages of NetRaker

• Fast
• can set up research in 3-4 hours
• get results in 36 hours
• More accurate
• can run with large samples (50-200 users -gt stat.
  sig.)
• uses real people (customers) performing tasks
• natural environment (home/work/machine)
• Easy-to-use
• templates make setting up easy
• Can compare with competitors
• indexed to national norms

58
Disadvantages of NetRaker

• Miss observational feedback
• facial expressions
• verbal feedback (critical incidents)
• Need to involve human participants
• costs some amount of money (typically
  20-50/person)
• People often do not like pop-ups
• need to be careful when using them

59
Summary

• GOMS
• provides info about important UI properties
• doesnt tell you everything you want to know
  about a UI
• only gives performance for expert behavior
• hard to create model, but still easier than user
  testing
• changing later is much less work than initial
  generation
• Automated usability
• faster than traditional techniques
• can involve more participants -gt convincing data
• easier to do comparisons across sites
• tradeoff with losing observational data

60
Next Time

• Advanced User Testing
• Appendix A from The Design of Sites
• Gomoll paper
• Statistica Ch1, and parts of Ch3
• Lewis Rieman Ch. 5