The Evolution of HumanPerformance Modeling Techniques for Usability

1
The Evolution of Human-Performance Modeling
Techniques for Usability

• Uri Dekel (udekel_at_cs.cmu.edu)
• Presented in Methods of Software Engineering,
  Fall 2004

2
Outline

• Motivation and scope
• From early models to GOMS
• Stimulus-Response-Controller models
• Information Processing models
• GOMS variants what to use?
• SW tools for GOMS
• Lessons learned

3
Motivation and Scope
4
Motivation

• Minor timing differences may have a major
  economic impact
• Consider a call center with 100 employees
• Average call length 1 min
• 144000 calls per day for entire call center
• Improvement of 2 seconds per call
• 80 person hours per day
• 29200 person hours per year

5
Where can we optimize?

• Moores law works for HW and SW
• In the past, system reaction time was slow
• Databases, networks and GUIs were slow
• Now practically instantaneous
• Moores law does not apply to humans
• But usability has significant impact on
  performance

6
Motivation

• Problems and solution
• How to design more usable interfaces?
• Partial solution usability methods and
  principles
• How to ensure a design can be used effectively?
• Inadequate solution use intuition
• Inadequate solution functional prototypes in a
  design-implement-test-redesign cycle
• Expensive and time consuming, especially for
  hardware
• Possible solution paper prototyping complemented
  by quantitative models for predicting human
  performance

7
Motivation

• We need to predict performance on a system which
  is not yet available
• Nielsen, 1993
• A Holy Grail for many usability scientists is
  the invention of analytic methods that would
  allow designers to predict the usability of a
  user interface before it has even been tested.
• Not only would such a method save us from user
  testing, it would allow for precise estimates of
  the trade-offs between different design solutions
  without having to build them.
• The only thing that would be better would be a
  generative theory of usability that could design
  the user interface based on a description of the
  usability goals to be achieved.

8
Cognitive modeling

• Definition
• producing computational models for how people
  perform tasks and solve problems, based on
  psychological principles
• Uses
• Predicting task duration and error potential
• Adapting interfaces by anticipating behavior

9
Outside our Scope

• Predicting the intent of the user
• Model the activities of the user
• Relies on AI techniques to make predictions
• Useful for intelligent and adaptable UIs
• Improves learning curve

10
Outside our Scope

• Predicting the intent of the user
• Model the activities of the user
• Relies on AI techniques to make predictions
• Useful for intelligent and adaptable UIs
• Improves learning curve
• But not always successful

11
Scope

• Predicting the usability of the UI
• Qualitative models
• Will the UI be intuitive and simple to learn?
• Is the UI aesthetic and consistent?
• Will the user experience be positive?
• Quantitative models
• How long will it take to become proficient in
  using the UI?
• How long will it take a skilled user to
  accomplish the task?

12
Goal of this talk

• The goal is NOT
• To introduce you to GOMS and it variants
• You got that from the reading
• The goal is
• To provide the theoretical foundation and
  evolution of models which led to GOMS
• To show tools that support GOMS
• To understand how it could be useful to you

13
Early models
14
Stimulus-Response-Controller

• Research predates Computer Science
• Attempts to improve usability of interactive
  electronic systems such as control panels, radar
  displays, air traffic control, etc.
• Early models developed by experimental psychology
  researches in the 1950s
• Limited to single short perceptual and motor
  activities
• Based on information and communications theory
• Human is a simple device which responds to
  stimuli by carrying out a motor behavior
• Based on Shannons definitions of entropy and
  channel capacity

15
Information Theory 101 Entropy

• Entropy of a random event is a measure of its
  actual randomness
• High entropy if unpredictable

16
Information Theory 101 Entropy

• Entropy of a random event is a measure of its
  actual randomness
• High entropy if unpredictable
• The winning numbers for this weeks lottery
• Same probability for all results
• Low entropy if predictable

17
Information Theory 101 Entropy

• Entropy of a random event is a measure of its
  actual randomness
• High entropy if unpredictable
• The winning numbers for this weeks lottery
• Same probability for all results
• Low entropy if predictable
• What lunch will be served at the next seminar?
• High probability of Pizza. Low probability for
  Sushi

18
Information Theory 101 Entropy

• Entropy can measure amount of information in
  message
• Consider a message encoded as a string of bits.
• Is the next bit 0 or 1 ?
• High entropy
• What if we add parity bit?
• Lower entropy for the parity bit
• What if we replicate every bit once?
• Even lower for replicated bits

19
Information Theory 101 Entropy

• Formally
• Let x be a random event with n possible values
• The entropy of X is

20
Information Theory 101 Channel Capacity

• Information rate in a perfect channel
• n bits per second
• H entropy per bit
• R nH
• Rn if entropy is 1 (pure data)
• The channel bandwidth curbs the rate

21
Information Theory 101 Channel Capacity

• Information rate in an analog channel
• Curbed by bandwidth and noise
• We can fix some errors using different encodings
• Is there a limit to how much we can transfer?

22
Information Theory 101 Channel Capacity

• Shannons definition of channel capacity
• Maximal information rate possible on the channel
• For every RltC, there is an encoding which allows
  the message to be sent with no errors
• Theoretical maximum effectiveness of error
  correction codes
• Does not tell us what the code is
• Capacity formula
• B bandwidth
• SNR Signal-to-noise ratio

23
Fitts Law

• Paul Fitts studied the human limitation in
  performing different movement tasks
• Measured difficulty of movement tasks in
  information-metric bits
• Movement task is the transmission of information
  through the human channel
• But this channel has a capacity

24
Fitts Law

• Fitts law 1954 predicts movement time from
  starting point to specific target area
• Difficulty index
• A distance to target center, W target width
• Movement time
• a device dependent intercept
• b device dependent Index of Performance
• The coefficients are measured experimentally
• e.g., mouse IP lower than stylus, joystick

25
Fitts Law Implications

• Primary implication
• Big targets at close distance are acquired faster
  than small targets at long range
• Used to empirically test certain designs
• Theoretical rationale for many design principles

26
Fitts Law Implications

• Should buttons on stylus based touch screen
  (e.g., PDA) be smaller, larger or the same as
  buttons in a mouse based machine?

27
Fitts Law Implications

• Should buttons on stylus based touch screen
  (e.g., PDA) be smaller, larger or the same as
  buttons in a mouse based machine?
• Answer larger, because it is more difficult to
  precisely point the stylus (higher index of
  performance)

28
Fitts Law Implications

• Why is the context sensitive menu (right-click
  menu in Windows) located close to the mouse
  cursor?

29
Fitts Law Implications

• Why is the context sensitive menu (right-click
  menu in Windows) located close to the mouse
  cursor?
• Answer mouse needs to travel shorter distance

30
Fitts Law Implications

• Which is better for a context sensitive menu, a
  pie menu or a linear menu?

31
Fitts Law Implications

• Which is better for a context sensitive menu, a
  pie menu or a linear menu?
• Answer if all options have equal probabilities,
  a pie menu. If one option is highly dominant, a
  linear menu

32
Fitts Law Implications

• In Microsoft Windows, why is it easier to close a
  maximized window than to close a regular window?

33
Fitts Law Implications

• In Microsoft Windows, why is it easier to close a
  maximized window than to close a regular window?
• Answer If the mouse cannot leave the screen,
  target amplitude is infinite in the corner of the
  screen where the close box is located

34
Fitts Law Implications

• Why use mouse-gestures to control applications?

35
Fitts Law Implications

• Why use mouse-gestures to control applications?
• Answer A mouse gesture starts at the current
  location and requires limited movement, compared
  to acquiring the necessary buttons.

36
Fitts Law limitations

• Addresses only target distance and size, ignores
  other effects
• Applies only single-dimensional targets
• Later research showed extensions to 2D and 3D
• Considers only human motor activity
• Cannot account for software acceleration
• Does not account for training
• Insignificant effect in such low-level operations

37
Fitts Law limitations

• Only supports short paths
• Research provided methods for complicated paths,
  using integration
• But most importantly
• Operates at a very low level
• Difficult to extend to complex tasks

38
Hicks Law

• Humans have a non-zero reaction time
• Situation is perceived, then decision is made
• Hicks law predicts decision time as a function
  of the number of choices
• Humans try to subdivide a problem
• Binary rather than linear search
• For equal probabilities
• coefficient a measured experimentally
• For differing probabilities

39
Hicks Law

• Hicks law holds only if a selection strategy is
  possible
• e.g., alphabetical listing
• Intuitive implications
• Split menus into categories and groups
• An unfamiliar command should be close to related
  familiar commands

40
Hicks Law Example

• The next slide presents a screenshot from
  Microsoft Word
• How fast can you locate the toolbar button for
  the WordArt character spacing command?

41
Hicks Law Example
42
Limitations of the Early Models

• Developed before interactive computer systems
  became prevalent
• Use metaphors of analog signal processing
• Human-Computer Interaction is continuous
• Cannot be broken down into discrete events
• Human processing has parallelism

43
Information Processing Models
44
Information Processing Models

• Developed in the 1960s
• Combine psychology and computer science
• Humans performs sequence of operations on symbols
• Generic structure

45
Information Processing Models

• Models from general psychology are fitted to the
  results of actual experiments
• Not predictive for other systems
• Zero-parameter models can provide predictions for
  future system
• Parameterized only by information from existing
  systems
• e.g., typing speed, difficulty index, etc.

46
Model Human Processor

• Card, Moral and Newell in 1983
• Framework for zero-parameter models of specific
  tasks
• Humans process visual and auditory input
• Output is motor activity
• Unique in decomposition into three systems
• Each consists of processor and memory
• Can operate serially or in parallel
• Each with unique rules of operation

47
Model Human Processor
48
Model Human Processor Properties

• Processor
• Cycle time limits amount of processing
• Memory
• Relatively permanent long term memory
• Short term memory
• Consists of small activated LTM chunks
• There are seven plus minus two chunks
• Every memory unit has
• Capacity, decay time, information type

49
Perceptual System

• Input arrives from perceptual receptors in
  outside world
• Placed in visual and auditory stores
• Stored close to physical form
• bitmap and waveforms rather than symbols
• Processor encodes symbolically in stores in LTM
• Memory and processing limitations lead to memory
  loss
• Attention directs items to be saved

50
Cognitive System

• Responsible for making decision and scheduling
  motor operations
• Performs a recognize-act cycle
• Uses association to activate LTM chunks
• Acts by modifying data in working memory
• Might require several cycles

51
Motor System

• Translates thoughts into actions
• Uses decisions stored in working memory to plan
  movements
• Different movement options, including speech
• Actually sequence of micro-movements
• Motor-memory cache simplifies actions
• But not explicitly represented in the MHP

52
Limitations of the MHP

• Does not address attention
• Works in a bottom-up manner
• Given a movement scenario, can help determine how
  long it would take
• In design, it is preferable to start with the
  goal and find how it should best be accomplished
• This is what GOMS tries to accomplish

53
Principles of Operation

• Problem space principle
• Users apply a series of operations to transform
  an initial state into a goal state
• Rationality principle
• Users will develop effective methods, given a
  task, its environment, and their limitations.
• MHP provides the human hardware. GOMS models
  will represent the software.

54
GOMS
55
GOMS

• Enables description of tasks and the knowledge to
  perform them
• GOMS model can be used to execute tasks
• Unlike less formal task analysis techniques
• Qualitatively used for developing training tools,
  help systems, etc.
• Quantitatively used for predicting performance

56
GOMS

• Goals
• What the user tries to accomplish
• Hierarchy of subgoals
• Operators
• Atomic operations for accomplishing the goal
• Perceptual, Cognitive or Motor
• Methods
• Algorithms for accomplishing goals
• Selection Rules
• Knowledge used to select method for task

57
Variants KLM

• Keystroke Level Model
• Proposed by Card, Moran Newell
• Predecessor of GOMS
• No selection rules
• Simple linear sequence of operators
• Analyst inputs keystrokes and mouse movements
• Simple heuristics used for placing mental
  operators
• Predicted time is sum of operator execution times
• Can be used to compare two scenarios of using the
  same system

58
Variants KLM

• ExampleMoving textin Word

59
Variants CMN-GOMS

• Card Moran Newell GOMS
• Original formulation of GOMS, extends KLM
• Procedural program form
• Hierarchy of subgoals
• Methods realize subgoals
• Selection rules pick method
• Variety of cognitive operations
• Serial processing

60
Variants CMN-GOMS

• Sample task moving text in word

61
Variants CPM-GOMS

• Cognitive-Perceptual-Motor GOMS
  orCritical-Path-Method GOMS
• Adds parallelism using the MHP
• Created from a CMN model
• Results in a PERT chart
• Execution time is the critical path
• Assumes extreme expertise
• Might underestimate.
• Demonstrated in the Ernestine project

62
Variants CPM-GOMS

• Example text editing

63
Variants NGOMSL

• Natural GOMS Language
• Represents models using natural language
• Relies on Cognitive Complexity Theory
• Internal operators for subgoals, memory
  manipulation
• Rules-of-thumb for number of steps in method,
  setting and terminating goals, information that
  needs to be remembered
• Can provide some estimates of learning time and
  possibility of errors

64
Variants NGOMSL

• Example

65
GOMS limitations

• Tooling
• Will be discussed later
• Valid only for expert users
• The primary goal of HCI was to make systems more
  accessible to novices
• Many systems are only used occasionally
• Ignores problem-saving nature of many tasks
• Assumes no errors
• Even experts make simple errors
• Error recovery may be important factor

66
GOMS limitations

• Does not address individual users
• Relies on statistical averages
• Provides only one metric
• Execution time should not be the only basis for
  comparing designs
• Does not utilize recent cognitive theories
• Even newer models are fuzzy on cognitive
  aspects.

67
Should GOMS be used?

• According to John and Kieras (93), GOMS should
  be used only if
• It is goal-directed
• It is routine and a user can become skilled
• It involves user control
• We want to analyze procedural properties of the
  system

68
Which GOMS model to use?

• Choice based on parallelism and on required
  information
• Functionality coverage
• Functional consistency
• Operator sequence
• Execution time
• Learning time
• Error recovery

69
Which GOMS model to use?

• (Table from John and Kieras)

70
Current Research
71
Rough research evolution

• Proposal of initial GOMS models
• KLM, CMN-GOMS
• Initial experimental validation
• Extension for parallelism
• CPM-GOMS
• Validation, esp. project Ernestine
• Improved model usability and basis
• Kieras NGOMSL
• Still no industrial adoption!
• Simplified methodologies
• Tooling

72
Quick (and Dirty) GOMS

• A very simplified GOMS
• Allows rough time estimates
• A generic task operator
• Probabilities instead of selection rules
• Model is formed as a tree
• Aimed for software engineers
• No need to learn complex methodologies
• Familiar tree paradigm

73
Quick (and Dirty) GOMS

• DEMO

74
Deriving KLM Models

• KLM models are simple to construct and understand
• Goal user creates interface mockup, performs
  activities, gets prediction
• Critique System at CMU
• Uses research-oriented subArctic GUI toolkit

75
Deriving KLM Models

• CogTools project at CMU
• User creates HTML mock-up interface in Macromedia
  Dreamweaver
• User demonstrate tasks in Netscape browser
• (continued)

76
Deriving KLM Models

• CogTools project
• System generates KLM modem and task prediction
• Too many software requirements
• Dreamweaver
• Java
• Netscape browser
• ACT-R
• Allegro Common Lisp
• Skipped the demo for obvious reasons

77
Deriving NGOMSL models

• GLEAN project
• Uses English-like notation
• User supplies
• NGOMSL model of tasks
• Associative representation of working memory
• Pairs of tags and values
• Can be used as method parameters
• Transition net
• Simulates user behavior on simulated device
• Difficult to represent interface without
  implementation

78
Deriving CPM-GOMS models

• CPM-GOMS models are most powerful
• Utilize parallelism of human processing
• Very difficult to create manually
• Given a serial CMN-GOMS model, must find
  acceptable interleaving on MHP resources
• Generate PERT chart

79
Deriving CPM-GOMS models

• NASAs Apex Architecture
• A framework for creating reactive intelligent
  agents for complex tasks
• Models how humans complex tasks
• Procedure Definition Language (PDL) for defining
  tasks and methods
• AI engine for making selections
• Interleaves activities into resources

80
Deriving CPM-GOMS models

• PDL example

81
Deriving CPM-GOMS models

• APEX is usedto model CPM-GOMS
• User specifiestask in PDL
• APEX interleavesand creates PERT chart

82
Deriving CPM-GOMS models
83
Lessons Learned
84
When is it important to predict performance?

• If the system will be used intensively
• If the tasks are routine enough
• If performance has significant economic impact
• Benefits should outweigh prediction cost

85
Cant we just rely on our intuition?

• As SW engineers, we might believe that our
  intuition is sufficient
• Improving qualitative properties is possible by
  following usability rules of thumb
• Quantitative properties are another story
• GOMS research shows that intuition is not
  sufficient
• Consider project Ernestine

86
Are the qualitative benefits to GOMS?

• Analyzing the task helps us focus on the user
• We obtain the sequence models of contextual
  design
• These models help us
• Realize what the user is trying to achieve
• Pinpoint different means for accomplishing the
  same goal
• Discover inconsistencies and opportunities for
  simplification
• Its a good way to create detailed use cases
• It lays the foundation for training and manuals

87
How much should we invest?

• Tradeoff between modeling costs and benefits
• What is the impact of each improvement?
• Is the system mission-critical?
• Even rough estimates can be useful
• Use QGOMS to analyze high level goals
• Create KLM models with a spreadsheet to analyze
  simple scenarios

88
Who should be assigned with GOMS modeling?

• SW engineers generally avoid learning techniques
  that dont help them construct new software
• Hire a usability engineer if modelling is
  critical enough
• Experts are needed for accurately using the
  complex GOMS variants
• Use QA or Testing engineers before using engineers

89
Cant SW engineers do the modelling?

• Accurate GOMS modeling requires knowledge in
  cognitive psychology
• SW Engineers will usually avoid tools not
  integrated with the IDE
• Vision a human performance profiler in the
  IDE, just like the execution profiler
• User interacts with designed UI, system suggests
  what the average execution time will be

90
Finally If you use GOMS

• Remember the limitations of GOMS
• Dont rely on it in inappropriate settings.
• Do not rely on it as your only metric
• Do not abandon the notion of prototypes

91
Questions?
92
BACKUP SLIDES
93
Information Theory 101 Entropy

• Entropy measures the amount of information
• An intuitive example Consider a sequence of n
  bits which encodes some random data
• Option 1 All bits are used to represent a
  message of length n
• Caveat error can destroy the message
• We got 2n possible messages, with equal
  probabilities
• We cannot predict or bias what the message
• Such as message is said to have a high entropy.
• Option 2 Use some bits for error correction
  (e.g., parity)
• Advantage we can detect some errors
• We got less than 2n possible messages
• We can predict one of the bits
• Such as message is said to have a lower entropy.

94
Information Theory 101 Entropy

• Other intuitive examples
• Tossing a legal coin entropy of 1 (same
  probability for each result)
• Skewed coin lower entropy (one result more
  likely)
• Consider encryption mechanisms for textual
  communications
• Characters in unencrypted text have non-uniform
  distribution
• Low entropy
• Bad encryption add the same value to every
  character
• Entropy is still low
• Good encryption increase entropy by a sequence
  of encryptions and shifting

95
(No Transcript)
96
Information Theory 101 Entropy

• Entropy measures the amount of information
• An intuitive example Consider a sequence of n
  bits which encodes some random data
• Option 1 All bits are used to represent a
  message of length n
• Caveat error can destroy the message
• We got 2n possible messages, with equal
  probabilities
• We cannot predict or bias what the message
• Such as message is said to have a high entropy.
• Option 2 Use some bits for error correction
  (e.g., parity)
• Advantage we can detect some errors
• We got less than 2n possible messages
• We can predict one of the bits
• Such as message is said to have a lower entropy.

97
Information Theory 101 Entropy

• Example
• E-commerce site located in the US
• Provides services to USA and N other countries
• Assume probability of US customer is p
• Equal distribution for other countries
• What is the entropy of user location?

98

• In a perfect analog channel, we could have
  unlimited bandwidth by increasing frequency and
  discrete voltage values
• Physics curbs frequency. Noise introduces errors
• Information rate in a noisy channel
• Both bandwidth and noise curb information rate
• Wrong intuition double the rate, double the
  errors