Making sense out of recorded usersystem interaction

1
Making sense out of recorded user-system
interaction

• Dr Willem-Paul Brinkman
• Lecturer
• Department of Information Systems and Computing
• Brunel University
• (willem.brinkman_at_brunel.ac.uk)

2
Topics

• VIVID Research Centre
• Motivation - Component-Based Software Engineering
• Experiment 1 Searching for a component-specific
  measure
• Experiment 2 Validating a component-specific
  measure
• New and future research

3
VIVID Research Centre

• Based in the Department of Information Systems
  and Computing, Brunel University (London)
• Original focus on visualisation, but now also
  includes
• - Mobile technology
• Design for diverse user groups
• - Novel input/output devices
• 11 academics member of staff, 13 PhD Students

disc.brunel.ac.uk/research/vivid/index.htm
4
Motivation
Studying the usability of a system
Work conducted together with Reinder Haakma
(Philips), Don Bouwhuis (Eindhoven University of
Technology)
5
Motivation
Internal Comparison trying to link usability
problems with parts of the systems
6
Component-Based Software Engineering

• Multiple versions testing paradigm
• Single version testing paradigm

Manage
Create
Support
Re-use
Re-use
7
Motivation

• PROBLEM
• Only empirical analysis of the overall system
  such as (task time, keystrokes, questionnaires
  etc) - not powerful
• Usability tests, heuristic evaluations, cognitive
  walkthroughs where experts problems unreliable
• SOLUTION
• Component-Specific usability measures more
  powerful and reliable

8
Searching for a component-specific measure

• Questions
• What is a component?
• What interaction data should be recorded?
• How do we link interaction data with the
  usability of a component?

9
Layered Protocol Theory(Taylor, 1988)
10
Interaction layers
Control results
01111 10111 Add
15 23
100110
38
Control equation
15

23

User
Calculator
11
Experiment 1 Fictitious Interface
User Task Rotate the Trumpet
12
Experiment 1 - Architecture
Other symbols
13
Experiment 1 - Architecture
Measures Task time Rotate(T0), Rotate(T-1), Ro
tate(T-2) change, rotate bike,aeroplane,
other clicks
Other symbols
14
Experiment 1 - Training
15
Experiment 1 Test Procedure

• 80 participants, all students of Eindhoven
  University of Technology
• 8 different trainings
• After training participants were asked to
  rotate, as fast as possible, a specific music
  instrument
• User interaction with the system was recorded in
  log file
• Once a task was complete the recording stops

16
Experiment 1 - Low-level Effect of Selector
training
Number messages
Clicks on
17
Experiment 1 - High-level Effect Rotator Training
Number messages
Rotate (X)
18
Experiment 1 Control Loop
Reliability how do we link interaction data
with the usability of a component?
Each message is a cycle of the control loop ?
Number of messages presents the users effort to
control the component
User
System
19
Experiment 1 - Conclusion

• What is a component?
• An interaction component is a unit within a
  device that directly or indirectly receives
  signals from the user. These signals enable the
  user to change the physical state of the
  interaction component
• What interaction data should be recorded?
• Message exchange between the interaction
  components

20
Experiment 2 Validation

• 80 users
• 8 mobile telephones
• 3 components were manipulated according to
  Cognitive Complexity Theory (Kieras Polson,
  1985)
• Function Selector
• Keypad
• Short Text Messages

21
Architecture Mobile telephone
Send Text Message
Function Selector
Keypad
22
Experiment 2 Function Selector
Versions Broad/shallow Narrow/deep
23
Experiment 2 Keypad
Versions
24
Experiment 2 Send Text Message
Simple
Versions
Complex
25
Statistical Tests
p-value probability of making type I, or ?,
error, wrongly rejecting the hypothesis that
underlying distribution is same.
26
Results Function Selector
Results of two multivariate analyses and related
univariate analyses of variance with the version
of the Function Selector as independent
between-subjects variable.
27
Results Keypad
Results of multivariate and related univariate
analyses of variance with the version of the
Keypad as independent between-subjects variable.
28
Results Send Text Message
Results of two multivariate analyses and related
univariate analyses of variance with the version
of the STM component as independent
between-subjects variable
29
Power of number of messages as a usability measure
Statistical Power 1 - ß
30
Results
Average probability that a measure finds a
significant (a 0.05) effect for the usability
difference between the two versions of FS, STM,
or the Keypad components
31
Component-Based Software Engineering

• Multiple versions testing paradigm
• Single version testing paradigm

Manage
Create
Support
Re-use
Re-use
32
Testing Different Components

• Component specific objective performance measure
• Messages received Weight factor
• A common currency
• Compare with ideal user
• A common point of reference
• Usability of individual components in a single
  device can be compared with each other and
  prioritized on potential improvements

33
Assigning weight factors to represent the users
effort in the case of ideal user
Set ltFill colour red, no bordergt
7
Call ltgt
2
Click ltleft on Fill tabgt
1
Click ltleft on on colour redgt
1
Click ltleft on Outline tabgt
1
Click ltleft No Line buttongt
1
Click ltrightgt
1
Click ltleft no Ok buttongt
1
Click ltleft on Properties optiongt
1
34
Total effort value

• Total effort ? MRi.W

• MRi.W Message received. Weight factor

Call ltgt
2
Click ltleft on Fill tabgt
1
Click ltleft on on colour redgt
1
Click ltleft on Outline tabgt
1
Click ltleft No Line buttongt
1
Click ltrightgt
1
Click ltleft no Ok buttongt
1
1
Click ltleft on Properties optiongt
5
2 7

2
35
Assigning weight factors in case of real user
Correction for inefficiency of higher and lower
components
36
Assigning weight factors in case of real user
Inefficiency of lower level components need more
messages to pass on a message upwards than
ideally required
Assign weight factors as if lower components
operate optimal
37
Assigning weight factors in case of real user
Inefficiency of higher level components more
messages are requested than ideally required

• UE User effort
• MRi.W Message received. Weight factor
• MSUreal Number of messages sent upward by real
  user
• MSUideal Number of messages sent upward by
  ideal user

38
Ideal User versus Real User
Calculate for each component

• Extra User Effort User Effort - Total effort

39
Experiment 2 - Single version

• 40 users
• 4 mobile telephones
• 2 components were manipulated (Keypad only
  Repeated-Key Method)
• Function Selector
• Short Text Messages

40
Results
Extra User Effort
Mobile phones
41
Results
Partial correlation between extra user effort
regarding the two components and other usability
measures
p. lt .05. p. lt .01.
42
Topics

• VIVID Research Centre
• Motivation - Component-Based Software Engineering
• Experiment 1 Searching for a component-specific
  measure
• Experiment 2 Validating a component-specific
  measure
• New and future research
• Extending the analysis outside the lab
• Extending the analysis beyond only usability
  issues

43
New Projects - Field usability

• CD player, which 10 users will use at home
• Record interaction online assignment of weigh
  factors, both optimal and real user, to messages
• Correlated interaction data with other data
  (questionnaire, dairy, interview)

(Pui-Fong Man)
44
New Projects - PROSKIN
User profiling for skinnable domestic technology

• Exciting Interface designed for the average user.
  However, the average user does not exist.
• Developing skins for specific user groups could
  be a way forward
• Question
• How to identify user groups?
• What do user groups want?

Work conducted together with Nick Fine
45
New Projects - PROSKIN
User profiling for skinnable domestic technology

• Possible solution
• Recording online interaction, Identifying user
  groups, Developing skins for these user groups
• Question
• How to establish user groups that are relevant
  for designer?
• This time, how to make sense of the interaction
  data beyond usability?

Work conducted together with Nick Fine
46
New Projects - PROSKIN
Approach
47
Conclusions and Final Remarks

• Interaction data can be used to study the
  usability of interaction components
• External Comparison between different versions
  More Powerful
• Internal Comparison prioritized on potential
  improvements
• Future questions
• Usability analysis of everyday life interaction
• Establishing new paradigms to understand
  interaction data beyond usability issues

48
Questions

• Thank you for your attention