2. Models for cognitive ergonomics

1
2. Models for cognitive ergonomics

• 2.1. the concept of models
• 2.2. models in cognitive psychology
• 2.3. cognitive ergonomics models

2
2.1. the concept of models

• M is a model of A if M can be used
• to answer questions about A (Ross,
  1983)
• A model is a representation of relevant
  characteristics
• of an object (Rohr Tauber, 1984)
• A model is a description that can be
  communicated, of a certain aspect of part of the
  real world, viewed at a certain level of
  abstraction (Oberquelle, 1984)

3
representation of relevant characteristics of an
object
Mental model of system
system
User inter-face
user
4
aspect of part of the real world, viewed at a
certain level of abstraction
Aspect 1, graphics
Aspect 2, dialogue
User interface
system
5
the concept of modelsinternal models
external models

• External models are for communication,
• should be represented explicitly
• Internal models are for execution there is an
  agent who uses the model to make decisions
  based on the behavior of the model, and to make
  predictions on the behavior of the modeled
  reality
• if the agent is human mental model
• if the agent is a machine program, database

6
2.2. models in psychology

• 2.2.1. models of human information processing
• modern theories, e.g., Barsalou
• mental models in Cognitive Psychology
• 2.2.2. mental models in Cognitive Ergonomics -
  Norman

7
2.2.1. Model of human information processing

• long term MEMORY
• THOUGHT
• working
• MEMORY

• input
• PERCEPTION
• output
• MOTOR/BEHAVIORAL

TASK/ STIMULI
8

• ACTIVITITES OF THE COGNITIVE SYSTEM
• PERCEPTION
• complex sensorial processes
• primary images containing all information about
  the concrete features
• of objects and phenomena
• that act direct action upon the sensory systems
• (visual, acoustic, kinesthetic, olfactory,
  gustatory).
• MEMORY
• ability to remember, recognize and recall
• information is encoded, stored and retrieved.
• active structuring, constructive and creative
  psychic mechanism.
• THOUGHT the process of information processing in
  working memory.

9
VISUAL PERCEPTION - A COMPUTATIONAL THEORY
(pattern recognition) PRIMARY PROCESSING
SECONDARY
PROCESSING - RECOGNITION
Visual Stimuli
Descendant processing
Primary sketch

2.5 D sketch 3D
Representation texture
movement color distance position
gestalt geons
depth
form principles
segmentation Processing
modules
10
VISUAL PERCEPTION - A COMPUTATIONAL THEORY
(pattern recognition) Data-driven
(bottom-up ) processing 1. mechanisms of edge
detection, processing of texture, movement,
color, distance and depth, position and form from
processing of shadow of stimulus
(2.5-D sketch,) - automated/ modular/preattentiona
l/ unconsciousness processing 2. edge
organization - gestalt principles of perception
(applied to 2.5-D sketch) proximity,
similarity, good continuation, closure

example IwOuLdLiKeToDrInKaBeErInArOmAnIaN
bAr. 3. geons generation (geometrical ions)

segmentation of the 2.5-D sketch (intermediary
image) in zones of maximal concavity 4.
Recognition

geons activate from memory the
objects made-up by the respective geons, matching
the 2.5-D sketch with the representation stored
in memory
11
GESTALT PRINCIPLES OF PERCEPTION
A. Proximity principle
B. Similarity principle
D. Closure principle
C. Good continuation principle
A
D
C
B
12

• VISUAL PERCEPTION - A COMPUTATIONAL THEORY
• (pattern recognition)
• Conceptually driven (top-down) processing
• concepts and higher-level processes influence
  pattern recognition
• applied to 2.5-D sketch
• recognition in the case of
  
  - verbal
  stimuli (word superiority effect sentence
  superiority effect)
• T E C T

THE WORK MUST GET DONE. WORK
13

• VISUAL PERCEPTION - A COMPUTATIONAL THEORY
• (pattern recognition)
• Im zhizkizg tz enzoy zhiz wezk-ezd az thz
  sezsize.
• - objects (object superiority effect)
• - human faces
• OBS. Importance of implicit/tacit knowledge
  about the (statistical) regularities of the world
  in pattern recognition (physical support of
  things, reciprocal occlusion, occurrence
  probability, relative size, position and rigidity
  of objects etc.).
• Violation of tacit assumptions causes visual
  illusions. This assumptions are not specific
  (they are applied automatically to any object).

14
The importance of context Illusions - at level
of modular processing
15

• AUTOMATIC PROCESSING
• Learned automatic processing
• Frequent association of a stimulus with a
  response produces a production having a
  relatively autonomous status. Then, the
  perception of the stimulus produces the response
  with no intention to do so (ex. classical
  conditioning).
• in perception orienting learned reflexes
  (shifting the attention to important stimuli in
  the environment - ex. a speaker who utters your
  name)
• (used by advertising industry)
• in memory upon encoding a particular stimulus,
  people may activate automatically information
  associated with it in memory

16

• STRATEGIC PROCESSING
• EXECUTIVE PRODUCTIONS
• cognitive mechanism that establish and execute
  acquired goals
• if-then productions
• current state of the environment and the
  cognitive system determine which executive
  production is fired in order to pursue the
  high-level goals of the cognitive system
• number of executive productions that can be
  fired at once is extremely limited (one or
  possibly few) because of centralized strategic
  resource

17

• MECHANISMS OF STRATEGIC PROCESSING
• executive productions operates on goals, scripts
  and reminded episodes to select and coordinate
  information processing subsystems during
  goal-directed behavior
• repeating particular patterns of strategic
  processing produces new productions that automate
  repeated parts of relevant scripts, it freeing
  executive productions to work on more subtle
  aspects of task or to perform multiple tasks
  simultaneously
• skills develops as increasing amounts of the
  processing (that executive production perform)
  become automated

18
PERFORMANCE CHARACTERISTICS OF INFORMATION
PROCESSING 1. Limited capacity information
processing resource that limits strategic
processing was theorized as - limited processing
energy (analogous with an electrical source) (if
one strategic task require all of the available
processing energy, no other task can be
accomplish simultaneously) - a single executive
processor that applies and manages executive
productions - it can only execute one strategic
task at a time but can switch quickly from one
strategic to another. Limits on strategic
processing arise from the limited ability of the
executive processor to switch between the tasks.
Elementary
operations are compare / choose /
repeat / compute / transform
19

• PERFORMANCE CHARACTERISTICS OF INFORMATION
  PROCESSING
• 2. Selectivity
• to achieve a goal a cognitive system must be
  able to select and coordinate information
  processing subsystems (ex. perceptual, motor,
  memory), locations in perceptual fields and
  categories in memory
• corresponds to what theorists often mean by
  attention
• selection can be specific (ex.particular ear or
  eye, particular information in memory)

20
DISCUSSIONS Presence of a single executive
does not imply that it controls the entire
cognitive system The executive may primarily
schedule and monitor (it may also direct
processing when goals are new, difficult and
dangerous). Many basic processes in perception
and movement, many acquired productions which
control many automated skills (ex. driving,
typing), lie beyond its scope.
21

• SENSORY MEMORY
• consist in persistence (prolongation) of the
  sensorial representation of the stimulus after
  the stimulus is no longer acting on the receptors
• specific to a certain type of sensation
• format neuro-physiological codes
• capacity unlimited but the cognitive system
  will process further only the relevant stimuli
• duration - visual memory 100 ms - auditory
  memory 200 ms / 2 s
• automatic/ pre-attentional retention of the
  precategorical information (information is in an
  unprocessed form)

22

• LONG TERM MEMORY (LTM)
• all the knowledge the cognitive system owns
• unlimited capacity
• duration whole life of neural system
• memory systems in LTM
• explicit vs. implicit
• semantic vs. episodic (memory of general
  knowledge about our environment vs. memory of
  personal events)
• format (encoding) of the information
• verbal / analog (images) / semantic
  (propositional)
• retrieval (activation) of knowledge parallel
  search process

23

• WORKING MEMORY (WM)
• activated part of long-term memory
• encoding verbal, analogical (image), semantic
  (propositional)
• capacity
• depending on the level of expertise (by
  chunking) maximum capacity of attention 7
  /- 2 chunks
• limited capacity for a certain type of
  information (auditory, visual, motor)
• limits for certain types of information are
  independent one from another (ex. if
  maximum of visual information is in WM this does
  not decrease the maximum of auditory information
  that can be in WM).

24
Relation between LTM, WM and attention
when there is a specific goal to be reached

when there is no specific goal to be reached
LTM
LTM
WM

Attention
WM Attention
25

• ORGANIZATION OF KNOWLEDGE IN
• LONG TERM MEMORY
• EXPLICIT MEMORY content is accessible to
  consciousness and can be tested by recall and
  recognition tests
• low level structures
• propositional network (semantic memory)
• semantic network (semantic memory)
• high level structures (complex units of
  knowledge)
• schema/script/frame/plan (semantic and episodic
  memory)
• mental models (semantic and episodic memory)
• IMPLICIT MEMORY content is hardly accessible to
  consciousness and cannot be tested by recall and
  recognition tests.
• production systems (cognitive and motor skills,
  priming, conditioned reflexes)

26
SEMANTIC NETWORK - represents semantic contents
from well-structured knowledge domains -
knowledge are represented by a network of nodes
and relation between nodes - nodes represents the
concepts and the relations between nodes are
labeled - meaning of a concept (or node) is
given by the pattern of its relations among which
it participates.
property
breath
animal
have skin
type
bird
property
type
fly
swim
fish
have feather
type
shark
dangerous
chicken
type
predator
eagle
not eatable
symbol of power
27

• SCHEMAS (Rumelhart, 1980)
• represents generic concepts stored in memory
  underlying objects, situations, events, sequence
  of events, actions and sequence of actions
• they are used for a class of stereotypical
  situation
• they vary the very simple to the very complex
• are organized in a hierarchical fashion
• variables which have fixe value (the kernel)
• slots with optional values (pheripheral) which
  can be filled in with particular instances of the
  concepts. If the instances are not specified then
  the slots will have default values (prototypes)
• can be embedded one in other, e.g.,
• Human body (Head, Trunk, Limbs)
• Head (Face, Ears, Hair)
• relation between the elements are in
  spatial-temporal contiguity (ex. bread - butter)
• active processing devices (top-down processing)
  which produce an interpretation of the world -
  they adapt reality to knowledge
• they are assumed to be shared across individuals
  (in a culture?)

28

• SPECIAL TYPES OF SCHEMAS
• SCRIPTS (Shank and Abelson,1977)
• schemas for frequently occurring sequences of
  events in a particular context
• 2 categories of variables roles (filled by
  persons) and props (filled by objects)
• includes - entries condition
• - scenes
• - results
• scripts are the result of social learning
• maintanance of the scripts is guaranted by a set
  of social contingencies
• FRAMES (Minsky, 1975) - static representation
• - schemas that do not possess active processors
• - between the elements of the frame are enabling
  or causal relations
• - framework that is adapted to fit reality
• generic frames (class)
• ex. car ( color, brand, driver, engine,
  transmission, wheels)
• specific frames (instantiated) - in a particular
  context

29

FRAME FOR CAR
gasoline
John
type
Driver
type
buys
Fuel
diesel
type
type
flows
Liz
operates
type
Engine
four-cylinder
operates
rotates
six-cylinder
type
Transmission
standard
rotates
Fixed values (kernel)
steel
Specific instances (default values)
Wheels
alloy
30
Task Arrange a project meeting Plan (meeting
(project)) Consult (information source,
information token, project meeting)
Identify (information source, information token,
project meeting) Search (information
source, information token) Retrieve
(information token, information source)
Store (information token, project meeting,
working memory) Select (media message)
Identify (long-term memory,constrain,
project meeting) Choose (media,
constrains ) Send message (meeting (project))
Consult (information token, information source)
Identify (information source,
information token, letter) Search
(information source, information token)
Retrieve (information token, information
source) Store (information token,
information source) Represent (information
token, message) Write (information
token, message, media) Compare
(message, information token) Edit
(information token, message) Store
(message copy, message file, media)
Execute (transaction requirements,
message) . frame-based representation how to
create and send message to arrange a project
meeting (Keane and Johnson, 1987)
31
Script eating at a restaurant Entry condition
hungry, had money, restaurant open Roles
diner, waiter, cashier Props
tables, money, chairs, menu,
cutlery, food Entry scene Diner enters
restaurant. Waiter
seats diner at table.
Waiter places menu on table.
Diner begins to read menu. Ordering scene
Diner selects food from menu.
Diner signals to waiter.
Waiter approaches the table.
Diner orders food.
Waiter leaves. Eating scene
Waiter brings food to the table.
Waiter leaves.
Diner eats food with cutlery.
Diner finishes eating food. Leaving
scene Diner signals to waiter.
Waiter approaches table.
Diner asks waiter for bill.
Diner checks bill.
Diner approaches cashier.
Diner gives cashier bill and
money. Cashier checks
money. Diner leaves
restaurant.
32

• MENTAL MODELS - dynamic representation
• frames in which the relations between (and
  attributes of) the elements are analogous to a
  physical/organizational/procedural structure in
  the world (component parts and relations between
  these) reflecting the actual state of affairs in
  the world.
• parts of it become instantiated being triggered
  by an input (stimulus, problem, event)
• can be run producing quasi-continuos simulation
  of the events (over space and time) and can
  explain how events occurred (comprises
  explanatory principles)

33

• IMPLICIT MEMORY - TYPES OF KNOWLEDGE
• Cognitive and motor skills
• they develop from a script-like representation of
  knowledge
• Conditioned reflexes
• it develops by association of stimulus with
  response
• Priming
• it develops by frequent exposure to a stimulus,
  modifying the judgement value of the stimulus

34

• IMPLICIT MEMORY - KNOWLEDGE REPRESENTATION
• representation are in the form of production
  rules a production is a condition-action
  (if-then) pair
• IF (condition for triggering) THEN (do these
  actions)
• production rules are organized in production
  systems
• production systems can be general or specific
  (defining expertise in a certain domain)
• they are triggered automatically by
  categorization of relevant stimuli by matching
  current state of problem-solving (as a pattern
  in working memory) or of a stimulus against the
  conditions of the productions rules
• they are hardly accessible to consciousness
• when needed, the script (on the basis on which
  production system has been developed) can be
  reconstructed

35
A variant of the model, Card, Moran, Newell

36

• Some
  principles
• perceptual processor cycle time varies inversely
  to stimulus intensity
• cognitive processor cycle time
• shorter with more task load
• shorter with more information
• shorter with practice in task domain

37
A variant of the model, Card, Moran, Newell

• Some laws
• Fitts law
• time T to move hand to target of size S at
  distance D
• Tpos 10070120 msec/bit log2 (D/S .5)
• Power law of practice
• time Tn needed to complete a task at trial n
• Tn T1 nª , where a .4.2.6

38
A variant of the model, Card, Moran, Newell

• Example
• simple reaction time

39
A variant of the model, Card, Moran, Newell

• Example
• symbol
• detection

40
Calculation symbol detection

• RT Tperceptual processor 2 Tcentral
  processor T motor processor
• 10050200 2(7025170)
  7030100
• 310130640 msec

41
2.2.2. Mental models from the point of view of
Cognitive Ergonomics

• The functions of mental models in using complex
  systems
• What type of mental models are needed for using
  complex systems
• How to measure mental models

42
The functions of mental models in using complex
systems

• Planning
• execution of task delegation
• evaluation
• interpretation

43
What type of mental models are needed for using
complex systems

44
(No Transcript)
45
(No Transcript)
46
(No Transcript)
47
(No Transcript)
48
(No Transcript)
49
(No Transcript)
50
(No Transcript)
51
(No Transcript)
52
Characteristics of mental models

• Incomplete (users are generaly aware of this)
• parsimoneus, just sufficient basic knowledge
• can only partly be "run"
• unstable, change over time,
• using different systems
• new experiences (even experts)
• vague boudaries (e.g., application / operating
  system)
• superstition

53
How to measure mental models

• Reconstructing semantic networks / frames
• Pathfinder Algorithm - allows insight in group
  knowledge and comparison between groups
• Hermeneutic interpretation of individual
  instantiations
• Instantiated frames Teach-back task what is ?
• Instantiated scripts Teach-back task how to ?

54
DIGITAL INK PEN
55
Scenario 1

• I discover a small garden yesterday, it was warm
  and dry and there was almost nobody around. And
  it was really the most relaxing place I had been
  to in months

56

• So I sat down, I did some writing and drawing
  and thinking that it was months that I
  communicated with my friends back home
• I decided that I should send them some of my
  garden sketches by e-mail.

record
57

• So, in order to send the e-mails
• I pressed the mode button on the side of the DI.

record
58

• That changes the mode from record my drawing and
  hand writing...
• to command mode which allows me to tell DI what I
  wanted to do next.

command
record
59

• While in command mode the elliptical screen
  slowly scrolls through the command words
  reminding me of what option I want to use.

send
60

• In this case I wanted to send e-mails to my
  friends so I simply write
• the word send followed by their e-mail
  addresses.
• The -gt terminates the command and sends the
  message
• Once the send command has been written and read,
  the screen displays the progress of the command
• Now my sketches are on their computers.

sending
Send to luisa_at_vu.nl-gt Send to pepe_at_vu.nl-gt
61
Scenario 2

• I was wandering around the city and I bumped
  into Chris. We have only met once or twice before
  but we ended up talking for about one hour about
  my new job and his new business in Piltsburg

62

• We had such a good time and I really wanted to
  get together again with him soon to talk again.
• So I took his phone number and his e-mail
  address and stored them in my pen just by writing
  on the back of an old receipt.

record
Chris_at_vu.nl 020- 4567890
63
Scenario 3

• I am getting ready to go on vacation and I have
  all this staff that I have to plan client
  meetings, tons of work to do and what better
  place to keep track of important clients than on
  a napkin of a cafeteria.

64

• The great think of the DI is that you can write
  on any surface that you want. I clipped my pen
  cap on the upper left hand corner of the napkin,
  then I mark two opposite corners. The clip
  combined with the two points lets the pen know
  the boundaries of the writing surface.

65

• So I wrote down a bunch of meetings, whom I am
  going to meet, and when and what I will be
  talking about and I was done.

record
1000 GUI meeting 1230 lunch with Chris 1400
meeting with the design team
66

• I left my list for the waiter to take away and
• all I have to do is take my DI home tonight, put
  it into the DI well
• and then I will download my list of things to do
  into my computer and I will be ready to go.

download
DI well
67
Scenario 4
Ding!

• With DI I can also check my e-mail.
• Incoming e-mail is indicated by a sound from DI
  (ding).

e-mail
68

• When I want to read e-mails I press the mode
  button
• and write or speak the word get message to see
  the e-mail or e-mails I received

get message
69

• then the senders of my e-mails appear numbered
  serially.
• To move through the list I can simply write or
  speak the words up or down

1.luisa
2.chris
70

• So if I want to read the message of Chris
• I write or speak the word show message
• And the message of Chris will appear on the
  screen.
• Then I could write back to him using the DI
  e-mail or fax command.

2. chris
71

• So those are things that DI can do. Its a new
  innovative product that works off patterns that
  we are already familiar with.
• In fact, when DI is off I can use it as a
  regular pen.

72
DIGITAL INK PEN
73
We would like to know how you imagine the
Digital Ink Pen after seeing the scenarios.
Therefore, explain to your friend Lucas what
the Digital Ink Pen is. You can use text,
drawings, etc.
74
Your friend Lucas wants to send a fax to the
administration of the faculty. Explain to him how
to do this using the Digital Ink Pen.
75
Lucas has five messages in his Digital Ink
Pen and he is not interested in the third one.
Try to explain to him how to delete this e-mail
from the list of e-mails he has.
76
How could Lucas dictate (speaking) a letter to
Digital Ink and download it later to his computer?
77

• Teach-back questions
• what is We would like to know how you imagine
  the Digital Ink Pen after seeing the scenarios.
  Therefore, explain to your friend Lucas what
  the Digital Ink Pen is. You can use text,
  drawings, etc.
• how to 1 Your friend Lucas wants to send a
  fax to the administration of the faculty. Explain
  to him how to do this using the Digital Ink Pen.
• how to 2 YLucas has five messages in his
  Digital Ink Pen and he is not interested in the
  third one. Try to explain to him how to delete
  this e-mail from the list of e-mails he has.
• how to 3 Y How could Lucas dictate (speaking)
  a letter to Digital Ink and download it later to
  his computer?

78
Some examples

• of protocols
• and their interpretation

79
Well understood, but rather conservative
regarding the dialogue possibilities
80
Extrapolation based on intuition and consistency
81
Reliability in interpreting visual protocols
? requires training
82
User created verbal command mode
83
User supposed restrictions in dialogue - why ??
84
A wish list of functionality and with context
specifically referred to
85
The pen does understand
86
2.3. models in cognitive ergonomics

• 2.3.1. model of changeability
• 2.3.2. conceptual models of systems and of
  interaction
• 2.3.3. task models

87
2.3.1. model of changeabilityof cognitive
functions

• Stable,
  
  mainly influenced
• resistant to change
  by
  environment
• personality cognitive
  strategies
  individual
• factors styles
  
  knowledge
• -intelligence
  -field(in)dependence -impulsivity
  -schemas
• (spatial ability)
  -verbalisers / imagers -serialists /
  holists -production rules
• adapt to the user
  the
  user may adapt

88
Individual knowledge

• Long term memory
• explicit
• implicit
• instruction / exploration
• meta-communication by system (user interface)
• implicit (dialogue, lexicon)
• explicit (error messages, help system, yellow
  stickers)
• content domain
• task knowledge
• system knowledge

89
strategies

• Domain dependent
• may be learned
• adapting to strategies is often useful
  impulsivity - undo
• an item of an impulsivity test
• find the picture identical to the
• target figure on the top

90
Cognitive styles

• Domain independent / generic
• changeable through education
• imagers - symbolisers / field
  (in)dependence
• two fragments from a field(in)dependency test
• read the codes of the figures in the matrix as
  fast as possible

91
Personality factors

• Stable, needs adaptation to
• e.g. spatial ability (intelligence factor)
• low s.a. is a handicap in working with complex
  system objects that cannot be presented as a
  whole
• adaptation can overcome handicaps
• navigation aids
• abstract overview

92
2.3.2. conceptual models of systems and of
interactionNorman

• Norman (1983)
• target system
• mental model
• conceptual model
• Conceptual model
• perceptual interface conceptual interface
• the users virtual machine
• the model for design for meta-communication
  (help)

93
conceptual models of systems and of
interactionCLG

• Moran (1981) Command Language Grammar (CLG)
• views on conceptual model
• linguistic view - modeling and analyzing the
  interaction
• the psychological view - modeling and analyzing
  user knowledge and mental model
• the design view - modeling and analyzing
  the interface (the users virtual
  machine)

94
conceptual models of systems and of
interactionCLG

• Levels of analysis
• conceptual component
• task level
• semantic level
• communication component
• syntax level
• interaction level (key stroke level)
• material component
• spatial layout level
• apparatus level

95
conceptual component

• task level
• tasks for the user / tasks delegated to the
  system
• task procedures, objects in task domain
• semantic level
• tasks to delegate
• system objects with attributes
• system operations
• user specifications, system states and state
  changes

96
communication component

• syntax level
• dialogue style
• lexicographic structure
• user actions and system acts
• temporal structure
• interaction level (key stroke level)
• user key-strokes, mouse handling, voice input,
  eye gaze
• system signs (beep, cursor indication, cursor
  movements, tactile feedback)

97
material component

• The domain of classical ergonomics
• spatial layout level
• screen design, window layout
• spatial relation of input and output devices to
  work environment (light conditions, sound
  conditions)
• apparatus level
• characteristics of hardware (shape of buttons,
  key press characteristics)

98
2.3.3. task models

• Users of complex systems have a task model
• related to the use of technology (internal
  model)
• designers of complex systems need a task model
• to make design decisions on how to support
  the user (external model)
• for details, see unit on (task) analysis