Perception and Cognition w'r't' Large Highresolution Displays

1
Perception and Cognition w.r.t. Large
High-resolution Displays

• Yonca Haciahmetoglu
• CS6724 Display Wall User Interfaces
• Spring 2006

2
Topics - Papers

• Perceptual user interfaces
• Turk, M. Robertson, G. (Eds.), 2000.
  "Perceptual user interfaces," Communications of
  the ACM (special issue), 43(3), 32-70.
• Productivity Benefits of Very Large Displays
• Czerwinski, M., G. Smith, T. Regan, B. Meyers, G.
  Robertson and G. Starkweather. Toward
  Characterizing the Productivity Benefits of Very
  Large Displays. INTERACT, 2003, pp. 9-16.
• Human Memory
• Tan, D.S., Stefanucci, J.K., Proffitt, D.R.
  Pausch, R. (2001) The Infocockpit Providing
  Location and Place to Aid Human Memory. Workshop
  on Perceptive User Interfaces 2001, Orlando,
  Florida.

3
Part I

• Perceptual User Interfaces

4
Perceptual User Interfaces (PUI)

• Special section on PUIs in the March 2000 issues
  of Communications of the ACM, edited by Matthew
  Turk and George Robertson.
• Definition
• PUIs combine natural human capabilities of
  communication, motor, cognitive, and perceptual
  skills with computer I O devices, machine
  perception, and reasoning.
• Different disciplines
• Vision, speech, graphics and visualization, user
  modeling, haptics, and cognitive psychology

5
Introduction

• No Moores Law for user interfaces
• HCI had not changed fundamentally for nearly two
  decades. Most users interact with computers by
  typing, pointing, and clicking.
• Large-scale displays are becoming more common
• One size does not fit all (Desktop, large
  display, mobile handheld devices)
• WIMP will not scale to match the computers in the
  future
• The need for more general and intuitive ways to
  interact

6
Essence of PUI

• How people interact with each other and with real
  world
• PUIs are characterized by interaction techniques
  that combine understanding of natural human
  capabilities (particularly communication, motor,
  cognitive and perceptual skills) with computer
  I/O devices and machine perception and reasoning.

7
PUI integrates perceptive, multimodal and
multimedia interfaces

• Perceptive UI
• Adding human-like capabilities to the computer
• Making the computer aware of what the user is
  saying or what the users face, body and hand are
  doing
• Multimodal UI
• Emphasizes human communication skills
• Using multimodal output modalities to engage
  human perceptual, cognitive and communication
  skills
• Multimedia UI
• Focuses on the media

8
Challanges

• Oviatt and Cohen
• Summarize multimodal interfaces
• Pentland
• Perceptual interfaces smart room, smart cloths
• Crowley et al
• Vision-based sensing and perception of human
  activity
• Visual perception to enhance graphical interfaces
• Reeves and Nass
• The need to better understand human perception
  and psychology as it relates to interaction with
  technology
• Bobick
• Large scale PUI application called KidsRoom

9
Multimodal Interfaces (Oviatt and Cohen)

• Toward embracing users natural behavior as the
  center of the human-computer interaction
• Our voice, hands, entire body, once augmented by
  sensors such as microphones and cameras, are
  becoming the ultimate transparent and mobile
  multimodal input devices

10
Why multimodal?

• Accessibility for diverse users and usage
  contexts
• Usage for different ages, skill levels, cognitive
  styles, sensory and motor impairments, native
  languages
• Performance stability and robustness
• Expressive power and efficiency
• For accessing and manipulating information, such
  as increasingly sophisticated visualization
  capabilities

11
Perceptual Intelligence (Pentland)

• Desks, doors, TVs, cars, eyeglasses, shoes are
  changing from static, inanimate objects to
  adaptive, reactive systems that can be more
  friendly, useful and efficient
• Ethological view of behavior
• Most appropriate, adaptive biological behavior
  results from perceptual apparatus
• Classifies the situation correctly
• Triggers simple, situation-specific learned
  responses
• In strong contrast to cognitive theories
  adaptive behavior is primarily the result of
  complex reasoning mechanism

12
Perceptual Intelligence - cont

• Key point
• To make machines aware of their environment and
  sensitive to the people who interact with them
• They should
• Know who we are
• See our expressions and gestures
• Hear the tone and emphasis of our voice
• Not George Orwells dark vision of a government
  observing your every move
• Local intelligence

13
Smart Rooms
14
Smart Rooms

• Where are the people?
• 2D, 3D tracking
• Who is it?
• recognition
• Facial expression detection
• To detect if the student is bored
• Recognize hand and body gestures

• Danger modifies user behavior?

15
Smart Clothes

• Computer can help you find your way by whispering
  directions to your ear
• Scenario for large displays
• You are collaborating with couple other people
• Your smart cloth can listen to you and notify you
  only when you are not talking to others by
  whispering Do you want to see the detail view
  of that section you are looking? only after you
  have looked to a specific region on the large
  display for more than 2 secs.
• It automatically updates the view only
• If you are autorized to do so
• If you confirm the systems request

16
Haptic Interfaces (Tan)

• Haptic sensory system has two components
• Tactile awareness of the stimulation to the
  outer surface of the body
• Kinesthetic awareness of limb position and
  movement
• Unlike vision and audition (mainly input systems
  to human), haptic system is bidirectional

17
Haptic Interface - Example

• Imagine a chair to communicate with large
  high-resolution displays
• The system tracks the sitting postures of the
  user
• Automatic control of the display, pan and zoom

18
Things that see (Crowley et al)

• Computer-vision based sensing and perception of
  human activity
• Research in HCI developed cognitive theories,
  design methods for building useful systems
• However, direct manipulation still in the form of
  desktop metaphor
• Jeopardizing the directness and affordance of the
  physical world

19
Magic Board
20
Perceptual Window

• Hand and mouse form the dominant stream
• Head is used as non-dominant stream
• Better than eye tracking
• Fixation and saccades

21
Perceptual Bandwidth (Reeves and Nass)

• The need to better understand human perception
• How machines might change or facilitate human
  perception
• Speaking, touching, gesturing, emoting, gazing

22
Definitions of Perception in HCI
23
Psychology of perception

• Stimulation of the senses
• Chemical senses (taste and olfaction)
• Cutaneous senses (skin and its receptors)
• Vision and hearing
• Vision
• visual mechanics, color, brightness and contrast,
  objects and forms, depth, size and movement
• Hearing
• Psychophysics (loudness, pitch, sound
  localization)
• Physiological mechanisms (auditory components of
  the ear, neural activity associated with hearing)
• Perception of speech (units of speech, such as
  phonemes and mechanics of wod recognition)

24
Implications for perceptual interfaces

• Social issues
• How interfaces should present consistent
  personalities, various social roles
• Prominence of vision and hearing
• Human speech
• Sense of touch
• Mechanoreceptor (indentation of the skin)
• Thermoreceptor (changes in the temperature)
• Nociceptor (intense pressure and heat)

25
Their conclusion

• New media engage old brains to the extent that
  new interactions mimic real life
• Then the principles that explain perception in
  real life can be applied straightforwardly to
  computers and other media
• Perception of
• Motion
• Can guide attention
• Novelty
• Novel people and places are perceptually more
  interesting than familiar ones
• They get attention
• Visual discontinuity increase cortical arousal
  (indicated by brain patterns during viewing)

26
Display size and the perception of media content

• The perception of size and distance is a
  significant psychological issue, largely because
  size matters a lot in the world of perception
• How far are you from danger or opportunity
• Size is a benefit in everything from job
  interviews to presidential elections (the taller
  candidate almost always wins)

27
Larger displays?

• Their research shows larger displays are
  preferred and they create greater sense of
  presence (see their ref 10)
• Larger displays are more arousing
• measured by skin conductance levels and heart
  deceleration when visual material first appears
• Solid relationship between arousal and memory
• The higher the arousal the better the memory
• The arousal competes for the same mental effort
  that could otherwise be given to thinking hard
  about information
• So, best not to overdo size when learning and
  memory for information are the important goals

28
KidsRoom (Bobick et al 2000)
29
Part II

• Productivity Benefits of Very Large Displays

30
Performance Benefits

• Larger displays are becoming increasingly
  available due to multi-monitor capability built
  into many systems and rapid decrease in their
  cost
• Little is know about their performance benefits
• Current software designs and interaction
  techniques have been tuned to these large
  displays?

31
Problem with single monitors

• Most users posses displays whose display surface
  area is less than 10 of their physical workspace
  area
• Productivity benefit with larger displays
• Task time, and overall preference
• Less cognitive load
• Reduction in the need of window management

32
Study

• Multi-application office work
• Significant benefits in the use of larger display

33
Part III

• Human Memory

34
Infocockpit

• Providing location and place to aid human memory
• Goal
• to build computer system interfaces that make
  information memorable
• People remember spatially distributed information
  based on its location relative to their body, as
  well as the environment in which the information
  was learned.

35
System
36
Various topics
37

• Cognitive load
• Perceptual and physical limitations of human
• Visual scalability

38
Basic perceptual issues of large high-resolution
displays.

• Questions from BalN05
• How do these results change as the data/pixel
  count scales up?
• How do the different navigation strategies, such
  as overviewdetail, and focuscontext, affect
  high resolution visualization?
• As resolution scales up, what are the physical
  navigation tradeoffs with large high resolution
  screens?
• How do our results differ when using a
  non-bezeled tiled display?

Ball R and North C. Effects of Tiled
High-Resolution Display on Basic Visualization
and Navigation Tasks. CHI 2005 (Portland,
Oregon), ACM 1196-1199
39
References

• Paper1
• TurR00 Turk, M. Robertson, G. (Eds.), 2000.
  "Perceptual user interfaces," Communications of
  the ACM (special issue), 43(3), 32-70.
• ReeN00 Reeves, B. and Nass, C. 2000. Perceptual
  user interfaces perceptual bandwidth. Commun.
  ACM 43, 3 (Mar. 2000), 65-70.
• Paper2
• Czerwinski, M., G. Smith, T. Regan, B. Meyers, G.
  Robertson and G. Starkweather. Toward
  Characterizing the Productivity Benefits of Very
  Large Displays. INTERACT, 2003, pp. 9-16.
• Paper3
• Tan, D.S., Stefanucci, J.K., Proffitt, D.R.
  Pausch, R. (2001) The Infocockpit Providing
  Location and Place to Aid Human Memory. Workshop
  on Perceptive User Interfaces 2001, Orlando,
  Florida.
• Others
• BalN05 Ball R and North C. Effects of Tiled
  High-Resolution Display on Basic Visualization
  and Navigation Tasks. CHI 2005 (Portland,
  Oregon), ACM 1196-1199.