MultiFinger and Whole Hand Gestural Interaction Techniques for MultiUser Tabletop Displays

1
Multi-Finger and Whole Hand Gestural
Interaction Techniques for Multi-User Tabletop
Displays

• Jason Letourneau

2
Purpose

• Problem in 2003
• hardware is evolving but software and ways users
  could interact and be understood was not for
  multi-input touch
• Social context
• meetings face to face usually come in front of a
  table
• a touch table for collaboration fits this niche

3
Background

• Interested in exploiting hand gestures people
  normally use when touching things on a table
• Came from a paper planning session they went
  through in moving an office
• Developed prototype software Room Planner
• Highlights
• competitive (personal) vs cooperative (shared)
  workspaces
• Natural gestures and appropriate feedback for
  interacting between spaces

4
Setup

• Top down projector on a table
• DiamondTouch used antennas to infer touch
• Detected disruption of signal strength and
  coupled user with electric signal
• Problems determining multiple touch from a single
  user
• Experimenters used temporal information from
  sequential image snapshots of table to gather
  more information about a gesture being performed
• Two Users at a table due to size constraints
• Cooperative Space was in the middle of the table
• Competitive Space was directly in front of the
  user

5
Interaction

• User can
• Move things into and out of cooperative space to
  and from the competitive space
• Can mimic the current state of cooperative space,
  edit individually, and re-integrate back into the
  cooperative space
• Context menus give normal processing functions
  like undo

6
Input

• 4 Categories
• Single finger
• Two finger
• Single hand
• Two hands

7
Examples

• Single finger
• Item selection
• Feedback due to 2D view point and obscuring of
  object under finger provided side view above
  selected object when finger was on it
• Used to bring up context sensitive menus with
  double tap gesture
• Used for flicking and catching items as
  interaction mechanism between users private
  spaces either copying or exchanging possession
  of items

8
Examples

• Two finger
• Rotation/Scaling
• Starts with one finger selecting the object and
  the other controlling the pivot
• Single Hand
• Rotate the entire room with flat hand
• Sweep away collections of furniture with vertical
  hand
• Tilted hand extended projection area and showed
  user private information specifically possible
  because of top down projection

9
Examples

• Two handed
• Selection box like in windowed operating system
  to select multiple objects
• Scatter a bunch of objects with sweeping motion

10
Experiment

• 5 people in hour long sessions each
• Each user used the table with a researcher at the
  same time to allow more interactive session like
  prompting questions to user instead of two
  novices using it
• Given a design scenario to solve that varied in
  room design goal - used to examine interaction
  techniques in different contexts

11
Results

• On the whole the gesture set they chose was well
  received and intuitive
• Many gestures are used today

12
Issues

• Displaying items under hand/finger hid them from
  view i.e. menu items
• Users wanted to resize selection planes with four
  fingers instead of the needed two and for axes
  snapping on rotation
• A user wanted more visual feedback on catching
  gesture
• Users wanted the flat hand to do the same thing
  as the vertical hand in terms of moving furniture
• Came up with some other more complex motions that
  combined hand configurations and one finger
  techniques

13
Further Work

• They are interested in letting users create
  custom gestures which furthers both their goals
  of selective awareness between individuals
  performing actions and expanding users comfort
  with gestures they are performing
• Scaling to a larger surface
• Allowing multi-person gestures

14
Conclusion

• Lots of these gestures (the simpler ones) are
  used for input today
• And are expanded to non-touch interfaces
• i.e. Tom Cruise Minority Report Clone
• Hardware and techniques have advanced

15
Hardware Technology in 2003

• Prior to study
• single user input
• computer vision to detect touch
• Advances at time of study
• DiamondTouch - antennas
• SmartSkin
• Dvit computer vision

16
Touch Tables Today

• Various techniques
• Jeff Han (2006) - Frustrated Total Internal
  Reflection i.e. Light refraction with cameras
• registers touch points from rear projection and
  sends data over UDP
• Capacitive - holds electrical charge - finger
  changes amount of charge at specific point
• Resistive - pressure causes components to touch
  and changes resistance at specific points
• Sound/vibration
• IR - sensors at corners of surface with IR beams
  traveling across surface - cameras capture the
  shadows and can triangulate the location
• All have tradeoffs with cost and scalability

17
Commercial

• DiamondTouch
• Still uses radio antennas, but receiver attached
  to chair is what determines uniqueness and who is
  doing
• Human as conduit
• Accuracy of multiple user touch identification
  seems to be way ahead of where it was
• Threshold of how many antennas are coupled what
  touch behavior is occurring
• Microsoft Surface
• surface computing
• uses IR from various points in a room
• Not sure if they can uniquely identify user
• http//www.popularmechanics.com/technology/industr
  y/4217348.html

18
Build Your Own

• Basically just need
• IR Camera
• IR LEDs
• Reflective Tape
• Software to detect input
• Johnny Lee
• Used a Nintendo Wii remote and IR LEDs custom
  software
• http//www.instructables.com/id/Low-Cost-Multi-tou
  ch-Whiteboard-using-the-Wii-Remo/