Mitsubishi Electric Research Labs MERL

1
Instant Replay Inexpensive High Speed Motion
Capture
Hideaki Nii, Jay Summet, Yong Zhao, Jonathan
Westhues, Paul H. Dietz, Shree K. Nayar, John
Barnwell, Michael Noland, Vlad Branzoi, Erich
Bruns, Masahiko Inami, Ramesh Raskar

• Mitsubishi Electric Research Labs (MERL)
• Cambridge, MA

2
Demo Video
3
Optical Capture

• Goal
• Find 2D or 3D location of tags
• Approach
• Each tag has a photosensor
• Label space using zones created by special high
  speed projectors
• Find zone in which the tag lies

4
Key Features

• 500 Hz Tracking
• Id for each Marker Tag
• Capture in Natural Environment
• Visually imperceptible tags
• Photosensing Tag can be hidden under clothes
• Ambient lighting is ok
• Unlimited Number of Tags Allowed
• Base station and tags only a few 10s

5
Potential Applications

• High Speed Motion Capture Tracking
• Motion Capture
• Character Animation
• CG alignment
• Industrial applications
• Crash Test Dummies
• Robot Guidance
• Indoor Positioning for Ubiquitous Computing
• Locating users and devices
• 3D User Input

6
Existing Solutions

• Optical Uses expensive cameras to locate tags in
  the environment.
• Passive tags give location, but no identity,
  leading to reacquisition problems
• Magnetic
• Environmental interference, expensive
• Inertial Tracking (Gyro/ Compass)
• Drift, Calibration

7
Solution Overview

• Use inexpensive solid state projectors to label
  space.
• Solid State lighting (LEDs) can be modulated
  extremely quickly (MHz to GHz).
• Each point in space receives a different code
  sequence.
• Inexpensive and small tags receive the codes, and
  can detect their own location.

8
Labeling Space
Each tag receives a unique signal based upon its
physical location.
9
How Labeling Works
Light source
Screen
Optics
GrayCode Mask
pos0
pos15
Light source blink one by one and each position
on the screen has different light pattern. 4
light make 4 bit position resolution
10
System Hardware

• One projector can send 10 bit one dimensional
  position data.
• No limitation on the number of tags.

11
Inside of Projector
Condensing Optics
Light Source
Focusing Optics
Gray code Slide
The Gray code pattern
12
Inside of Tag
13
Specification
14
Low cost
Proto type system
Mass Produced System

• Projector
• Electrical parts50
• Projector body10
• Optics hundreds
• (4 custom glass lenses)
• Tag
• Electrical parts50
• Wireless unit50

• Projector
• Electrical parts30
• Projector body10
• Optics10 (Plastic lens)
• Tag
• Electrical parts10
• Wireless unit Optional

15
Expand to 2D/3D System
2D measurement
3D measurement
X2 data
X data
X data
Y data
Y data
16
Result Videos
17
Tags Hidden Under Costume

• Tags can be hidden under an actors costume with
  only small sensors exposed.
• This enables motion capture during primary
  filming.

18
High Speed Tracking
19
Flying Tags!
20
E-Tech Demo
21
Key Features

• 500 Hz Tracking
• Id for each Marker Tag
• Capture in Natural Environment
• Visually imperceptible tags
• Photosensing Tag can be hidden under clothes
• Ambient lighting is ok
• Unlimited Number of Tags Allowed
• Base station and tags only a few 10s

22
Take Away Points

• Very Inexpensive Projectors and Tags.
• Reasonable accuracy which will be improved with
  better optical design.
• Very fast sampling rates possible!
• We want to license the technology!

23
Thank you!

• Contact
• Ramesh Raskar raskar_at_merl.com

24
New Smaller Tag