New dimensions of media

1
New dimensions of media
Michael M. Bronstein
Department of Computer Science Technion Israel
Institute of Technology cs.technion.ac.il/mbron

MMSN, San Jose 1 November 2007
2
Dimensions of media
Radio
Color television
Black-and-white television
3D video
3
Conventional 2D TV
You see exactly what the camera shot
Video Bullettime
4
Free viewpoint TV (FTV)
Interactive selection of viewpoints
Video Bullettime
5
Augmented reality
Place 3D generated objects in the scene
Video Japanese TV
6
Emerging computer vision applications
Analysis of 3D non-rigid objects
3DV systems
Bronstein et al, 2003
Gesture recognition (user interface)
Face recognition (biometrics)
7
3D/Stereoscopic TV
Three-dimensional depth perception of the scene
8
Evolution
1851 Brewster streoscope sold in London
1891 Anaglyph is invented
1952 First color 3D movie premiere
Today 3D content available User-grade 3D
displays 3DTV broadcast
1922 First 3D movie premiere
Early 2000s Commercial 3D monitors appear
?
1800
1820
1840
1860
1880
1900
1920
1940
1960
1980
2000
1839 Talbot invents photographic process
1838 Wheatson explains binocular vision
1928 First television broadcast in USA
1995 First IMAX 3D fiction movie
9
Ecosystem
Computer graphics
3D video acquisition
3D scene representation
Coding Transmission
Rendering Display
10
Our 3D perception

• Binocular vision
• Each eye sees a slightly different picture
• Depth perception (stereopsis) achieved by merging
  two pictures

11
Shape from stereo
Closer objects have larger parallax

• Depth recovered from parallax (disparity) between
  corresponding points
• Correspondence problem
• Can be generalized to multiple views

Left eye
Right eye
Parallax
12
Stereo cameras
Multiview camera array (Carnegy Mellon)
IMAX 3D camera
NASA Mars rover
13
Structured light
Moving projector
Camera

• Active sensor
• 3D shape extracted from deformation of the
  projected light pattern
• Typically slow and unsuitable for fast moving
  objects

14
3D on your desk
Can be literally done using just a pencil and a
lamp!
Jean-Yves Bouguet, 1998
15
Coded light
0
1
0
1
1
0
0
Camera
Projector
Light pattern
Angle

• Sequence of black/white patterns projected onto
  the object
• Patterns form a binary code encoding the angle
  relative to the projector
• Depth recovered by triangulation
• Active sensor

16
Time-of-flight
Nanosecond gate
Object
Object
Sensor
Transmitted pulse of light
Reflected pulse of light

• Principle of a laser range finder
• Distance to surface measured by timing the travel
  of a pulse of light
• Requires nanosecond gating
• Active sensor

17
3D scene representation
Stereo pair
Depth map
Mesh
Layered depth map (2.5D)
IMAGE-BASED
MODEL-BASED

• No 3D reconstruction
• Allows view synthesis
• Natural for stereo cameras

• Explicit 3D model
• Requires rendering
• Used for synthetic CG content

18
Stereo pair
Left
Right

• Left and right views are stored side-by-side
• Can use existing video compression and
  distribution standards
• Large redundancy special compression algorithms
  are desirable

19
Stereo compression
Motion-compensated prediction (MCP)
CONVENTIONAL VIDEO
I
P
P
P
I
P
P
P
I
Motion-compensated prediction
I
P
P
P
I
P
P
P
I
Disparity-compensated prediction (DCP)
STEREOSCOPIC VIDEO
I
P
P
P
I
P
P
P
I
Motion-compensated prediction
20
2DZ
Conventional video (2D)
Depth (Z)

• 2D video and corresponding depth map are stored
  side-by-size
• Uses existing video compression and distribution
  standards
• Z-component increases bandwidth only by 5-20
• Used by Philips (WOW vx)

21
Geometry compression
Trajectory compression
Motion trajectories
Common representation coding
Motion analysis
3D video
Mesh
Mesh compression

• MPEG spirit
• Compress mesh and motion separately
• Active research field

22
Correspondence again
Bronstein et al, 2007

• Mesh motion displacement of corresponding
  points
• Non-rigid shapes are the biggest challenge
• Intrinsic geometric correspondence
• Methods borrowed from data mining (generalized
  MDS)

23
Mesh compression
Hoppe, 1996
Progressive mesh compression

• Both similar and dissimilar to image compression
• Laplace-Beltrami (spectral methods)
• Successive approximation (progressive meshes)
• Connectivity compression

24
Viewpoint selection

• Capture the scene with multiple cameras
• closely located
• Transition between different views
• If the cameras are dense enough, the
• transition will be smooth
• Used in QuickTime VR
• Only existing viewpoints

Object
25
Panorama

• Produce a wide-angle or 360 degrees panorama
• Change the viewpoint by selecting a subset of
  the panorama
• Used in QuickTime VR
• Limited range of viewpoints

26
Image-based rendering

• Rendering based on existing views
• instead of geometric model
• Light field 2D collection of images
• (4D array) time
• New viewpoint 2D slice of the 4D
• array, produced by interpolation
• Requires multiple views (tens)
• Problems with shadows,
• specularities, etc.

Existing views
New view
Visualization of light field
27
3D display
How to create three-dimensional depth perception
of the scene? Display to each eye the image it
would see
Head mounted display (VR glasses)
IMAGE-BASED
MODEL-BASED

• Stereo camera eyes
• Multiview synthesize the views
• the eyes would see

• Render the views the eyes
• would see
• Computer graphics methods

28
3D on a big screen
Block unwanted rays
Screen
29
Anaglyph
Screen


Left
Right
Anaglyph

• Compatible with any color display
• Requires color glasses
• Unnatural colors

Color glasses
30
Polarized light

• Used in IMAX 3D
• Requires special projection
• Requires polarized glasses

Screen
Polarized projector
Polarized glasses
Polarized projector
31
Shutter glasses

• Left and right image displayed interleaved in
  time (double framerate)
• Shutter glasses keep only one of the eyes open at
  a time
• Requires screen and glasses sync

32
Do we need glasses?
33
Lenticular display

• Lenticular lens sending parts of the image
• to different eyes
• No glasses needed (autostereoscopic)
• Can be attached to a legacy monitor
• User-grade commercial products
• available from Sharp, Philips, etc.
• Viewing angle is a challenge

Screen
Lenticular lens
34
Volumetric display

• Display a periodically time-varying 2D image on a
  rotating mirror
• Illusion of 3D object due to visual persistence
• 360 degrees view

Rotating screen
Projector
Perspecta display
35
Summary acquisition
PASSIVE
ACTIVE
Coded light
Stereo
Structured light
Time-of-flight
36
Summary representation compression
PIXEL-TYPE DATA
GEOMETRIC DATA
Conventional (MCP-based)
Mesh compression
2DZ
Stereoscopic (DCP-based)
37
Summary display
HEAD-MOUNTED
STEREOSCOPIC GLASSES
VR glasses
Color glasses
Polarized glasses
Shutter glasses
VOLUMETRIC
AUTOSTEREOSCOPIC
Lenticular
38
Conclusion

• 3D applications are an active research and
  development field
• Another hype?
• Looks like 3D is going to become a commodity
• Still many challenges
• Wait 5-10 years and see