Rendering and Display for MultiViewer TeleImmersion

1
Rendering and Display for Multi-Viewer
Tele-Immersion

• Andrew Nashel
• Advisor Henry Fuchs
• Department of Computer Science
• The University of North Carolina at Chapel Hill

2
Multi-viewer tele-immersion
UNC mockup
Cisco TelePresence 3000
Several users at each site looking at a common
display
3
A face-to-face group meeting
Top down view of a meeting around a conference
table
4
A face-to-face group meeting
Top down view of a break room scenario
5
Two important spatial cues are not supported by
commercial systems
Gaze awareness is the ability to gauge the
current object of someone elses visual
attention. Monk Gale 2002 Eye contact is a
special case of gaze awareness.
6
Why are these cues important?

• Face-to-face communication superior to video
• Cooperation (Drolet Morris, 2000)
• Trust (Rocco, 1998 Bos et al., 2002)
• Turn taking (Vertgaal et al., 2000)
• Without gaze awareness, video can be worse than
  audio-only
• Video may convey incorrect eye contact (Short et
  al., 1967)
• Video can increase interruptions (Argyle et al.,
  1968)

7
Gaze awareness improves performance

• Spatially correct video improves trust and task
  completion
• (Nguyen Canny, 2007)

8
My focus Improving gaze awareness in group
tele-immersion

• Improving spatial cues on conventional 2D
  displays
• New displays to provide distinct views to each
  user

9
Thesis statement

• Through a combination of rendering techniques and
  display engineering, we can provide more
  personalized experiences to individuals in a
  group of viewers via
• virtual camera views that improve local spatial
  cues while preserving scene continuity
• multi-view displays that (a) trade off
  stereoscopic viewing for a wider range of viewing
  positions or (b) use randomization to eliminate
  the spatial viewing conflicts that occur at
  regular intervals.

10
Outline

• Problem statement
• New solutions
• Conventional displays new rendering techniques
• Line light field
• Depth-dependent camera
• Video silhouettes
• New displays personalized views
• Monoscopic multi-view
• Random hole display
• Conclusions and future work

11
Live line light field session
12
Active view control
View steered to left
View steered to right
or
Stationary book
13
Outline

• Problem statement
• New solutions
• Conventional displays new rendering techniques
• Line light field
• Depth-dependent camera
• Video silhouettes
• New displays personalized views
• Monoscopic multi-view
• Random hole display
• Conclusions and future work

14
Limitations of line light field

• We have assumed
• Users at same depth.

15
Problem Location of virtual camera
In a room with people at different distances.
Virtual camera
16
Viewing error eye contact

• Gaze error for a remote person is the angular
  deviation between the displayed image and the
  actual gaze direction of the remote user.

x
z
?
zcam
Gaze error per user ? abs (arctan (x/(zzcam))
17
Solution Depth-dependent rendering

• From back to front we render a slice of the scene
  at that depth from a viewpoint that moves from
  near to far and composite the slices.

Render slices of the scene from a virtual camera
at varying depths
18
Depth-dependent slices
19
Depth-dependent camera image
20
DDC vs conventional rendering
Depth-Dependent Camera
Conventional camera at 16
21
DDC vs conventional rendering
Depth-Dependent Camera
Conventional camera at 48
22
Gaze error
? abs (arctan (x/(zzcam))
23
Depth-Dependent Camera Benefits
Simultaneously provides 1) local gaze awareness
people close to display can make eye contact 2)
significant perspective depth cues
24
Outline

• Problem statement
• New solutions
• Conventional displays new rendering techniques
• Line light field
• Depth-dependent camera
• Video silhouettes
• New displays personalized views
• Monoscopic multi-view
• Random hole display
• Conclusions and future work

25
Limitations of depth-dependent camera (and other
reconstructions)

• Good image synthesis requires
• Good depth estimation, or
• A very large number of closelyspaced cameras

Erroneous image synthesis from 9 cameras
Reconstruction is still not of acceptable quality
compared to ordinary video cameras with real
scenes (multiple people, objects, occlusions,
movement).
26
Observations

• Reconstruction introduces visual artifacts
• Image quality is most important for a users face
• Implications
• Avoid 3D reconstruction of the users face
• Use direct camera imagery

27
One-camera-per display problems
C
D
A in one view B unseen C in two views D at
optimal depth
A
B
28
Video of one-camera-per display
29
Video Silhouettes Reconstruction using direct
camera imagery

• Segment people in each camera image
• Match segments between images
• Choose best camera view for each person

C
D
A
B
Cameras 1 2 3 4 5
6 7
30
Video of 7 camera linear array
31
Video of segmented objects
32
Video of silhouette reconstruction
33
Outline

• Problem statement
• New solutions
• Conventional displays new rendering techniques
• Line light field
• Depth-dependent camera
• Video silhouettes
• New displays personalized views
• Monoscopic multi-view
• Random hole display
• Conclusions and future work

34
Autostereoscopic displays

• Provide different imagery at different viewing
  angles to the display without worn encumbrances.
• For stereoscopic display, these views must be
  spaced by the interpupillary distance (IPD) or
  less (6cm).
• Volumetric too small, difficulty with occlusion
• Holographic too small, slow update rates
• Parallax Lenticular and barrier

35
Total viewing zones width
View repeat distance
48cm
Interpupillary distance
6cm
48cm
1
2
3
4
5
6
7
8
1
2
3
4
5
6
7
8
1
2
3
4
5
6
7
8
Calibrated viewing distance
Limitations of parallax autostereo displays
36
Conflicting viewing regions
Available regions
1
2
3
4
5
6
7
8
1
2
3
4
5
6
7
8
1
2
3
4
5
6
7
8
Calibrated viewing distance
Limitations of parallax autostereo displays
37
Outline

• Problem statement
• New solutions
• Conventional displays new rendering techniques
• Line light field
• Depth-dependent camera
• Video silhouettes
• New displays personalized views
• Monoscopic multi-view
• Random hole display
• Conclusions and future work

38
Custom wide field-of-view lenticular display
39
Video of monoscopic wide FOV display
40
Outline

• Problem statement
• New solutions
• Conventional displays new rendering techniques
• Line light field
• Depth-dependent camera
• Video silhouettes
• New displays personalized views
• Monoscopic multi-view
• Random hole display
• Conclusions and future work

41
Total viewing zones width
View repeat distance
48cm
Interpupillary distance
6cm
48cm
1
2
3
4
5
6
7
8
1
2
3
4
5
6
7
8
1
2
3
4
5
6
7
8
Calibrated viewing distance
Limitations of parallax autostereo displays
42
Conflicts between two stereo viewers
100
1
2
Percent interference
Display
0
Second user position
43
Simulation of the conflicts between two stereo
viewers
(x,y)
(1m, 1m)
Newsight/X3D 19 display 4.4mm barrier to display
spacing 9 views
43
44
Fix this with a new barrier design
Random hole barrier
Regular hole barrier
45
Random hole barrier simulation
(x,y)
(1m, 1m)
Random hole 19 display 4.4mm barrier to display
spacing 19 hole to barrier
46
Interference comparison
47
Poisson disk distribution
With a purely random distribution, samples bunch
in places and leave gaps in others.
Poisson disk is random sampling with a minimum
distance constraint
48
Fourier transforms of barrier patterns
Regular barrier
Random barrier
Poisson disk barrier
49
Viewing conflictspixels visible by multiple
viewers

• 1) Turn pixels black this decreases display
  brightness
• 2) Blend colors this maintains brightness but
  introduces error
• 3) Blend only similar colors
• Future work Blend and error-diffusion dither.

50
Prototype construction
Plastic barrier sheet
LCD panel
glass spacer
glass cover
Viewers
0.02
0.25
Thickness
0.0625
51
Prototype display
Close up photo of mask
52
Calibration
Which pixels are visible at each eye
position? Capture from viewer location with two
high resolution cameras in stereo configuration
53
Per color channel calibration
Red
Blue
Green
Each color channel is captured separately and
then combined to form the final view mask
53
54
First calibrated views
Left and right eye views for one stereo viewer
1.5m
55
Generating imagery for two users
3m
1.5m
56
4 simultaneous images
Image sent to display
Photographs from two stereo viewing positions
57
Depth perception
Left and right eye source images
Image sent to display
Is the red square in front or behind the green
box? Most were able to accurate judge relative
position and some could accurately estimate the
depth of the square.
58
An example stereo pair
(Streaking from manufacturing defects)
59
Outline

• Problem statement
• New solutions
• Conventional displays new rendering techniques
• Line light field
• Depth-dependent camera
• Video silhouettes
• New displays personalized views
• Monoscopic multi-view
• Random hole display
• Conclusions and future work

60
Thesis statement revisited

• Through a combination of rendering techniques and
  display engineering, we can provide more
  personalized experiences to individuals in a
  group of viewers via
• virtual camera views that improve local spatial
  cues while preserving scene continuity
• multi-view displays that (a) trade off
  stereoscopic viewing for a wider range of viewing
  positions or (b) use randomization to eliminate
  the spatial viewing conflicts that occur at
  regular intervals.

61
Contributions

• Conventional displays new rendering techniques
• Line light field steerable virtual camera
• Depth-dependent camera local gaze awareness and
  perspective cues
• Video silhouettes camera-quality reconstruction
  of people
• New displays personalized views
• Monoscopic multi-view feasible multiple viewing
  positions
• Random hole display autostereo at arbitrary
  positions for multiple viewers

62
Future opportunities

• Depth-dependent camera with real world imagery
• Video silhouettes with improved geometric proxies
• Tabletop-surface autostereoscopic display with
  random hole barrier
• Animatronic Shader Lamps Avatars

63
Acknowledgements

• My advisor Henry Fuchs
• My committee members Ketan Mayer-Patel,
• Leonard McMillan, Leandra Vicci, and Greg Welch
• The research staff, Herman Towles and Andrei
  State
• The technical staff, John Thomas, David Harrison,
  and Bil Hays
• The entire administrative/support staff,
  especially Janet Jones
• Student collaborators Peter Lincoln, Andrei
  Ilie, Ruigang Yang
• Supported by
• Sandia National Laboratories/California
  Christine Yang, Corbin Stewart
• Cisco Systems Mod Marathe, Bill Mauchly

64
Acknowledgements

• My many friends, including fellow UNC students
  Sharif Razzaque, Yuanxin Liu, Greg Coombe, David
  Marshburn, Mark Harris, Eli Broadhurst, Dan
  Samarov, and many more.
• My girlfriend Megan Orrell.
• And especially my parents, to whom this
  dissertation is dedicated.

65
Questions?
Line light field
Depth-dependent camera
Thank you. Questions?
Monoscopic multi-view
Video silhouettes
Random hole display