High Quality Novel View Synthesis Based on Low Resolution Depth Image and High Resolution Color Image

1
High Quality Novel View Synthesis Based on Low
Resolution Depth Image and High Resolution Color
Image
ICPR/WDIA-2012

• Jui-Chiu Chiang, Zheng-Feng Liu, and Wen-Nung
  Lie
• National Chung Cheng University
• Taiwan

2
Outline

• Introduction
• Our hybrid camera system
• The proposed depth processing algorithm
• Experimental results
• Conclusion

3
Color Depth in 3DTV/FTV

• Depth image, in combination with DIBR (Depth
  Image-Based Rendering) technique, is important in
  multi-view 3DTV/FTV applications.

Depth image
Color image
DIBR
view8
view1
view2
view3
view4
view5
view6
view7
view9
4
Depth acquisition

• From stereo matching via disparity estimation
• Unreliable for textureless regions
• Computationally intensive
• From active photo-electrical sensing
• Lower resolution
• Higher acquisition speed and accuracy

Kinect
SR4000 ToF
5
Hybrid camera systems

• High-resolution color cameras Low-resolution
  depth camera (mis-alignment between their optical
  axis)
• 12 Use warped data of depth camera to refine
  ROI disparity of the dual-eye camera
• Proposed one color image is used to refine the
  warped (to both sides) and up-sampled depth
  images

6
Processing diagram of our system
7
Generation of preliminary high-resolution depth
image

• Purpose to generate high-resolution depth image
  at the left and right sides

Calibrated camera parameters
8

• Pre-processing of the depth map
• 3?3 median filter
• Low resolution to high resolution depth warping
• Forward warping with a block size of 5?5 pixels
• Post-processing by morphological operation
• Morphological opening

9
Generation of high quality high-resolution depth
image

• Color-image-guided depth map refinement

10

• Binarizing the warped depth image to have
  foreground/background classification
• Color segmentation by mean-shift algorithm
• Label each segment as FG or BG
• Refine the corresponding depth values according
  to the segment categorization
• Bilateral filtering is performed for the edge
  parts of the modified depth image

11
Experiments system configurations

• Color image 640x480
• Depth image 176x144 (SR4000)
• Intel Core 2 Duo Q6600 2.33 GHz, and DDR2 800 4GB

12
Preliminary high-resolution depth image
After depth warping
After morphological operation
Traditional depth warping
13
High-quality high-resolution depth image
After correction
Mean-shift
Incorrect pixels
Bilateral filtering
14
A series of generated depth maps
15
Virtual view synthesis based on various depth maps
Hole filled
Bilateral filtering on edges
Bilateral filtering on edges Gaussian
Bilateral filtering
16
Speed performance
Procedure Time (sec)
Pre-processing Depth Warping 0.117
Pre-processing Color segmentation 0.742
Post-processing Bilateral filtering on edges 0.352
Post-processing Gaussian filtering 0.104
Novel view synthesis by DIBR (including hole filling) Novel view synthesis by DIBR (including hole filling) 0.144
17
Conclusion

• A hybrid camera system consisting of
  high-resolution color camera and low-resolution
  depth camera is proposed
• A high-quality high-resolution depth images can
  be obtained for the left and right eyes
  multi-view video plus depth (N2 channels).
• Future work
• Combine Kinect device and stereo camera to get
  better depth image and cheaper cost