Recent Development of ImageBased Rendering ZHANG Xiaohui Nakajima Lab' Tokyo Institute of Technology

1
Recent Development of Image-Based
RenderingZHANG XiaohuiNakajima Lab.Tokyo
Institute of Technology
2
What Is an Image

• A 2D array of pixels
• (a continuous function on )
• each pixel (x,y) has
• a RGB (and ?) value
• more ?

3
What Is Rendering (1)

• Generation of a 2D image from a 3D scene
• The rendering pipeline
• Modeling
• Arranging geometric primitives in space
• Assembling objects from sets or hierarchies of
  primitives
• Assigning appearance parameters to the objects
  (color, shininess, texture, transparency)
• Describing how the objects move over time
  (animation)

4
What Is Rendering (2)

• Visibility
• Hidden-line
• Hidden-surface
• Hidden-volume
• Shading
• Display (frame buffer, z-buffer, CRT)

5
Computer Graphics
6
Problems of Geometric Model Based Rendering

• Modeling is hard
• lacks of realism
• Rendering is slow
• cost of rendering is dependent on the scene
  complexity

7
Computer Vision
8
Computer Graphics Computer Vision
9
(No Transcript)
10
What Is Image-Based Rendering?(1)
11
What Is Image-Based Rendering(2)

• Creating new views of a 3D environment based on
  existing images
• Advantages
• Speed independent of scene complexity
• Source of images real or synthetic

12
Previous Work Before Image-based Rendering

• Texture-mapping
• Environment-mapping
• Movie Map

13
Categories of Image-based Rendering

• Image mosaics
• Interpolation from dense samples (Light field
  rendering)
• Geometrically Valid Pixel Reprojection
• CG Rendering Acceleration

14
Image Mosaics
Different Images
combination
Higher resolution or lager image
15
Mosaic Image Representation
Planar Image
Cube
Sphere
Cylinder
16
Interpolation From Samples
Interpolation Morphing
Novel image
gt2 images
17
Image Morphing

• Rearrange pixels in an image

18
View Morphing (1)

• Morphing between parallel views
• parallel views
• camera position
• camera parameter projective matrices
• linear interpolation of pixels of two images.

19
View Morphing (2)
20
View Morphing (3)

• Morphing between Non-parallel views
• Prewarp
• Morph
• Postwarp

21
View Morphing (4)
22
Light Field Rendering (1)

• The Plenoptic Function

23
Light Field Rendering (2)
24
Light Field Rendering (3)

• 4D Light Field
• a snap shot in time
• monochromatic wavelength
• convex hull of a bounded object

25
Light Field Rendering (4)
26
Light Field Rendering (5)
27
Light Field Rendering (6)

• Creating a Light Field

28
Light Field Rendering (7)
29
Light Field Rendering (8)
30
Light Field Rendering (9)

• Array of Images

31
Light Field Rendering (10)

• Fast Rendering

32
Light Field Rendering (11)

• 4D Interpolation

33
Light Field Rendering (12)
34
Light Field Rendering (13)
35
Light Field Rendering (14)
36
Light Field Rendering (15)
37
Light Field Rendering (16)
38
Light Field Rendering (17)
39
Light Field Rendering (18)
40
Light Field Rendering (19)
41
Light Field Rendering (20)

• Limitation
• Sampling density must be high
• Large, densely occluded environments
• Fixed illumination, static scenes

42
Image Warping
43
Corresponding Pixel
P
Camer1
Camer2
44
Perspective Projection
O
45
Warping Equation
46
Special Cases

• Infinite Depth
• Translation Invariant
• Environment Map
• Co-planar points
• Texture mapping
• Share a common center-of-projection-- Texture
  mapping
• Nearby images
• 2D affine transform

47
Problems of Image Warping

• Visibility problem
• ---More than two pixels map to the same pixel
  in the desired image
• Exposure Errors
• --- A background region that should have been
  occluded is visible in a desired image
• Occlusion Errors
• --- a false foreground element that covers
  background regions visible in the actual scene

48
Geometrically-Valid Pixel Reprojection

• Transfer method
• Use relatively small number of images
• Use geometric constraints
• Epipolar constraints
• Trilinear tensors

49
Epipolar Geometry (1)
Left image plane
Right Image Plane
Left Optical Camera
Base Line
Right Optical Camera

• Epipole
• Epipolar plane
• Epipolar line

50
Epipolar Geometry (2)
Camera 1
Camera 2
51
Epipolar Geometry

• Fundamental Matrix
• Point is in the novel view image

Point in image 0 point
in image 2 F fundamental
matrix of rank 2
52
CG-Rendering Acceleration

• Depth Image
• RGB(?) value
• a Z (depth) value
• a Normal
• Depth image generation
• Ray tracing
• Z buffer

53
CG Rendering Acceleration (2)

• Sprite
• 2D affine or projective transformation
• Impostors
• Transparent polygon rendered with an opaque image
  of the object mapped onto it
• QuickTimeVR
• Commercially available image-based CG system
• cylindrical images

54
Depth Image Rendering
55
Special Case

• Infinite Depth
• Translation Invariant
• Environment Map
• Co-planar points
• Texture mapping
• Nearby images
• 2D affine transform

56
Image-Based RepresentationsComparison
Representation
Movement
Geometry
Lighting
Geometry Materials
Continuous
Global
Dynamic
Geometry Images
Continuous
Global
Fixed
Image Depth
Continuous
Local
Fixed
Light Field
Continuous
None
Fixed
Movie Map
Discrete
None
Fixed
Panorama
None
None
Fixed
57
Open Problems of Image-based Rendering

• Real imagery computer vision
• Camera pose hard to get
• Depth even harder to get
• Data size maybe huge
• Changes Difficult
• light
• geometry

58
My Research Work on IBR

• Research on video image based driving simulation
  (Forward Moving Image Mosaics)
• View Synthesis based on layered images
• Virtual view for different weather condition

59
Structure of Our Image-based Modeling and
Rendering System (1)
Forward Moving Video Image
Multi-structure Database
Realistic virtual view
Geometry Data
Modeling
Data Acquisition
Rendering
60
Structure of Our Image-based Modeling and
Rendering System(2)

• Depth abstraction
• Layer ordered on depth
• Lighting information

Modeling

• Geometry data
• View-dependent texture
• Depth image
• Video image
• Lighting information
• Visibility information

Multi-structure database
Rendering

• Reprojection
• blend
• weather and season change

61
Video Image-Based Modeling and Rendering
Algorithm for Driving Simulation System

• Real-time is essential
• Viewpoint does not change violently
• The change of viewpoint is almost horizontal
  movement

62
Forward Moving Image Mosaics (1)

• The feature of forward moving image sequence

The wider view of information is included in
the forward moving image sequence
63
Algorithm (2)

• Depth Recovery
• Levenberg-Marquardt algorithm

Motion parameters
d depth value of pixel (x, y)
64
Algorithm (3) Depth Image Reprojection
65
Algorithm (3)
Blended image
low resolution
Large field of view
Viewpoints
High resolution
Rotate transform
Source image
Reprojected image
View angle of virtual viewpoint
Virtual View
View angle of source image
View angle of reprojected image
66
Experiment Result
67
View Synthesis Based on Layered Images

• Generate layered image based on depth value
• Use different approaches for each layer
• environment mapping
• view-dependent texture
• depth reprojection
• Blend layered images virtual view

68
Generate Layered Image
69
Generate Virtual View Based on Layered Image