Novel View Synthesis using Multiple View Geometry

1
Novel View Synthesis using Multiple View Geometry

• Olga Sorkine
• June 2003

2
2D image of 3D object

• Images of 3D world are some projective
  transformations 3D ? 2D
• Expressed in matrix (linear) form using
  homogeneous coordinates

object in 3D
camera center
2D image plane
3
Relation between different images
Stereo (two views)
X
p
p
How are matching points in different images
related?
4
Relation between different images
p
X
p
p
How are matching points in different images
related?
5
Relation between different images
is trilinear tensor (volume matrix of 3?3?3)
6
Trilinear tensor relation
p
p
X
sj
rk
p
7
Motivation novel view synthesis

• We are given several images of a 3D world.
• Want to generate images of novel views without
  reconstructing the 3D model and rendering it
• Reconstructing 3D model is a hard task
• Rendering complex 3D models under complex
  lighting environment is hard

8
Motivation novel view synthesis

• Input two images of a statue
• Close view parameters relatively easy
  to establish correspondence

Images taken from the demo for Novel View
Synthesis in Tensor Space, S. Avidan and A.
Shashua, CVPR 1997
9
Motivation novel view synthesis

• Output images of novel views, without
  reconstruction of the 3D model

Images taken from the demo for Novel View
Synthesis in Tensor Space, S. Avidan and A.
Shashua, CVPR 1997
10
Outline

• Reminder of homogeneous coordinates and camera
  matrices
• Epipolar geometry two views
• Trilinear tensor three views

11
Literature

• What can two images tell us about the third
  one? O.D. Faugeras and L. Robert, European
  Conference on Computer Vision, 1994
• Algebraic functions for Recognition, A.
  Shashua, IEEE PAMI, 1995
• Robust Recovery of Camera Rotation from Three
  Frames, B. Rousso, S. Avidan, A. Shashua and
  S. Peleg, CVPR 1996.
• Trilinear Tensor The Fundamental Construct of
  Multiple-view Geometry and its Applications, A.
  Shashua, AFPAC, 1997
• Tensor Embedding of the Fundamental Matrix, S.
  Avidan and A. Shashua. Post-ECCV
  SMILE Workshop, June 1998
• Novel View Synthesis by Cascading Trilinear
  Tensors, S. Avidan and A. Shashua, IEEE
  TVCG, 1998
• Multiple View Geometry in Computer Vision, R.
  Hartley and A. Zisserman, Cambridge Press 2000

12
Homogeneous coordinates

• Add one dimension to our vector representation
• Representation of points at infinity

13
2D lines as homogeneous vectors

• 2D line representation ax by c 0
• Homogeneous vector (a, b, c)T
• (?a, ?b, ?c)T, ? ? 0 represents the same line
• Point p (x, y, 1)T is on line l (a, b, c)T
  ?
• lTp pTl 0

14
Cross product

• A line through two points p1 and p2 is l p1 ?
  p2
• p1T l ltp1, p1 ? p2gt 0
• p2T l ltp2, p1 ? p2gt 0
• Therefore, both p1 and p2 lie on l
• The intersection point of lines l1 and l2 is p
  l1 ? l2

15
Matrix form of cross-product

• a?b a?b ? b?a
• a? is the anti-symmetric matrix

16
Projective transformation

• Linear transformation of homogeneous coordinates
• From 3D model to 2D image 3?4 matrix

camera matrix
17
Simple pinhole camera model

• The camera center C is in the origin of R3
• The image plane is z f

principal point
Y
y
Z
x
f
X
18
Simple pinhole camera model

• The simple camera matrix

y
Iy / f y / z Iy fy/z
Iy
z
f
19
General pinhole camera model

• 1) The image plane origin may be translated

K
Y
(px, py)
Z
f
X
20
General pinhole camera model

• 2) The world coordinates may be rotated and
  translated

camera center (3?1)
rotation 3?3 matrix from world to camera
Y
(px, py)
Z
C
X
21
When the calibration is not known

• All we know is that the matrix looks like this

image of the origin
3?3 matrix of rank 3
C is the center of projection (the only point
that doesnt have a legal image)
22
Geometry of two views
X
p
p
C
C
23
The epipoles

• e is the image of C (second camera origin) under
  the first camera

X
p
p
C
e
e
C
24
The epipolar plane

• Clearly, C, C, e, e, X, p, p all lie in the
  same plane!
• A plane that contains the baseline CC is called
  epipolar plane

X
p
p
C
e
e
C
25
Epipolar lines

• Epipolar plane intersects the image plane at
  epipolar line
• The epipolar line will always pass through the
  epipole

X
p
p
C
e
e
C
26
Epipolar lines

• Epipolar line l is the image of the ray CX in
  the second camera

X
Q
p
p
C
e
e
C
27
Relation between corresponding points

• p is the image of X in the first camera
• p is the corresponding point in the second
  camera
• p must lie on the intersection of the image
  plane with the epipolar plane through C, p, e!

X
l
p
p
e
C
e
C
28
The basic incidence equation

• p must lie on the epipolar line in the second
  image plane
• pT l 0
• Derivation of l let the camera matrices be P,
  P
• PX p
• The ray CX (back-projection)
• p(?) Pp ?C ( PP I )
• Points C and Pp are on the ray
• The camera P sees the ray p(?) as
• l (PC) ? (P (Pp)) e ? P Pp
• ? l (e? P P) p
• F e? P P

29
The fundamental matrix

• l Fp
• F is a 3?3 matrix
• Expresses the incidence relation
• pT F p 0
• Fp is the epipolar line,
• so we get back pT l 0

30
Properties of the fundamental matrix

• Rank 2 (because Fe 0. Maps points to pencil
  of lines)
• Thus, 7 degrees of freedom out of 9 coefficients
• one constraint is det F 0
• scaling multiplying F by a constant doesnt
  change anything because of homogeneity
• F stays the same under projective transformation
  of the world
• PX (PH)(H ?1X)
• If p ? p are images of X, then after projective
  mapping H, p, p are images of H?1X ? p, p
  still correspond!

31
Finding the fundamental matrix

• Given 2 images with unknown camera matrices
• We can find F by 7 point correspondences (gives
  us 7 equations for the coefficients of F)
• Practical algorithms find many more
  correspondences and use a robust-statistics
  solution

32
Generating novel views
C3
e32
e31
e23
e13
e21
C2
C2
e12
C1
We have three cameras C1, C2, C3 eij denotes
intersection of CiCj with the image plane of Ci
33
Generating novel viewsWhat can two images tell
us about the third one?, O.D. Faugeras and L.
Robert, 1994
C3
p
e23
e21
P
e23
e12
p
p
e31
C2
C2
e13
C1
The image of P in the 3rd camera is the
intersection of the epipolar lines from C1, C2
34
The algorithm

• Given
• Fundamental matrices F31, F32 computed from 7
  point correspondences each, supplied by the user
• Point correspondences between the two given
  images
• This is found using optical flow algorithm
  Lucas and Kanade 81
• For each correspondence p ? p find the position
  of the corresponding point p in the third view
• Copy the intensity to p

p
p
p
35
Generating novel views

• Denote the fundamental matrix from Cj to Ci by
  Fij (i, j 1, 2, 3)
• The intersection of the epipolar lines is given
  by
• p (F31 p)?(F32 p)

36
Degeneracies

• Cant reproject points on the plane of C1, C2 ,
  C3 (no proper intersection of the epipolar lines)
• Numerical problems for all points close to that
  plane
• Epipolar geometry is not defined when Ci Cj
  (only rotational movement of the camera)
• Cant reconstruct third view if C1, C2 , C3 are
  on one line

37
Critical surfaces

• Problems in F recovery from point
  correspondences
• When the corresponding points originate from 3D
  object points that together with the cameras lie
  on ruled quadric, F is not uniquely defined
• Important case in practice if the corresponding
  points originate from a plane in 3D

38
Geometry of three views
C3
p
X
p
p
C2
C2
C1
How are corresponding p, p, p related?
39
Notations

• The three camera matrices (3?4)
• P1 I 0, P2 A v, P3 B
  v
• Corresponding points

40
Algebraic derivation of the tensor Algebraic
functions recognition, A. Shashua, 1995
41
Algebraic derivation of the tensor
second image plane
42
Algebraic derivation of the tensor
second image plane
43
Einstein notation

• Points upper indices p (p1, p2, p3)
  pi (i 1, 2, 3)
• Lines lower indices l (l1, l2, l3)
  li
• Contraction (summation) pi li p1 l1 p2 l2
  p3 l3
• Incidence relation 0 pi li (
  li pi )

44
Einstein notation

• Matrix representation
• mapping point to point, line to line
• mapping point to line
• mapping line to point

i runs over columns
j runs over rows
Ai are columns of A
45
Algebraic derivation of the tensor

• Summarizing the derivation for x and y

46
Algebraic derivation of the tensor

• We can do the same derivation for x, y using
  the third camera matrix B v

47
Algebraic derivation of the tensor

• Now we eliminate from the two equations

48
The trilinear tensor

• We got a relation between two lines and a point

49
Geometric meaning
p
p
X
sj
rk
p
50
The trilinear tensor

• The tensor has 3?3?3 27 entries
• For each p, p, p we get 4 equations (?,? 1,2)

51
The trilinear tensor

• Therefore, each correspondence p, p, p
  gives 4 independent equations
• Thus, 7 correspondences linearly recover the
  tensor components!

52
Reprojection using the tensor

• If p, p and the tensor are known easy
  (linear equations for p)

53
Overall algorithm Novel view synthesis by
cascading trilinear tensors, S. Avidan and A.
Shashua, 1998

• Input two or three images is dense
  correspondence
• Construct seed tensor
• From images lt1, 2, 2gt (if two images are given)
• From images lt1, 2, 3gt (if three images are given)
• Accept from the user the novel view parameters
  (R, t)
• Generate the new tensor lt1, 2, new_view gt
• Use the tensor for reprojection and generate the
  novel image

54
Tensor embedding of the fundamental matrix
Tensor embedding of the fundamental matrix, S.
Avidan and A. Shashua

• If we only have two images in correspondence, we
  can find the tensor lt1, 2, 2gt

55
Tensor embedding of the fundamental matrix

• In fact, this is embedding of the fundamental
  matrix in tensor form

Two-view tensor
Fundamental matrix
The cross-product tensor
56
Tensor embedding of the fundamental matrix

• Proof

Intersection of 2 lines through p
57
Cascading tensors

• We have the tensor for lt1, 2, 3gt or lt1, 2, 2gt
• Want to rotate the third camera by R and
  translate by t
• How to generate the new tensor lt1, 2, ?gt?

58
Cascading tensors

• Cameras I0, Av, Bv, Cv
• Let D be a matrix such that C DB
• The tensor lt1, 2, ?gt

59
Cascading tensors

• So, we need to know A and we then have the
  operator
• Calibration assumptions on cameras 2 and ? (new)
• Principal point in the middle of the image
• Focal length image width
• So, we only need rotational component

60
Reconstruction of rotation

• Assuming small rotation angle between cameras 1
  and 2, the rotational component of the second
  camera can be approximated as follows Rousso et
  al. 96

61
Advantages of the tensor technique

• Robustness
• No degenerate camera motions (three collinear
  camera centers are OK)
• Can handle points on the trifocal plane
• Less estimation problems tensors have at most
  critical curves, or finite set of critical points
  (as opposed to critical surfaces)
• Better user interface
• R, t specification instead of point
  correspondences

62
Conclusion

• Two-views epipolar geometry
• Three views cannot be always fully described by
  triplets of fundamental matrices
• Three-views trilinear tensor
• Trilinear tensor solves singularity and
  robustness problems of epipolar geometry in
  context of novel view generation

63
Thank you