6.088 Digital and Computational Photography 6.882 Advanced Computational Photography Panoramas

1
6.088 Digital and Computational Photography
6.882 Advanced Computational PhotographyPanor
amas
Frédo DurandMIT - EECS
Lots of slides stolen from Alyosha Efros, who
stole them from Steve Seitz and Rick Szeliski
2
Why Mosaic?

• Are you getting the whole picture?
• Compact Camera FOV 50 x 35

Slide from Brown Lowe
3
Why Mosaic?

• Are you getting the whole picture?
• Compact Camera FOV 50 x 35
• Human FOV 200 x 135

Slide from Brown Lowe
4
Why Mosaic?

• Are you getting the whole picture?
• Compact Camera FOV 50 x 35
• Human FOV 200 x 135
• Panoramic Mosaic 360 x 180

Slide from Brown Lowe
5
Mosaics stitching images together
virtual wide-angle camera
6
How to do it?

• Basic Procedure
• Take a sequence of images from the same position
• Rotate the camera about its optical center
• Compute transformation between second image and
  first
• Transform the second image to overlap with the
  first
• Blend the two together to create a mosaic
• If there are more images, repeat
• but wait, why should this work at all?
• What about the 3D geometry of the scene?
• Why arent we using it?

7
A pencil of rays contains all views
Can generate any synthetic camera view as long as
it has the same center of projection!
8
Aligning images translation

• Translations are not enough to align the images

9
Questions?
10
Image reprojection

• The mosaic has a natural interpretation in 3D
• The images are reprojected onto a common plane
• The mosaic is formed on this plane
• Mosaic is a synthetic wide-angle camera

11
Image reprojection

• Basic question
• How to relate 2 images from same camera center?
• how to map a pixel from PP1 to PP2

PP2

• Answer
• Cast a ray through each pixel in PP1
• Draw the pixel where that ray intersects PP2

PP1
But dont we need to know the geometry of the two
planes in respect to the eye?
Observation Rather than thinking of this as a 3D
reprojection, think of it as a 2D image warp
from one image to another
12
Back to Image Warping
Which t-form is the right one for warping PP1
into PP2? e.g. translation, Euclidean,
affine, projective
13
Homography

• Projective mapping between any two PPs with the
  same center of projection
• rectangle should map to arbitrary quadrilateral
• parallel lines arent
• but must preserve straight lines
• same as project, rotate, reproject
• called Homography

PP2
PP1

• To apply a homography H
• Compute p Hp (regular matrix multiply)
• Convert p from homogeneous to image coordinates

14
1D homogeneous coordinates

• Add one dimension to make life simpler
• (x, w) represent point x/w

w
w1
x
15
1D homography

• Reproject to different line

w
w1
x
16
1D homography

• Reproject to different line

w
w1
x
17
1D homography

• Reproject to different line
• Equivalent to rotating 2D points
• ? reprojection is linear in homogeneous
  coordinates

w
w1
x
18
Same in 2D

• Reprojection homography
• 3x3 matrix

PP2
PP1
19
Image warping with homographies
image plane in front
image plane below
20
Questions?
21
Digression perspective correction
From Photography, London et al.
22
From Photography, London et al.
23
From Photography, London et al.
24
From Photography, London et al.
25
Tilt-shift lens

• 35mm SLR version

26
Photoshop version (perspective crop)

• you control reflection and perspective
  independently

27
(No Transcript)
28
Back to Image rectification
p
p

• To unwarp (rectify) an image
• Find the homography H given a set of p and p
  pairs
• How many correspondences are needed?
• Tricky to write H analytically, but we can solve
  for it!
• Find such H that best transforms points p into
  p
• Use least-squares!

29
Least Squares Example

• Say we have a set of data points (X1,X1),
  (X2,X2), (X3,X3), etc. (e.g. persons height
  vs. weight)
• We want a nice compact formula (line) to predict
  Xs from Xs Xa b X
• We want to find a and b
• How many (X,X) pairs do we need?
• What if the data is noisy?

AxB
overconstrained
30
Questions?
31
Solving for homographies
p Hp

• Can set scale factor i1. So, there are 8
  unkowns.
• Set up a system of linear equations
• Ah b
• where vector of unknowns h a,b,c,d,e,f,g,hT
• Note we do not know w but we can compute it from
  x ywgxhy1
• The equations are linear in the unknown

32
Solving for homographies
p Hp

• Can set scale factor i1. So, there are 8
  unkowns.
• Set up a system of linear equations
• Ah b
• where vector of unknowns h a,b,c,d,e,f,g,hT
• Need at least 8 eqs, but the more the better
• Solve for h. If overconstrained, solve using
  least-squares
• Can be done in Matlab using \ command
• see help lmdivide

33
Questions?
34
Panoramas

1. Pick one image (red)
2. Warp the other images towards it (usually, one by
   one)
3. blend

35
Recap

• Panorama reprojection
• 3D rotation ? homography
• Homogeneous coordinates are kewl
• Use feature correspondence
• Solve least square problem
• Se of linear equations
• Warp all images to a reference one
• Use your favorite blending

36
Questions?
37
changing camera center

• Does it still work?

38
Nodal point

• http//www.reallyrightstuff.com/pano/index.html

39
Planar mosaic
40
Questions?
41
Cool applications of homographies

• Oh, Durand Dorsey

42
Limitations of 2D Clone Brushing

• Distortions due to foreshortening and surface
  orientation

43
Clone brush (Photoshop)

• Click on a reference pixel (blue)
• Then start painting somewhere else
• Copy pixel color with a translation

44
Perspective clone brush

• Oh, Durand, Dorsey, unpublished
• Correct for perspective
• And other tricks

45
(No Transcript)
46
Questions?
47
Rotational Mosaics

• Can we say something more about rotational
  mosaics?
• i.e. can we further constrain our H?

48
3D ? 2D Perspective Projection
K
49
3D Rotation Model

• Projection equations
• Project from image to 3D ray
• (x0,y0,z0) (u0-uc,v0-vc,f)
• Rotate the ray by camera motion
• (x1,y1,z1) R01 (x0,y0,z0)
• Project back into new (source) image
• (u1,v1) (fx1/z1uc,fy1/z1vc)
• Therefore
• Our homography has only 3,4 or 5 DOF, depending
  if focal length is known, same, or different.
• This makes image registration much better behaved

f
50
Pairwise alignment

• Procrustes Algorithm Golub VanLoan
• Given two sets of matching points, compute R
• pi R pi with 3D rays
• pi N(xi,yi,zi) N(ui-uc,vi-vc,f)
• A Si pi piT Si pi piT RT U S VT (U S
  UT) RT
• VT UT RT
• R V UT

51
Rotation about vertical axis

• What if our camera rotates on a tripod?
• Whats the structure of H?

52
Do we have to project onto a plane?
53
Full Panoramas

• What if you want a 360? field of view?

54
Cylindrical projection

• Map 3D point (X,Y,Z) onto cylinder

unit cylinder
55
Cylindrical Projection
Y
X
56
Inverse Cylindrical projection
57
Cylindrical panoramas

• Steps
• Reproject each image onto a cylinder
• Blend
• Output the resulting mosaic
• What are the assumptions here?

58
Cylindrical image stitching

• What if you dont know the camera rotation?
• Solve for the camera rotations
• Note that a rotation of the camera is a
  translation of the cylinder!

59
Assembling the panorama

• Stitch pairs together, blend, then crop

60
Problem Drift

• Vertical Error accumulation
• small (vertical) errors accumulate over time
• apply correction so that sum 0 (for 360 pan.)
• Horizontal Error accumulation
• can reuse first/last image to find the right
  panorama radius

61
Full-view (360) panoramas
62
Spherical projection

• Map 3D point (X,Y,Z) onto sphere

Y
Z
X

• Convert to spherical coordinates

f
unwrapped sphere
63
Spherical Projection
Y
X
64
Inverse Spherical projection

• f

(x,y,z)

• cos f

• sin f

(sin?cosf,cos?cosf,sinf)
Y
Z

• cos ? cos f

X
65
3D rotation

• Rotate image before placing on unrolled sphere

• f

(x,y,z)

• cos f

(sin?cosf,cos?cosf,sinf)

• sin f

• cos ? cos f

• _ _

p R p

• _ _

66
Full-view Panorama




67
Polar Projection

• Extreme bending in ultra-wide fields of view

68
Other projections are possible

• You can stitch on the plane and then warp the
  resulting panorama
• Whats the limitation here?
• Or, you can use these as stitching surfaces
• But there is a catch

69
Cylindrical reprojection
70
Whats your focal length, buddy?

• Focal length is (highly!) camera dependant
• Can get a rough estimate by measuring FOV
• Can use the EXIF data tag (might not give the
  right thing)
• Can use several images together and try to find f
  that would make them match
• Can use a known 3D object and its projection to
  solve for f
• Etc.
• There are other camera parameters too
• Optical center, non-square pixels, lens
  distortion, etc.

71
Distortion
No distortion
Pin cushion
Barrel

• Radial distortion of the image
• Caused by imperfect lenses
• Deviations are most noticeable for rays that pass
  through the edge of the lens

72
Radial distortion

• Correct for bending in wide field of view
  lenses

Use this instead of normal projection
73
Blending the mosaic
An example of image compositing the art (and
sometime science) of combining images together
74
Multi-band Blending
75
Multi-band Blending

• Burt Adelson 1983
• Blend frequency bands over range ? l

76
Poisson blending
77
Questions?
78
Traditional panoramas
79
(No Transcript)
80
19th century panorama
81
Chinese scroll
82
Questions?
83
Magic ghost removal
M. Uyttendaele, A. Eden, and R. Szeliski.
Eliminating ghosting and exposure artifacts in
image mosaics. In Proceedings of the
Interational Conference on Computer Vision and
Pattern Recognition, volume 2, pages 509--516,
Kauai, Hawaii, December 2001.
84
Magic ghost removal
M. Uyttendaele, A. Eden, and R. Szeliski.
Eliminating ghosting and exposure artifacts in
image mosaics. In Proceedings of the
Interational Conference on Computer Vision and
Pattern Recognition, volume 2, pages 509--516,
Kauai, Hawaii, December 2001.
85
Magic automatic panos
http//research.microsoft.com/brown/papers/iccv20
03.pdf
86
Extensions

• Video
• Additional objects
• Moks panomorph
• http//www.sarnoff.com/products_services/vision/te
  ch_papers/kumarvb.pdf
• http//www.cs.huji.ac.il/peleg/papers/pami00-mani
  fold.pdf
• http//www.cs.huji.ac.il/peleg/papers/cvpr00-rect
  ified.pdf
• http//www.cs.huji.ac.il/peleg/papers/cvpr05-dynm
  os.pdf
• http//citeseer.ist.psu.edu/cache/papers/cs/20590/
  httpzSzzSzwww.sarnoff.comzSzcareer_movezSztech_pa
  perszSzpdfzSzvisrep95.pdf/kumar95representation.pd
  f
• http//www.robots.ox.ac.uk/vgg/publications/paper
  s/schaffalitzky02.pdf

87
Software

• http//photocreations.ca/collage/circle.jpg
  http//webuser.fh-furtwangen.de/7Edersch/
  http//www.ptgui.com/ http//hugin.sourceforge.n
  et/ http//epaperpress.com/ptlens/
  http//www.panotools.info/mediawiki/index.php?tit
  leTutorials http//www.fdrtools.com/front_e.php
  

88
Refs

• http//graphics.cs.cmu.edu/courses/15-463/2004_fal
  l/www/Papers/MSR-TR-2004-92-Sep27.pdf
• http//www.cs.ubc.ca/mbrown/papers/iccv2003.pdf
• http//www.cs.washington.edu/education/courses/cse
  p576/05wi/readings/szeliskiShum97.pdf
• http//portal.acm.org/citation.cfm?id218395dlAC
  Mcollportal
• http//research.microsoft.com/brown/papers/cvpr05
  .pdf
• http//citeseer.ist.psu.edu/mann94virtual.html
• http//grail.cs.washington.edu/projects/panovidtex
  /
• http//research.microsoft.com/users/mattu/pubs/Deg
  hosting.pdf
• http//research.microsoft.com/vision/visionbasedmo
  deling/publications/Baudisch-OZCHI05.pdf
• http//www.vision.caltech.edu/lihi/Demos/SquarePan
  orama.html
• http//graphics.stanford.edu/papers/multi-cross-sl
  its/

89
View morphing
subject
common view plane
viewpoint 1
viewpoint 2