September 2003

1

Constructing 3D Models for Urban Environments
Avideh Zakhor
Video and Image Processing Lab University of
California, Berkeley
September 2003
2
Participants

• Avideh Zakhor (PI)
• Christian Frueh (Post-Doc)
• Siddarth Jain (Grad. Student)
• Ali Lakhia (Grad. Student)
• Rusty Sammon (Research Staff)

3
Overview

• Introduction
• Airborne Modeling
• Ground-Based Modeling
• Model Fusion
• Conclusion and Future Work

4
Goal
Construct 3D city model usable for both virtual
walk- and fly-throughs

• Fast scalable
• Automated
• Photorealistic

5
Approach
Combine airborne and ground-based laser scans and
camera images
Airborne Modeling
Ground-Based Modeling

• rooftops terrain

• building facades

6
Airborne Model
Airborne DSM
DSM Triangulation
DSM Post-processing
Multiple Aerial Images
Image Registration (manual or automated)
Image Selection
Texture Mapping
City Model
7
Airborne Laser Scans
Re-sampling to regular grid
Scanning city from plane
unstructured point cloud
Problem
Digital Surface Map (DSM)
Jittery edges, bumpy roofs
8
Model Generation from DSM

• Segmentation based on depth discontinuity
• Planar subdivision of segments
• Removing small segments (e.g ventilation ducts)
• Fill by extrapolating from nearby planar segments
• Find polygonal approximation for segment
  perimeter based on RANSAC
• Triangulate by connecting adjacent cells
• Simplify mesh (qslim)

Edge Map
Polygonalization
Surface Mesh
9
Airborne Modeling
Früh and Zakhor, CVPR 2003
Digital Surface Model (DSM) from airborne laser
scans
Automated Processing
Triangulation
Surface Mesh
10
Mesh Generation from DSM (2)
Triangulation after processing
11
Texture-Mapping with Aerial Imagery

• Multiple images
• Automated Registration
• Start with approximate pose from GPS and INS and
  query pose space
• Use pose for which image lines and projected 3D
  lines match best

12
Registration of Aerial Images to 3D Molels
Idea Match 2D lines with 3D lines
3D lines from DSM
2D lines from aerial image
13
Searching for True Pose

• For a given pose
• Project 3D lines into image
• Compare 2D lines to projected 3D lines via
  quality function Qpose

(direction) (location)

• li - images lines
• Lpose,j - projected DSM lines for pose
• prox(li, Lpose,j) - proximity of line li to line
  Lpose,j

• 7-Dimensional Search Space
• roll, pitch, yaw, x, y, z, focal length
• coarse initial pose from GPS/INS

• Search to find highest-rated pose
• 42 million poses

14
Search Results
Good Match
For comparison Bad Match
x 1533.3 m y 3968.1 m z 262.3 m focal length 18.2 mm yaw 35.57 deg pitch 54.71 deg roll 0.30 deg rating 1604
x 1533.3 m y 3968.1 m z 262.3 m focal length 18.2 mm yaw 35.57 deg pitch 53.71 deg roll 0.30 deg rating 767
15
Texture-Mapping with Aerial Imagery (2)
Multiple aerial images
Problem
Which one to select for a particular mesh
triangle?
Automatic image selection

• For each triangle, rate images based on
• Visibility (in field of view, not occluded)
• Normal vectors (direct views better than oblique)
• Resolution (the more pixels, the better)
• Neighbor triangles (try to texture-map adjacent
  triangles with same images)

Choose Image with highest score
16
Texture-Mapping with Aerial Imagery (3)
Images selected for individual areas
Different colors different images for texture
mapping
17
Texture-Mapping with Aerial Imagery (4)

• Large image areas unused
• low utilization of texture memory
• Many images
• limits model size during rendering

Problems
blue image area used for texture mapping
18
Texture Packing
Compressing texture by copying used texture
patches into atlas image
12 images, 225 MB texture
1 image, 72 MB texture
19
Airborne Modeling Texture-Mapping
20
Airborne Model from Last Year
View from back side
21
Airborne Modeling
3D city models created from airborne view
look good for fly-throughs
but not for walk-throughs!

• Low resolution
• Oblique views

22
Ground-Based Modeling
Drive-by Scanning
Continuous data acquisition from ground level
while driving

Data acquisition system

• Pickup truck, driving under normal traffic
  conditions
• 2 fast 2D laser scanners
• Synchronized digital camera

Früh and Zakhor, MFI 2001
23
Drive-by Scanning
buildings
vehicle
z
y
x

• Vertical 2D laser scanner to acquire geometry

24
Drive-by Scanning
buildings
vehicle
z
y
x

• Vertical 2D laser scanner to acquire geometry
• Synchronised camera to acquire texture

25
Drive-by Scanning

?
?
?
z
?
y
x

• Vertical 2D laser scanner to acquire geometry
• Synchronized camera to acquire texture

Problem Localization?
26
Drive-by Scanning

z
y
x

• Vertical 2D laser scanner to acquire geometry
• Synchronized camera to acquire texture

Horizontal 2D scanner
Problem Localization?
27
Relative Pose from Scan Matching
Horizontal laser scans

• Continuously captured during vehicle motion

• Overlap

v
Relative pose estimation by scan-to-scan matching
u
t t0
??
t t1
(?u, ?v)
Translation (?u,?v) Rotation ??
Scan matching
28
Initial Path Reconstruction
??i
Concatenate to form initial path
(?u1, ?v1, ??1)
(?ui, ?vi)
(?u2, ?v2, ??2)

(?ui, ?vi, ??i)
Translations (?ui,?vi) Rotations ??i
path
Locally accurate !
1..3 cm
vertical scanning direction
horizontal scan points
10 m
29
Initial Path Reconstruction (2)
2 Data Acquisitions
Driving time 78 minutes
Length 24.3 km
Scans 665,000
Scan points 85 million
Camera images 19,200
Needed Global correction
200 m
30
Global Correction with Monte-Carlo Localization
Frueh and Zakhor, CVPR 2001, CVPR 2003
p(xi1)

• Global registration/localization
• Probabilistic approach
• Implementation Particle filtering
• Match ground-based laser scans with edge map from
  DSM
• Use images for roll angle

z
yaw
pitch
x,y
6 Degrees of freedom (DOF) pose of vehicle
31
Monte-Carlo-Localization movie
32
Monte-Carlo-Localization
Registration of ground-based data with airborne
data
33
Path After MCL Correction
2 Data Acquisitions
Driving time 78 minutes
Length 24.3 km
Scans 665,000
Scan points 85 million
Camera images 19,200
200 m
34
Reconstructed Path
2 Data Acquisitions
Driving time 78 minutes
Length 24.3 km
Scan points 85 million
Camera images 19,200
500 m
500 m
before MCL correction
after MCL correction
35
Facade Model Reconstruction
Vehicle pose known
Transform vertical 2D laser scans into global 3D
Coordinates
Point cloud
36
Automated Facade Reconstruction
Früh and Zakhor, 3DPVT 2002
1. Point cloud from vertical scans
2. Triangulate raw mesh
3. Remove foreground and fill holes
4. Texture map with camera images
37
Mesh Hole Filling movie
Animation created by John Flynn
38
Foreground Removal movie
Animation created by John Flynn
39
Texture Mapping movie
Animation created by John Flynn
40
Texture Mapping Occluded Background Facades

• Texture of foreground objects should NOT be
  mapped to background triangles
• Need foreground segmentation algorithms
• Some background triangles are occluded in all
  captured imagery
• Need texture completion algroithms

41
Texture Mapping Occluded Background Facades
Steps
1. Identify and remove foreground in images -
Occlusion handling

• 2. Mosaic pieces of several images to texture
  atlas
• texture memory reduction
• LOD generation

3. Synthesize texture for blank areas in atlas -
model appears complete despite missing data
42
Identifying Foreground in Images
1. Project foreground vertices/triangles into
images

• Problems
• Laser resolution ltlt Image resolution
• Not all parts of foreground captured (occlusions,
  size)

43
Identifying Foreground in Images (2)
2. Region Growing around foreground pixels

• flood filling
• color constancy

3. Optical Flow foreground identification

• foreground moves more in images
• find motion via correlation

44
Foreground Segmentation Examples
45
Foreground Segmentation Examples
46
The Copy-Paste Method for Hole Filling

• Search image for areas similar to the hole
  boundaries

search for bestmatch
bestmatch

• Fill holes by copying missing pixels from similar
  areas

copy and paste
47
Search for Match Using Pyramids
perform exhaustive search at coarsest scale to
find few good matches
25
refine the few matches at higher levels to get
the best match
50
100
48
Hole Filling Examples
before hole filling
after hole filling
49
Ground-Based Modeling - Results
12 block facade model of downtown Berkeley
50
Ground-Based Modeling - Results

• Acquisition time 25 min
• Processing time 4 hours 45 min
• Fully automated!

12 block facade model of downtown Berkeley
51
Model Fusion
Combining Ground-Based and Airborne Model
Facade model
Airborne model
52
Registration of Models
Ground-based facades are already registered
because of MCL
Priority High-res ground-based model!
53
Model Fusion
1. Mark ground-based areas in DSM
54
Model Fusion
2. Remove triangles in airborne model where
ground based geometry is available
55
Model Fusion
3. Insert ground-based facades into airborne model
56
Model Fusion
4. Connect airborne and ground-based model
(blend mesh)
57
Model Fusion
5. Texture map blend mesh upper facade area
with aerial imagery
58
Fused Model Walk-through View
Downtown Berkeley University Avenue
59
Fused Model Fly-through View
Download at http//www-video.eecs.berkeley.edu/fr
ueh/3d/
60
Fused Model Movie
Download at http//www-video.eecs.berkeley.edu/fr
ueh/3d/
61
Future Work

• Speed up fully automated registration of aerial
  imagery
• Re-constitute foreground objects, e.g. trees and
  cars
• Compact representation
• Interactive high resolution rendering of
  large-scale areas (gt 1 km2)
• Indoor modeling of buildings
• Feature extraction from models