Parallel Tracking and Mapping for Small AR Workspaces Vision Seminar

1
Parallel Tracking and Mapping for Small AR
WorkspacesVision Seminar

• 2008. 9. 4 (Thu)
• Young Ki Baik
• Computer Vision Lab.

2
References

• Parallel Tracking and Mapping for Small AR
  Workspaces
• Georg Klein, David Murray, ISMAR 2007
• Visual Odometry
• David Nister et. al. , CVPR 2004

3
Outline

• What is AR?
• Previous works
• Proposed methods
• Demo
• Conclusion

4
What is AR?

• AR Augment Reality
• ugument reality(AR) uses the real scene as
  the background, and
• makes applications with putting 3D
  objects to the background.
• Since the cost of augment reality is not as
  expensive as full 3D virtual reality(VR), AR has
  been very popular research topic.

A
Full 3D (X) Expensive? (X)
Full 3D? (O) Expensive? (O)
5
Demonstration
6
The Aim

• AR with a hand-held camera
• Visual Tracking provides registration

7
The Aim

• AR with a hand-held camera
• Visual Tracking provides registration
• Track without prior model of world

8
The Aim

• AR with a hand-held camera
• Visual Tracking provides registration
• Track without prior model of world
• Challenges
• Speed
• Accuracy
• Robustness
• Interaction with real world

9
Existing attempts SLAM

• SLAM Simultaneous Localization and Mapping
• can use many different types of sensor to
  acquire observation data used in building the map
  such as laser rangefinders, sonar sensors and
  cameras.
• Well-established in robotics (using a rich array
  of sensors)
• Demonstrated with a single hand-held camera by
  Davison at 2003 (Mono-SLAM).
• Mono-SLAM was applied to
• AR system at 2004.

10
Existing attempts Model based tracking

• Model-based tracking is
• More robust
• More accurate
• Proposed by Lepetit et. al.
• at ISMAR 2003

11
Frame by Frame SLAM

• Why?
• is SLAM fundamentally harder?

Time
One frame
Find features
Many DOF
Update camera pose and entire map
Draw graphics
12
Frame by Frame SLAM

• SLAM
• Updating entire map every frame is so
  expensive!!!
• Needs sparse map of high-quality features
• - A. Davison
• Proposed approach
• Use dense map (of low quality features)
• Dont update the map every frame Keyframes
• Split the tracking and mapping into two threads

13
Parallel Tracking and Mapping

• Proposed method
• - Split the tracking and mapping into two threads

Time
Thread 2 Mapping
Update map
One frame
Thread 1 Tracking
Find features
Simple easy
Update camera pose only
Draw graphics
14
Parallel Tracking and Mapping

• Tracking thread
• Responsible estimation of camera pose and
  rendering augmented graphics
• Must run at 30 Hz
• Make as robust and accurate as possible

• Mapping thread
• Responsible for providing the map
• Can take lots of time per key frame
• Make as rich and accurate as possible

15
Tracking thread

• Overall flow

Map
Pre-process frame
Project points
Project points
Measure points
Measure points
Update Camera Pose
Update Camera Pose
Coarse stage
Fine stage
Draw Graphics
16
Pre-process frame

• Make for pyramid levels

80x60
160x120
640x480
320x240
17
Pre-process frame

• Make for pyramid levels
• Detect Fast corners
• E. Rosten et al (ECCV 2006)

80x60
160x120
640x480
320x240
18
Project Points

• Use motion model to update camera pose
• Constant velocity model

Estimated current Pt1
Previous pos Pt
?t
Previous pos Pt-1
?t
Vt (Pt Pt-1)/?t
Pt1Pt?t(Vt)
19
Project Points

• Choose subset to measure
• 50 biggest features for coarse stage
• 1000 randomly selected for fine stage

50
1000
80x60
160x120
640x480
320x240
20
Measure Points

• Generate 8x8 matching template (warped from
  source key-framemap)
• Search a fixed radius around projected position
• Use Zero-mean SSD
• Only search at Fast corner points

21
Update caemra pose

• 6-DOF problem
• Obtain by SFM (Three-point algorithm)

?
22
Dray graphics

• What can we draw in an unknown scene?
• Assume single plane visible at start
• Run VR simulation on the plane

23
Mapping thread

• Overall flow

Stereo Initialization
Tracker
Wait for new key frame
Add new map points
Optimize map
Map maintenance
24
Stereo Initialization

• Use five-point-pose algorithm
• D. Nister et. al. 2006
• Requires a pair of frames and feature
  correspondences
• Provides initial map
• User input required
• Two clicks for two key-frames
• Smooth motion for feature correspondence

25
Wait for new key frame

• Key frames are only added if
• There is a sufficient baseline to the other key
  frame
• Tracking quality is good
• Key frame (4 level pyramid images and its
  corners)
• When a key frame is added
• The mapping thread stops whatever it is doing
• All points in the map are measured in the key
  frame
• New map points are found and added to the map

26
Add new map points

• Want as many map points as possible
• Check all maximal FAST corners in the key frame
• Check score
• Check if already in map
• Epipolar search in a neighboring key frame
• Triangulate matches and add to map
• Repeat in four image pyramid levels

27
Optimize map

• Use batch SFM method Bundle Adjustment
• Adjusts map point positions and key frame poses
• Minimize reprojection error of all points in all
  keyframes (or use only last N key frames)

28
Map maintenance

• When camera is not exploring, mapping thread has
  idle time
• Data association in bundle adjustment is
  reversible
• Re-attempt outlier measurements
• Try measure new map features in all old key
  frames

29
Comparison to EKF-SLAM

• More Accurate
• More robust
• Faster tracking

lt
SLAM based AR
Proposed AR
30
System and Results

• Environment
• Desktop PC (Intel Core 2 Duo 2.66 GHz)
• OS Linux
• Language C
• Tracking speed

Total 19.2 ms
Key frame preparation 2.2 ms
Feature Projection 3.5 ms
Patch search 9.8 ms
Iterative pose update 3.7 ms
31
System and Results

• Mapping scalability and speed
• Practical limit
• 150 key frames
• 6000 points
• Bundle adjustment timing

Key frames 2-49 50-99 100-149
Local Bundle Adjustment 170 ms 270 ms 440 ms
Global Bundle Adjustment 380 ms 1.7 s 6.9 s
32
Demonstration
33
Remaining problem

• Outlier management
• Still brittle in some scenario
• Repeated texture
• Passive stereo initialization
• Occlusion problem
• Relocation problem

34
Conclusion

• Conclusion
• Parallel tracking and mapping process are
  presented using multi-thread.
• Contribution
• Visual odometry system was well presented.
• Overcome computation by multi-thread
• Opinion
• The proposed algorithm can be applied to our
  research
• Navigation system
• 3D tracking system

35
Q A