Feature Tracking with H'264 Motion Vectors for Mobile Augmented Reality

1
Feature Tracking with H.264Motion Vectors
forMobile Augmented Reality

• Vijay Chandrasekhar
• Gabriel Takacs
• Louis Chen
• EE398B Project
• 29 May 2007
• Stanford University

2
Outline

• Introduction to Mobile Augmented Reality (MAR)
• Background on Tracking
• Good Points to Track
• Point-Based Scheme
• Model-Based Scheme
• Extension to Various GOP Structures
• Results
• Conclusion

3
Introduction to MAR
4
Still-Image MAR

• Extract scale-invariant features (SIFT or SURF)
• Features are unique, identifiable points in image
• Match features with database

5
MAR System Overview

• Group pictures by GPS location
• Loxel (Location Cell)
• Kernel (Neighborhood of Loxels)

6
Motivation for Tracking Features

• Feature matches are interest points
• Inefficient to extract/match features in all
  frames
• Smooth user experience

7
Background on Tracking

• Constant Intensity Assumption
• Aperture Problem

8
Background on Tracking

• Frequency Domain
• Optical Flow
• Block Matching

9
H.264 Motion Estimation

• Block-matching
• Video encoder implemented in hardware
• Problems
• Intra-blocks do not have motion-vectors
• Rate-distortion optimized motion-vectors
• Motion-vectors point backwards in time
• Aperture problem

10
Pixel Connectivity
?
(b2)
(c)
(b)
(b1)
(a)
(a)
t
t1
11
H.264 Motion Field
12
Good Points to Track

• Corners
• Texture

Bergman, Nachlieli, Ruckenstein 2007
Original
Corners
Texture
13
Point-Based Scheme

• Track each feature independently

Extract Features
Match Features
Initialize
no
Enough Points in Frame?
yes
(xit1, yit1)
(xit, yit)
Track with Motion Vectors
Next Frame
Mavlankar, Varodayan 2007
14
Model-Based Scheme

• Assume
• Corners and Texture are trustworthy for tracking
• Features are from background
• Estimate motion-model with trusted features
• Use motion-model to track untrusted features

15
Model-Based Scheme
Extract Features
Detect Corners
Detect Texture
Initialize

no
Match Features
5
(xit1, yit1)
yes
Enough Points in Frame?
Next Frame
(xit, yit)
Model?
yes
Connected
Inter-coded block?
Inliers
Track with Motion Vectors
Form Motion Model
U
3
2
1
Intra
Model
Disconnected
Outliers
U
4
Extrapolate
16
Model Estimation
RANSAC
Texture or Corner
Choose points
Well Spaced?
no
Point pairs (xit, yit) (xit1, yit1)
Iterate K times
yes
Form model from sample points
no
Reflect? Shear?
yes
Find model with low error many inliers
Possible Model
no
17
GOP Structure Extensions

• Multiple-Reference P-Frames
• B-Frames

I
B
P
B
B
P
1 3 2 5 6 4
18
Multi-Connected Frames
Pt-1
Bt
Pt1
B-Frame
Pt-a
Pt
Pt-b
Multiple-Reference P-Frame
19
Tracking Videos

• Point-Based Slow Pan
• Model-Based Slow Pan
• Model-Based Fast Pan
• Point-Based Zoom
• Model-Based Zoom
• Point-Based Occlusion
• Model-Based Occlusion

20
Ground Truth

• SURF-match every frame w.r.t. anchor frame

21
Qualitative Results - Pan

• 85 Features

Anchor
Frame 40
Frame 40
Point-Based
Model-Based
22
Qualitative Results - Occlude

• 220 Features

Anchor
Frame 50
Frame 50
Point-Based
Model-Based
23
Qualitative Results - Zoom

• 176 Features

Anchor
Frame 60
Frame 60
Point-Based
Model-Based
24
Evaluation Metric

• Tracking Error Euclidean Distance
• between ground-truth and tracked feature
  positions
• Average error vs. time
• Maximum error vs. time

25
Perspective vs Affine
26
Quantitative Results - Pan
27
Quantitative Results - Occlude
28
Quantitative Results - Zoom
29
Information Overlay
Frame 1
Frame 100

• Use Models to Overlay Information

30
Conclusion

• Novel, low-complexity, tracking scheme
• H.264 motion vectors
• Model-based approach
• Low tracking error
• Applications to MAR
• Lower feature extraction/matching frequency
• Overlay information