Title: Motion Detection in UAV Videos by Cooperative Optical Flow and Parametric Analysis Masaharu Kobashi
1Motion Detection in UAV Videos by Cooperative
Optical Flow and Parametric AnalysisMasahar
u Kobashi
2Objectives
- Show contrast between simple research videos and
UAV videos. - Describe strengths and weaknesses of well-know
conventional techniques. - Propose a new system designed for robustness.
3Overview of Presentation
- Overview of motion analysis
- UAV videos and their characteristics
- Strengths and weaknesses of conventional
techniques - Design of our new system
- Performance of our system
4What is motion analysis?
- Motion is a relative concept
- Relative to image plane (static camera)
- Relative to objects in the image
- Factors that make it difficult
- Changing extrinsic and intrinsic camera
parameters (Pan, tilt, translation, rotation, and
zooming) - Illumination changes
- Overlap of multiple motions with
semi-transparency (dust, fog, mist, glasses, etc.)
5Classes of Motion Analysis Methods
- Detecting flow of regularly-shaped regions
- Optical flow, Patch/Block flow
- Detecting flow of irregularly-shaped regions
- Segmentation-based flow analysis
- Variable-window/block-based flow analysis
- Detecting difference between two frames in terms
of an objective function - Parametric motion analysis
6Popular Videos for Motion Analysis
7What are UAV videos?
8Characteristics of UAV videos
- Noisier and of lower quality than standard videos
- Low bandwidth of UAV transmitter
- Quality of camera
- Constant changes of camera parameters (airplane
motion, camera's pan, tilt, zooming) - Often abrupt and quick changes
- Unrestricted natural scenes
- Including dimly lit scenes, deserts, fields
9Contrast betweenResearch videos and UAV
videos(See following demo movies.)
10Strengths and Weaknesses ofTechniques Related
to Motion Analysis
- Gradient-based Search
- Pyramid-Based Coarse-to-Fine Approach
- Optical-Flow-Based Motion Detection
- Parametric Motion Estimation
- Hybrid Approaches
11Gradient-Based Search
- Hill-Climbing for global optimization based on
local gradient information - Examples
- Lucas-Kanade registration algorithm
- Light constancy equation
- Strength
- Fast (avoiding exhaustive search)
- Weakness
- Brittle (1-pixel support a pixel noise can
derail it)
12Pyramid-Based Coarse-to-Fine Approach
- Strengths
- Can summarize information compactly
- Remedy for range-limited gradient-based methods
13Weaknesses of Pyramid
- Fixed coverage of child cells (Fixed
partitioning) - Same information to all child cells
- Vital information can be lost in summarizing
process - All parts of image required to use same number of
levels
14Results from Pyramid's Weaknesses
- The more levels, the more errors.
- Small motion areas suffer unnecessary errors due
to the height of the pyramid needed for areas of
greater motion. - Motion of thin/small parts tends to be detected
incorrectly.
15Optical-Flow-Based Motion Analysis
- Steps
- Detection of flow for all pixels in each frame.
- Accurate segmentation of the flow into moving
and stationary objects.
16Strengths and Weaknesses of Optical-flow-Based
Motion Analysis
- Strength
- Dense motion information (for every pixel)
available. - Weaknesses
- Difficult to obtain accurate vectors for all
pixels. - Segmenting flow into multiple moving and static
objects is difficult.
17Difficulties in Finding Flow Discontinuity
18Parametric Motion Estimation
- Steps
- Assume a model of transformation (Affine,
Perspective). - Compute transformation matrix of the static
regions from the point correspondences of static
regions. - Find point correspondences between two frames f1
and f2. - Find difference between frame f1 and an image
created by warping frame f2 by the
transformation matrix.
19Strengths and Weaknesses of Parametric Motion
Estimation
- Strengths
- Capable of precise matching between frames at
all points even at sub-pixel precision. - Unlike optical flow analysis, it can find
different motion layers equally well in any
transformation. - Weaknesses
- Needs reasonably accurate point correspondences.
- Must know which points belong to static region.
20Hybrid Approach
- Use of multiple motion detection techniques in a
single system (e.g. optical flow detection
parametric motion estimation) - Examples
- Wang and Adelson's motion layers
- Our system
21 Overview of Our System
- Design philosophy
- Hybrid system based upon optical flow detector
and parametric motion estimator - Takes advantage of strengths of both and avoid
their weaknesses - Priority on robustness rather than efficiency
- System structure
22 Module 1 Optical Flow Detector
- Block matching with relatively large block
(16x16) - Use of Disk Balance Ratios (DBR)
- Use of Sum of Squared Error (SSD)
23 Disk Balance Ratios (DBR)
- Objective
- To describe the orientation of a region in terms
of intensity. - Definition
24 Limitations of SSD
25 Use of Sum of Squared Difference (SSD)
- Steps to find the flow vector of block i
- If the SSD distribution is too flat among
candidates, make no decision. (Near homogeneous
region.) - If there are two or more candidates which have
distinctively better SSD than the rest and flow
vectors from block i to these candidates are far
apart, make no decision. (Aperture problem.) - Otherwise, find the candidate block having the
least SSD. The vector from block i to that
candidate is the flow vector of block i.
26 Module 2 Extraction of Reliable Flow
- Reliable flow is defined by the following
reliability measure.
27Module 3Finding Connected Reliable Flow Groups
- A variant of the single-link clustering
algorithm. - Connect adjacent reliable blocks if their flow
vectors are similar in both angle and magnitude.
28Module 4Finding Ground-Covering Flow Groups
- Steps
- If there is a group whose area is more than 40
of the screen, the group is regarded to cover the
ground only. - If available, use the grouping information of the
previous frame in the following way - For a group g, if more than 70 of the area of
g is covered by the ground region in its
predecessor frame, then g belongs to the ground.
Otherwise, make the decision by the next step. - Select the set of groups that minimizes the SSD
between the intensities of corresponding pixels
between the template frame and the image created
by warping the next frame by the inverse of the
transformation matrix.
29Computation of Transformation Matrix
30(No Transcript)
31Singular Value Decomposition (SVD)
- Merit
- Capable of handling even singular matrix
32Module 5Evaluation of Difference Data
- Steps
- Compute the mean difference m of all pixels in
the reliable ground blocks. - Exclude the reliable ground blocks from the
search region for the moving objects. - In the remaining blocks find the pixels whose
difference value is greater than m1.5s. - Remove isolated small regions from the list of
candidate regions. - Mark all blocks which contains a candidate
region as candidate blocks. - Form connected components of candidate blocks.
Each connected set of blocks is a moving object. - For each connected set of blocks, output its
bounding box to indicate the location of the
moving object.
33Output
34Performance Comparison
- Lucas-Kanade plus pyramid v.s. Our system
- Michael Black's Algorithm v.s. Our system
- Simple research videos v.s. UAV videos
35Lucas-KanadePyramid
36Lucas-Kanade Pyramid
37Black's Algorithm
38Black's Algorithm