Floor Field Models for Tracking in High Density Crowds

1
Floor Field Models for Tracking in High Density
Crowds

• Saad Ali and Mubarak Shah
• Computer Vision Lab
• University of Central Florida
• ECCV 2008

2
Tracking
Single Point
Multiple Points
Bounding Box
Contour
3
Traditional Approaches to Tracking

• Features
• Appearance
• Color, texture
• Motion
• Shape
• Motion Correspondence
• Multipoint Correspondences
• Parametric transformation of a patch
• Contour Evolution

4
Object Tracking A Survey
A. Yilmaz, O. Javed, and M. Shah, Object
Tracking A Survey, ACM Journal of Computing
Surveys, Vol. 38, No. 4, 2006.
5
Crowded Scenes
6
Homography Constraint
Occluded Feet detected!
H1to2
Plane parallax error
View 1
View 2
Feet Segmented
7
Tracking Using Multiple Cameras
8
Related Work - Detection in (of) Crowds
9
High Density Crowded Scenes
High Density Moving Objects
Political Rallies
Religious Festivals
Marathons
10
Related Work

• Only works in moderate density scenarios.
• Will fail in high density scenes where it is
  difficult to establish ownership of features.
• Approaches are object centric and do not exploit
  scene and contextual knowledge.

11
Tracking in High Density Crowds
12
Challenges

• Tracking in high density scene is challenging
• Number of pixels on the object decreases with the
  increase in the density.
• Hard to discern individual from each other due to
  constant interaction among them.
• Inter-object occlusion.
• Dynamics of a crowd itself is complex
• Goal Directed
• Psychological characteristics

13
Approach

• Observation
• Behavior of an individual in a crowded
  situation is a function collective behavioral
  patterns evolving from the space-time interaction
  of a large number of individuals among
  themselves, and with the structure of the scene

14
Approach

• This collective behavioral Scene Structure can
  be channeled in as an auxiliary source of
  information.
• Constrains likely locations and paths.
• Scene Structure Force Model
• in terms of Floor fields
• Inspired by evacuation dynamics

15
Overview

• Treat the crowd flow as a collection of mutually
  interacting particles.
• Reasonable assumption, because when people are
  densely packed, individual movement is restricted.

16
Overview

• Model instantaneous movement of the individual
  with a matrix of preferences.
• Takes into consideration multiple sources of
  information.
• Appearance
• Scene Structure

17
Overview

• Each cell encodes probability of move in a
  certain direction.
• Takes into consideration multiple sources of
  information.
• Appearance
• Scene Structure

18
Scene Structure

• Encoded in terms of Floor Fields
• Floor Fields
• Model the interaction between individuals and
  their preferred direction of movement by
  transforming the long ranged forces into local
  ones.
• E.g. Long range force compelling the individual
  to move towards the exit door can be converted
  into a local force.
• Probability of move of an individual depends on
  the strength of the floor field in his/her
  neighborhood.

19
Floor Fields
Instantaneous information about the direction of
motion in the vicinity of target
Specifies regions which are more attractive in
nature exit locations
Specifies regions in the space which are more
repulsive in nature. Walls, No-go areas
20
Algorithmic Overview
Crowd Video
Probability Computation
21
Static Floor Field (SFF)

• Specifies regions of the scene which are more
  attractive in nature
• exit locations
• Captures static properties of the scene.
• Computed only once.

22
Static Floor Field (SFF)

• Attractive Regions Sinks
• If we have the knowledge about the sinks
• For any point in the scene, we can compute
  tendency of the individual at that point to move
  towards the sink.
• Local Force Function of shortest distance to
  sink in term of some metric
• Distance Metric Number of steps to wade through
  the point flow field of the scene.

23
Static Floor Field (SFF)

• Two step procedure
• Computation of Point Flows
• Sink Seeking

24
Point Flows

• Point flow instantaneous motion at any locations
• location and velocity (Zi(Xi, Vi))
• Motion in a single frame a set of point flows
• Motion in a video point flow field

25
Point Flows

• Point flow instantaneous motion at any locations
• location and velocity (Zi(Xi, Vi))

26
Sink Seeking
Sink seeking process
Sink path
sink

• Property a particle dropped in the point flow
  field will shift to a sink

27
Shift Estimation
K1
Vik1
K
t1
n
Given a point i Zi(Xi, Vi), Compute
Zik1 Xik1 Xik Vik Vik1 mean shift on the
neighbors
28
video
29
Sinks
Sink seeking
30
Example SFF
31
Example SFF
32
Dynamic Floor Field (DFF)

• Instantaneous information about the direction of
  motion.
• Instantaneous motion constraints likely
  locations.
• Based on particle advection.

33
Dynamic Floor Field (DFF)

• Use particle advection through the instantaneous
  flow field.
• Count number of particles that pass through
  location x and y
• Number of common particles Strength of
  association between x and y

34
Example Dynamic Floor Field
35
Boundary Floor Field (BFF)

• Specifies regions in the space which are more
  repulsive in nature.
• Walls, No-go areas etc.
• Crowd flow segmentation is used to generate it.

36
Crowd Flow Segmentation (CVPR2007)
Input Video
37
Optical Flow Computation

• Two Schemes
• Block based correlation in Fourier domain
• Locate peaks in the correlation surface for
  displacement calculation.
• Adaptive local median filtering for outlier
  removal.
• Thomas Brox et al., High accuracy optical flow
  Estimation based on a theory for warping, ECCV
  2004
• Grey value constancy Gradient Constancy
  Smoothness Multi-scale

38
Optical Flow - Examples
Color Coded Optical Flows
Video
39
Optical Flow - Examples
Color Coded Optical Flows
Videos
40
Particle Flow Map
?

• Keeps track of evolution of particles
• Numerical Integration
• Two-dimensional Phase Space Two Flow Maps

41
Particle Flow Map - contd
..
t0
t1
tn
42
Particle Flow Maps
Forward X Particle Flow Map
Forward Y Particle Flow Map
Backward X Particle Flow Map
Backward Y Particle Flow Map
43
Particle Flow Maps
Forward X Particle Flow Map
Forward Y Particle Flow Map
Backward X Particle Flow Map
Backward Y Particle Flow Map
44
Spatial Gradients of Flow Maps

• Observation
• High gradient between particles of different
  dynamics in the flow maps.
• Belong to dynamically distinct crowd flows.
• Spatial gradients will emphasize locations of
  such particles.

45
Flow Maps Spatial Gradients
46
Flow Map Spatial Gradients
47
Finite Time Lyapunov Exponent Field
FTLE Field

• LCS appear as ridges in FTLE field.
• FTLE field computed using the spatial gradients
  of the flow maps.

48
Lyapunov Exponent

• Lyapunov Exponent
• Asymptotic quantity.
• Measures divergence of nearby particles.
• Measures chaoticity of the system.
• For a given dynamical system

49
Finite Time Lyapunov Exponent Field

• Analysis over a grid of particles

?
Magnitude of the separation between the particles
Seek to maximize over all possible choices of
50
Finite Time Lyapunov Exponent Field
Using the operator norm
Finite version of Cauchy-Green deformation tensor
51
Finite Time Lyapunov Exponent Field
FTLE Field

• Coherent motion within a flow segment.
• Eigen-values close to zero.
• Incoherent motion at the boundary.
• Higher Eigen-values.
• Higher values - Ridges in the FTLE field -
  Ridges point to locations of LCS.

52
FTLE Field Example 1
53
FTLE Field Example 2
54
FTLE Field Example 3
55
FTLE Field Example 3
FTLE Field Example 4
56
FTLE Field Example 2
FTLE Field Example 5
57
FTLE Field Segmentation
FTLE Field
FTLE Field
FTLE Field

• Ridges in FTLE field - Watershed lines dividing
  individual catchment basins.
• Region growing
• Watershed Segmentation Algorithm

58
Segmentation Result-1
National Geographic Documentary - Inside Mecca
Using both Forward and Backward FTLE
59
Segmentation Result-2
Segmentation Result-2
National Geographic Documentary - Inside Mecca
Video
Using both Forward and Backward FTLE
60
Segmentation Result-2
61
Boundary Floor Field
62
Example Boundary Floor Fields
63
Tracking
64
Experiment - 1

• Average chip size 14 x 22 pixels
• 492 Frames
• Selected 199 athletes for tracking
• Successfully tracked 143 athletes

65
Experiment - 1
66
Experiment -1
67
Experiment 1
68
Qualitative Results
69
Qualitative Results
70
Experiment-2

• Average chip size 13 x 16 pixels
• 333 Frames
• Selected 120 athletes for tracking
• Successfully tracked 117 athletes

71
Experiment - 2
72
Experiment -2
73
Experiment 2
74
Qualitative Results
75
Experiment-3

• Average chip size 14 x 17 pixels
• 453 Frames
• Selected 50 athletes for tracking

76
Experiment - 3
77
Experiment 3
78
Qualitative Results
79
All Tracks
Successes
80
Failures

• Reasons
• Severe occlusion
• Illumination Changes

81
Quantitative Analysis

• Ground truth generated for
• 50 athletes from seq-1.
• 20 athletes from seq-2.
• 15 athletes from seq-3.

82
Quantitative Analysis

• Ground truth generated for
• 50 athletes from seq-1.
• 20 athletes from seq-2.
• 15 athletes from seq-3.

83
Quantitative Analysis

• Ground truth generated for
• 50 athletes from seq-1.
• 20 athletes from seq-2.
• 15 athletes from seq-3.

84
Contribution of Fields

• Analysis
• SFF commits less error when motion in straight
  path, no side ways
• DFF commits more error when athletes overtake
  each other

85
Summary
Tracking in Crowded Scenes

• Used knowledge of the scene for tracking.
• Concept of floor fields.
• Demonstrated tracking on challenging scenarios.

86
Thank YOu
87
Tracking Against the Flow
88
PART III Motion and Appearance Contexts for
Re-Acquiring Targets
89
PART III Motion and Appearance Contexts for
Re-Acquiring Targets