ESTIMATION%20ALGORITHMS%20FOR%20AMBIGUOUS%20VISUAL%20MODELS%20Reconstructing%203D%20Human%20Motion%20from%20Monocular%20Video - PowerPoint PPT Presentation

View by Category
About This Presentation
Title:

ESTIMATION%20ALGORITHMS%20FOR%20AMBIGUOUS%20VISUAL%20MODELS%20Reconstructing%203D%20Human%20Motion%20from%20Monocular%20Video

Description:

Methods for locating and tracking multiple minima. Covariance ... to Locate. Multiple Minima? ... transition state location. KJS. deterministic inter ... – PowerPoint PPT presentation

Number of Views:76
Avg rating:3.0/5.0

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: ESTIMATION%20ALGORITHMS%20FOR%20AMBIGUOUS%20VISUAL%20MODELS%20Reconstructing%203D%20Human%20Motion%20from%20Monocular%20Video


1
ESTIMATION ALGORITHMS FOR AMBIGUOUS VISUAL
MODELSReconstructing 3D Human Motion from
Monocular Video
  • Cristian Sminchisescu

2
Goal
  • Track human body motion in monocular video
  • Estimate 3D articular motion
  • Why Monocular ?
  • Video footage processing (movies, etc.)
  • Action / gesture tracking and interpretation
    (HCI)
  • Cognitive viewpoint humans can do this robustly
    !


3
Approach to Visual Modeling
  • Generative Human Model
  • Complex, kinematics, geometry, photometry
  • Predicts images or descriptors
  • Model-image matching cost function
  • Associates model predictions to image features
  • Robust, often probabilistically motivated
  • Search/Optimization
  • Discovers well supported configurations on
    resulting cost function

4
Presentation Plan
  • Introduction
  • Goals, applications, general formulation
  • Modeling (human model and image features)
  • Parameterization, constraints and priors
  • Error models, image descriptors
  • State of the art and parameter estimation
    difficulties
  • Methods for locating and tracking multiple minima
  • Covariance Scaled Sampling
  • Hyperdynamic Sampling
  • Eigenvector Tracking Hypersurface Sweeping
  • Kinematic Jump Sampling

5
Human Body Model
  • Explicit 3D model allows high-level
    interpretation
  • 30-35 d.o.f. articular skeleton
  • Flesh of superquadric ellipsoids with tapering
    bending
  • Model ? image projection maps points on skin
    through
  • kinematic chain
  • camera matrix
  • occlusion (z buffer)

6
Parameter Space Priors
  • Anthropometric prior
  • left/right symmetry
  • bias towards default human
  • Accurate kinematic model
  • clavicle (shoulder), torso (twist)
  • robust prior stabilizes complex joints
  • Body part interpenetration
  • repulsive inter-part potentials
  • Anatomical joint limits
  • hard bounds in parameter space

7
Image Features, Integrated Robustly
  • 1. Intensity
  • The model is dressed with the image texture
    under its projection (visible parts) in the
    previous time step
  • Matching cost of model-projected texture against
    current image (robust intensity difference)

8
2.Contours
  • Multiple probabilistic assignment integrates
    matching uncertainty
  • Weighted towards motion discontinuities (robust
    flow outliers)
  • Also accounts for higher order symmmetric
    model/data couplings
  • partially removes local, independent matching
    ambiguities

9
3.Silhouettes
Use attraction-explanation pair
  • Attract the model inside image silhouette
  • use distance level functions DT
  • To avoid inconsistency, demand the model explains
    the image
  • maximize model/image silhouette overlap

10
Tracking and Optimization
  • Formulate tracking as an optimization problem
  • Very nonlinear
  • kinematics, image projection, occlusion ...
  • High dimensional search space (35 d.o.f.)
  • Most depth information is lost
  • about 1/3 of d.o.f. are statically unobservable
  • Data association is hard
  • non-rigidity, modeling imperfections
  • clothing, shadows, occlusion confuse the signal
  • General unknown motions
  • Many unexpected variables, humans switch
    activities, avoid obstacles

Difficult to optimize
Local minima
Hard to trace temporally
11
Why is monocular pose hard?
Depth ambiguities
Image matching ambiguities
Violations of physical constraints
12
Tracking/Sampling Methods
  • Local (Kalman filter, local optimization)
  • Multiple cameras to regularize the cost surface
  • Rehg95, Gavrila96 (discrete search),
    Kakadiaris96, Bregler98, Wachter99 (1 camera),
    Delamarre99, Plankers01
  • Quasi-Global
  • CONDENSATION/particle fliters (unscented,
    auxiliary,etc)
  • 3D annealing, temporal models (3 cameras)
    (Deutscher00), importance sampling, strong
    prior model (Sidenbladh00,02)
  • MCMC
  • Annealing - Neal97,01
  • Tampering (Marinari92, Neal94)
  • Hybrid - Duane87, Forsyth00, Choo01
  • ADS Gilks94, MTM Liu 00
  • Computational chemistry
  • Deterministic trajectories, transition states -
    Jensen95
  • Stochastic (infrequent event acceleration) -
    Voter97, Sorensen00
  • Tunneling (global optimization, Torn94)
  • Local optimization repulsive potentials

13
How to Sample to Locate Multiple Minima?
  • Multi-modal high-dimensional distributions need
    complex sampling methods
  • Sometimes only a subset of modes is available
  • Trapping during model initialization
  • Mode splitting during tracking
  • Need to widen the sampling region, but focus the
    search

14
Condensation Temporal Density Propagation
(from Isard and Blake, 1996)
15
Condensation
  • Advantages
  • Robust, effective management of uncertainty
  • Propagates arbitrary multimodal distributions
  • Problems
  • Trapping in suboptimal local minima
  • Sample wastage in high cost regions
  • High dimensionality and/or ill conditioning make
    these much worse!

16
Saddle/Minima Statistics
  • Inter-minimum transition states (saddles) are
    far in parameter space (7 stdev)

17
Trapping in Local Minima
  • Adjacent likelihood peaks are typically
    separated by many standard deviations
  • e.g. reflective ambiguities, false model-image
    correspondences
  • The state density is too localized to be a good
    search hypothesis generator
  • Samples almost never reach nearby peaks
  • Devote some of the samples to more global search

18
Why Boosting Dynamics Wastes Samples
  • For ill-conditioned / high dimensional problems
    any nearly uniform noise causes either
  • insufficient scatter if it is small
  • massive sample wastage if it is large

19
Covariance Scaled Sampling
  • Avoids large volume wastage factors
  • Condensation vs. CSS volume factor 1054

20
Sampling is not enough
  • In high dimensions volume increases very rapidly
    with radius
  • High cost samples are not resampled, so

To find minima efficiently, you also need to
optimize locally after sampling
21
Covariance Scaled Sampling Algorithm
  • A density propagation method combining
  • mode covariance mixture representation
  • wide tailed sampling to reduce trapping
  • covariance scaling to reduce volume wastage
  • local optimization (robust, constraint
    consistent) to reduce needle-in-haystack effect

22
(No Transcript)
23
Clutter Video
Original sequence
Tracked sequence (model overlayed)
24
(Quasi-)Globality and Adaption
  • The CSS sampling criteria is yet fairly local
  • Can sample widely but adaption is only based on
    estimated covariances at the minima
  • The energy landscape changes further away
  • How wide to spread ?
  • Are there longer-range forms of adaption?
  • Can we be more precise on where we want to
    sample?

25
The Dividing Surface (DS)
  • For a minimum M1
  • n-1 dimensional surface separating M1 from its
    neighbors
  • Essentially non-locally defined
  • Steepest descent trajectories converge to other
    minima than M1
  • May include positive curvature regions, various
    saddles

26
The Dividing Surface (contd.)
  • Sampling involves
  • Long periods of vibration (exploration) in one
    minimum
  • Infrequent transitions to other minima through
    low DS regions
  • The flux through DS defines the inter-minima
    transition rates
  • But the low DS regions of high-flux are the
    transition states !

27
Importance of Transition Neighborhoods
  • Transition States (TS) are
  • Co-dimension 1 saddle points
  • Zero gradient, 1 negative, (n-1) positive
    curvatures
  • Cols rather than mountain tops
  • Low cost saddles (TS) lead to low cost minima
  • Provide useful local approximation to the DS
  • f cost function, g gradient, (e1, V1)
    smallest Hessian (eigenvalue, eigenvector)

28
Sampling a modified potential P(h,d)
small h
large h
small d large d
29
The Adaptive Bias Potential
  • Add a bias fb(x) to the cost
  • h is height of bias well
  • d is a length scale ( ? distance to nearest
    minimum)
  • e1 is smallest Hessian eigenvalue
  • g1 is cost gradient in first eigendirection
  • fb(x)0 at a saddle (e1 lt 0, g1 0)

30
Hyperdynamics
  • A generalized adaptive search broadening method
  • Focuses samples near transition states by
    adaptively raising the cost
  • in the cores of local minima
  • in high gradient regions
  • Exponentially increased probability of reaching a
    transition state

31
Pure MCMC vs. Hyperdynamics
  • Hyperdynamics escapes trapping and explores
    multiple minima (8000 simulation steps)

32
How can we find nearby minima deterministically ?
Local minima
Transition states
  • From any Transition state (saddle point), we can
    usually slide downhill to an adjacent minimum
  • Low cost saddles lead to low cost minima

33
Local Minimization versus Saddle Point Search
  • Minimization
  • many local descent based algorithms
  • measure progress by decrease in function values
  • local sufficient decrease ensures global
    convergence to some local minimum
  • Local Saddle Point Search
  • ascend, using modified descent algorithms
  • no universal progress criterion
  • initialization needed (e.g. ascent direction)

34
Newton Minimization
  • Pure Newton iteration
  • efficient near minimum, but globally unreliable
  • may diverge, converge to any type of stationary
    point, etc
  • Damped Newton iteration
  • globalize convergence by adding damping matrix
    D
  • in an eigenbasis where D I

35
Eigenvector Tracking
  • Choose an ascent eigenvector k and track it as
    you progress
  • Move uphill along k and downhill along all
    other eigenvectors
  • Uses modified damping signs to minimize an
    equivalent local virtual cost model
  • But tracking the same eigenvector is hard
  • when eigenvalues cross, their eigenvectors slew
    around rapidly...

36
Hypersurface Sweeping .
  • (reminder) A codimension 1 saddle is
  • a local maximum in one direction
  • a local minimum in the other n-1
  • Sweep space with a moving hypersurface
  • hyperplane, expanding hyper-ellipsoid
  • Find track a local minimum on cost surface
  • optimize over hypersurface
  • Find local (temporal) maxima of track
  • Hypersurface vs. cost isosurface relative
    curvatures control search diversity

37
Hypersurface Sweeping
  • Long trajectories started in one minimum
  • Hypersurface ? cost isorsurface, but flattened
    orthogonally w.r.t. the search direction
  • Find other minima and saddles
  • For illustration didnt stop after each saddle
    detection

38
Hypersurface Sweeping Issues
  • Only finds saddles that surface cuts in positive
    curvature directions
  • Plane / ellipse orientation fixes initial track
    direction
  • Some local track maxima are topological
    transitions, not saddles

39
Minima Caused By Incorrect Edge Assignments
  • Intensity edges

Edges only
40
Pose Ambiguities -Video
  • Monocular static pose minima for model to image
    3D-2D correspondences

41
Can we Find other Minima by Exploiting Global
Problem Structure?
  • For any given model state, we can explicitly
    build the interpretation tree of alternative
    kinematic solutions
  • Forward-backward ambiguities, one per body part

42
Efficient Inverse Kinematics
  • The inverse kinematics is simple, efficient to
    solve
  • Constrained by many observations (articulation
    centers)
  • The quasi-spherical articulation of the body
  • Mostly in closed form
  • The iterative solution is also competitive
  • 1 local optimization work per new minimum found
  • Can explore the kinematic minima by jumping
    either systematically or randomly
  • An adaptive diffusion method (CSS) is necessary
    for correspondence ambiguities

43
KJS CSS in Action
44
Overview Adaption and Globality
  • CSS
  • Adaption based on curvature at local minima
  • scaling by local covariances
  • Hyperdynamics
  • generalized, cost-based adaption
  • local gradient and curvature
  • long range bias towards transition neighborhoods
  • ET/HS
  • Deterministic transition state location
  • KJS
  • deterministic inter-minimum jumps

In practice the methods can be combined
45
Technicalities How to use the methods?
  • Mixture density propagation (finding and tracking
    local minima)
  • CSS, KJS as is
  • ET / HS saddle detection local descent
  • Hyperdynamics MCMC sampling (Hybrid, Langevin)
    local descent
  • Sampling fairly
  • CSS importance sampling proposal with
    correction
  • Hyperdynamics as is / local descent
    generalized targeted MCMC (Sminchisescu03,
    Sminchisescu et al03 upcoming)
  • ET / HS / KJS darting MCMC

46
Conclusions and Perspectives
  • Non-convex high-dimensional problems
  • Importance of (quasi-)globality and adaption
    during search
  • Flexible learning
  • Model, likelihood, search (priors, other dynamic
    rules)

47
Thank you
About PowerShow.com