Estimation of Missing Markers in Human Motion Capture

1
Estimation of Missing Markers in Human Motion
Capture

• Guodong Liu and Leonard McMillan
• Department of Computer Science
• University of North Carolina at Chapel Hill

2
Outline

• Introduction
• Previous work
• Our approach
• Experimental results (Demo)
• Conclusions and future work

3
What Benefit from Motion Capture?

• Motion pictures and video games
• VR applications
• Physical therapies
• Many other applications

4
Background

• Optical mocap system utilizes video cameras to
  track reflective markers
• Marker position
• Estimated via triangulation from multiple
  cameras
• Missing marker
• Not visible to at least two cameras

5
Motivation

• Some marker positions are missing due to
  occlusions or ambiguities.
• Most missing marker recovering procedures require
  manual intervention and are not satisfactory with
  
• diverse motions,
• high percentage of missing markers, and/or
• extended occlusions.

6
Goals

• Provide fast and plausible estimation of a
  full-body pose based on available marker set
• Simple, fast and robust in recovering missing
  markers and estimating human motions
• Allow different set of markers to be missing for
  a moderate-to-long period of time

7
Rationales

• Properties of human motion data
• Highly coordinated among body parts
• Exhibit considerable redundancy
• Over-specified by the features
• Do not fully fill the feature space
• Exhibit local linearity by forming locally linear
  groups

8
Previous Work

• Control using low dimensional signals
• Lee et al 2002 , Chai and Hodgins 2005 , etc.
• Motion synthesis from low dimensional space
• Safonova et al. 2004, Grochow et al. 2004
• Motion segmentation
• Rose et al. 1998, Barbic et al. 2003
• Segment-based motion modeling
• Liu et al. 2006

9
Motion Data Representation

• Each pose/frame
• yi3m-dim column vector, containing 3D positions
  (x, y and z coordinates) of m markers for the ith
  pose
• Each pose corresponds to a data point
• Motion sequence with N poses/frames
• Map it to 3m?N data matrix Yy1, y2, , yN

10
Missing Marker Estimation Process
11
Global Linear Modeling

• Retrieve a mean vector of all the poses in the
  data set
• Compute PCA to obtain an eigenvector matrix P and
  take the leading d principal components, i.e. the
  leftmost d columns of the matrix P
• Save the mean vector and the principal components

12
Piecewise Linear Modeling
13
Motion Segmentation

• Probabilistic PCA approach
• Model marker positions of poses with Gaussian
  distribution
• Segment when there is a local change to the
  distribution

14
Characterization of Motion Segments

• Retrieve a feature vector from each motion
  segment
• Compute mean and covariance matrix of segment
  poses
• Concatenate them into a vector (a very long
  vector)
• Dimensionality reduction on the mean/covariance
  vectors
• Run PCA on all the vectors
• Each vector is projected onto the space spanned
  by the leading principal components and become a
  point, i.e. feature point, in that space

15
Motion Hierarchy Construction

• Divisive clustering on feature vectors of
  segments with distance metric being Euclidean
  norm
• Splitting recursively to two children based on
  K-means method until all the clusters at those
  branches satisfy a preset distance tolerance

• Each cluster represents a local linear model with
  each segment/frame in the cluster labeled with
  the same model ID

16
Training Random Forests (RF) Classifier

• Classification of frames to local linear models
• Input data full marker positions of frames
• Each frame is labeled with a local linear model
  ID
• Purpose identify the most appropriate local
  linear model for a frame with full marker
  positions

17
Local Linear Modeling

• For each local linear model
• Retrieve and save the mean pose as well as the
  principal components, computed out of all the
  poses that belong to the model

18
Missing Marker Estimation

• Two-step, coarse-to-fine estimationof missing
  Marker positions
• Estimation with the global linear model
• Estimation using the local linear models
• Smooth out the frames at the transitions between
  local linear models

19
Estimating Missing Markers with Global Linear
Model

• Given a frame with missing markers
• Retrieve the positions of the available markers
• Retrieve the global linear model
• Mean vector
• Principal component vectors
• From the available marker positions, find the
  frames projection onto the principal space
• A least-squares solution
• Project back to the original marker space

20
Estimating Missing Markers with Local Linear
Models

• Take the results from the global linear model
  estimation
• Use the Random Forest classifier
• Identify the most appropriate local linear model
  for each frame
• Reconstruct missing markers based on the linear
  model
• Similar to the method in the previous slide

21
Estimating Poses in Transitions with a Mixture
of Models

• Strategy
• Estimation with a mixture of models Put more
  weights on the model favored by more poses prior
  to the current pose
• Let zt a vector containing the 3D positions of
  missing markers at time t
• zt Si wi Ei
• Ei Maker estimation result from the ith local
  linear model
• wi ri / (h1) a weight for the ith model
• ri of poses classified to the ith model among
  the prior h poses and current pose

22
Demo of experimental results
23
Conclusions

• A data-driven, segment-based, piecewise linear
  approach
• Exploited spatial correlations among markers
• Complement to spline-interpolation-based methods
• More effective with long missing gaps