Temporal Factorization Vs. Spatial Factorization

1
Temporal Factorization Vs. Spatial Factorization

• Lihi Zelnik-Manor and M. Irani
• Presented by Thommen Korah

2
Factorization approaches to segmentation

• Traditional Spatial Factorization
• Spatial segmentation by grouping points moving
  with consistent motion
• Sparse segmentation due to difficulty of reliably
  tracking many points

3
Temporal Factorization

• Temporal grouping of frames that capture
  consistent shapes
• Can give a dense segmentation
• Simple modification of the spatial case

4
Shot-boundary detection

• Image appearance-based approaches Zhang93
• Motion based approaches Rui00

5
Temporal Factorization

• Segmentation and clustering of frames based on
  non-rigid changes in shape

Smiling
Serious
6
Correspondence Matrix W
Single frame
Single feature

• N point features tracked over F frames
• W is the 2F x N correspondence matrix

7
Spatial Factorization

• W can be factorized into motion and shape
  matrices
• W MS

Columns of W are grouped
Motion of K objects
Shape of K objects
8
Key insight

• The columns of M span the columns of W
• The rows of S span the rows of W
• Clustering columns of W into independent linear
  subspaces will group together points with same
  motion (spatial factorization)
• Clustering rows of W will group frames with the
  same shape (temporal factorization)

9
Sorting rows of W

• Resulting motion matrix M is block-diagonal
• We need a method to permute and group rows of W
  to obtain a block-diagonal structure for the
  motion matrix M

10
Neat trick

• Apply factorization to WT instead of W
• The matrix on the right side is block-diagonal
• Similar to the spatial factorization case
• So use any of the previous algorithms for
  segmenting/clustering columns of WT

11
Independent shapes

• Rows of two shape matrices are different if some
  of the columns in those matrices are different
• Non-rigid shape changes cause some points to move
  differently than others
• A different shape matrix will result causing
  those frames to be assigned to separate temporal
  clusters

12
Invariant to rigid motion
13
Results
Feature points tracked using KLT-tracker
14
Comparing Temporal and Spatial Factorization

• Data dimensionality
• Clustering of W is time consuming as it requires
  finding the eigenvalues of an N x N matrix (N is
  number of pixels) WTW
• Factorization of WT performs this operation on
  WWT, a 2F x 2F matrix
• Sparse tracked points
• Sparse spatial factorization
• Dense temporal factorization

15
Comparing Temporal and Spatial Factorization
16
Temporal vs Spatial clustering
17
Comments/project ideas

• Scalability to more modes ?
• Important to have tracked features that capture
  the non-rigid shape change
• Where is the change occurring?

18
Conclusions

• Explored properties of temporal factorization and
  duality to spatial factorization of W
• Temporal clustering and segmentation according to
  non-rigid changes in shape
• May lead to a combined approach for simultaneous
  spatio-temporal factorization