Capturing and Animating Occluded Cloth

1
Capturing and Animating Occluded Cloth
Ryan White, Keenan Crane, D.A. Forsyth SIGGRAPH
2007

• Presented by Mu-Heng Li

2
(No Transcript)
3
Outline

• Introduction
• Previous work
• Acquisition
• Mesh processing
• Result and Applications
• Discussion

4
Introduction

• Marker-based capture system
• - Color marker pattern
• - Multiple views
• Reconstruction
• Folds and occlusion
• - full colorspace
• - strain constraints
• Hole fiiling
• - Mesh-Based Inverse Kiematics

5
Previous work

• Pritchard and Heidrich 2003
• - Cloth motion capture, Eurographics
• Guskov et al. 2003
• - Trackable surfaces, SCA
• Scholz et al. 2005
• - Garment motion capture using color-coded
  patterns, Eurographics
• White et al. 2005
• - Cloth capture, Technical report
• White et al. 2006
• - Capturing real folds in cloth, Technical
  report

6
Contributions

• Improve color pattern and matching procedure
• Introduce strain constraints to simplify
  correspondence
• Create a data-driven hole filling technique that
  splices previously captured cloth into the mesh

7
System overview
Process color
Match With local neighborhoods
3D reconstruct
Prune with strain
images
Acquisition
Point cloud
Static Connectivity Triangle mesh
Mesh
Hole fill
Temporally smooth
Mesh processing
8
Acquisition 1

• Goal
• - compute correspondence using minimal
• neighborhoods
• Iterative algorithm
• - compute correspondence
• - prune bad matches
• Better than label propagation methods
• - strain constraints
• - color detection

9
Acquisition 2

• Color processing
• - compute color information for each
• marker
• - compute correspondence between image
• and parametric domain
• - eliminate correspondences that have
• large color differences

10
Acquisition 3

• Neighborhood matching
• - the approach works from flat regions in
• the first iteration to foldy regions in the
• later iteration
• Occluding contours no longer disrupt matching
  procedure

11
Acquisition 4
12
Acquisition 5

• 3D reconstruction
• - use textbook methods, Hartley and Zisserman
  2000 Multiple View Geometry

13
Acquisition 6

• Pruning with strain
• - two strain prining steps
• - one on reconstructed 3D points
• - one on marker obervations in each
• image

14
Acquisition 7
15
Acquisition 8

• Representation

Affinity between image marker i and parametric
marker j
Image neighbors of marker i
parametric neighbors of marker j
When only one affinity for image marker I is
above a threshold, we declare a
correspondence Use value for
threshold, N number of neighbors
16
Mesh processing 1

• Occlusion inevitably creates holes in the
  reconstucted mesh in acquisition
• Fill hole with previously observed sections of
  cloth
• Requirements
• - same topology
• - obey point constraints around edge of hole
• - splicing method respects property of cloth

17
Mesh processing 2

• Meshing and seams
• - insert artificial points where two pieces of
• fabric come together
• - 3D location of these points are recreated
• in each frame by averaging points near
• the seam

18
Mesh processing 3

• Hole filling
• - use occlusion free meshes from other
• frames to interpolate holes
• - use MeshIK to reconstruct the surfaces

19
Mesh processing 4

• MeshIK work

20
Mesh processing 5

• Smoothing
• - use flexibility preserving smoothing,
• smoothes rigid movement more heavily
• than non-rigid movement

21
Result and Applications 1
22
Result and Applications 2

• In pants, on a P4 2.4 GHz machine, acquisition
  takes roughly 6 minutes and mesh processing 2
  minutes perframe, code is written in MATLAB

23
Result and Applications 2

• Retargeting animations
• - use a set of captures frames to skin
  skeletal human motion capture data
• Drawback
• - loss of secondary kinematic motion
• - because MeshIK doesnt use velocity
• information, the resulting animation
• appears damped

24
Result and Applications 3
25
Discussion

• Future work
• - more cameras
• - higher resolution
• (leading to smaller denser marker)
• - different garments
• - different materials