Appearance Models

1
Appearance Models

• Shape models represent shape variation
• Eigen-models can represent texture variation
• Combined appearance models represent both

2
Appearance Models

• Statistical model of shape and texture
• Generative model
• general
• specific
• compact

3
Building Appearance Models

• For each example extract shape vector
• Build statistical shape model,

Shape, x (x1,y1, , xn, yn)T
4
Building Appearance Models

• For each example, extract texture vector

Shape, x (x1,y1, , xn, yn)T
Texture, g
Warp to mean shape
5
Warping texture

• Problem
• Given corresponding points in two images, how do
  we warp one into the other?
• Two common solutions
• Piece-wise linear using triangle mesh
• Thin-plate spline interpolation

6
Interpolation using Triangles
Region of interest enclosed by triangles. Moving
nodes changes each triangle Just need to map
regions between two triangles
7
Barycentric Co-ordinates
8
Barycentric Co-ordinates
Three linear equations in 3 unknowns
9
Interpolation using Triangles

• To find out where each pixel in new image comes
  from in old image
• Determine which triangle it is in
• Compute its barycentric co-ordinates
• Find equivalent point in equivalent triangle in
  original image
• Only well defined in region of convex hull of
  control points

10
Thin-Plate Spline Interpolation

• Define a smooth mapping function (x,y)f(x,y)
  such that
• It maps each point (x,y) onto (x,y) and does
  something smooth in between.
• Defined everywhere, even outside convex hull of
  control points

11
Thin-Plate Spline Interpolation

• Function has form

12
Building Texture Models

• For each example, extract texture vector
• Normalise vectors (as for eigenfaces)
• Build eigen-model

Warp to mean shape
Texture, g
13
Face Texture Model
14
Textured Shape Modes
Generate position of control points Warp mean
texture image (Mean points go to new points, X)
Shape variation (texture fixed)
15
Textured Shape Model
16
Combined Models

• Shape and texture often correllated
• When smile, shadows change (texture) and shape
  changes
• Learning this correlation leads to more compact
  (and specific) model

17
Learning Correlations
Model accounting for correlations between shape
and texture
Model assuming shape and texture independent
18
Learning Correlations

• For each image in training set we have best
  fitting shape and texture param.s
• Construct new vector,
• Apply PCA (mean eigenvec.s of covar.)

19
Combined Appearance Models
Varying c changes both shape and texture
20
Combined Appearance Model

• Generate shape, X, and texture, g
• Warp texture so mean control points lie on new X

21
Face Appearance Model
22
Face Appearance Model
23
Sub-cortical structures

• 72 examples
• 123 points
• 5000 pixel model

Caudate Nucleus
Lentiform Nucleus
Ventricles
24
Shape and Texture Modes
Shape variation (texture fixed)
Texture variation (shape fixed)
25
Combined Appearance Model

• Shape and texture correlated

26
Full brain slice
Shape
Texture
27
Full brain slice
Combined Mode 1
Combined Mode 2
28
Problems with viewpoint

• Models require all points visible
• Sometimes a problem for 2D images of 3D objects
• Small rotations (/-30o) of face modelled well
• Large rotations cause occlusions
• Eg eye hidden behind nose etc
• Solutions
• Use multiple view based 2D models
• Use a full 3D model

29
View-Based Models

• Build 3 distinct models
• Exploit symmetry

Profile
Profile (Reflected)
Half-Profile
Half-Profile (Reflected)
Frontal
30
Face Profile Model
Mode 1
Mode 2
31
Half-Profile Model
Mode 1
Mode 2
32
3D Models

• Use 3D shape model (3n-D vectors)
• Points control a polyhedral mesh
• Texture mapped onto mesh and modelled
• Reconstruct by generating new texture and mapping
  onto 3D mesh described by shape model

33
3D Models
Mesh


Texture
34
Interpreting Images (1)
Place model in image
Measure Difference
Update Model
Iterate