IPAM-UCLA Tutorial May 14-18, 2001 GBM in Image Processing, Computer Vision, and Computer Graphics

1
IPAM-UCLA TutorialMay 14-18, 2001 GBM in Image
Processing,Computer Vision, andComputer
Graphics

• Guillermo Sapiro
• Electrical and Computer Engineering
• University of Minnesota
• guille_at_ece.umn.edu

Supported by IPAM, NSF, ONR
2
GBM Moving Curves
3
Basic curve evolution

• Planar curve
• General flow
• General geometric flow

4
Mathematical morphology

• Classical theory, based on Minkowsky addition.
• The old and (probably wrong) way of doing
  geometric image analysis.
• Has very important lessons to learn!!!!
• Basic definitions
• A Image in Euclidean space (R or Z)
• B Structuring element (symmetric)
• Nothing else than Minkowsky addition

5
Mathematical morphology Definitions
6
Mathematical morphology Is it good or bad?

• Advantages
• Nice mathematical properties (set theory)
• Extension to Lattices
• Disadvantages
• Discrete Minkowsky addition does not look good,
  has to be replaced by better ways of computing
  discrete distances.
• Major important concept Level-sets
• Commutes with thresholding (level-sets) Do
  binary on each level sets or do gray-level on all
  the image gt same result
• It is in certain sense a particular case of curve
  evolution (before the lattices part)

7
Mathematical morphology via curve evolution
Huygens principle
8
Mathematical morphology via curve evolution
(cont.)

• General velocity
• Examples
• Nothing else than changing the metric (distance).
  
• Can be explained also based on dynamic
  programming and time of arrival
• See Sapiro et al., Brocket-Maragos, Alvarez et
  al., Evans, Falcone

9
Planar differential geometry

• Euclidean invariant parametrization

• Affine invariant parametrization

C
s
(
)



1


s


C
C
,
lt
gt

0
s
ss


C
C

s
ss
k


C

ss
10
Planar differential geometry (cont.)

• Curvature constant for circles or straight lines
  (0)
• Curvature defines curve up to Euclidean motion
• At least 4 points with dk/ds0
• Defined for all curves

• Curvature constant for ellipses (gt0), hyperbolas
  (lt0), and parabolas (0)
• Curvature defines curve up to affine motion
• At least 6 points with dk/ds0
• Defined only for convex curves segment at
  inflection points

11
Planar differential geometry (cont.)
X
C(s)
Distance has a local extrema iff X is on the
normal
12
3D Differential geometry

• Remember mean and Gaussian curvatures?
• Each regular surface has two principal
  curvatures. The average is the mean curvature,
  the product the Gaussian. These are also related
  to the tangential map, etc, etc. See DoCarmo for
  details.

13
Riemannian geometry, Lie theory

• What about other non-Euclidean metrics?
• What about invariants to other (Lie) groups,
  e.g., projective?
• What about differential invariants?
  Semi-differential invariants? Are there any
  general theories?

14
Smoothing by classical heat flow

• Linear
• Equivalent to Gaussian filtering
• Unique linear scale-space
• Non geometric
• Shrinks the shape
• Implementation problems

15
Invariant shape deformations

• Formulate shape deformations
• Geometric
• Invariant to camera transform
• The best possible
• Change only the desired features
• Motivation
• Mathematics
• From static differential geometry to dynamic
• Beautiful
• Computer vision and image processing
• Invariant shape segmentation and analysis
• Image processing via image deformations
• Robotics
• Motion planning
• Accurate geometric object detection and tracking
• Robot manipulation and grasping

16
Basic planar differential geometry

• For every Lie group we will consider, exists and
  invariant parametrization s, the group arc-length
• For every such a group exists and invariant
  signature, the group curvature, k

Low curvature
High curvature
Negative curvature
17
What and why invariant

• Camera
• motion
  Deformation
• Camera/object movement in the space
• Transformations description (for flat objects)
• Euclidean
• Motion parallel to the camera and planar
  projection
• Affine
• Planar projection
• Projective

18
Euclidean geometric heat flow

• Use the Euclidean arc-length
• The deformation
• Smoothly deforms to a circle (Gage-Hamilton,
  Grayson)
• Geometric smoothing
• Reduces length as fast as possible

19
Affine geometric heat flow (Sapiro-Tannenbaum)

• Use the affine arc-length
• The flow

20
Affine geometric heat flow (cont.)

• Geometric smoothing (preserving area if desired)
• Total curvature decreases
• Maxima of curvature decreases
• Number of inflections decreases
• Smoothly deforms a shape into an ellipse
• Decreases area as fast as possible (in an affine
  form)
• Existence also for non-smooth curves
• Viscosity framework (Alvarez-Guichard-Morel-Lions)
  
• Polygons (Angenent-Sapiro-Tannenbaum)
• Applications
• Curvature computation for shape recognition
  reduce noise (Morel et al.)
• Simplify curvature computation (Faugeras 95)
• Object recognition for robot manipulation
  (Cipolla 95)

21
General invariant flows

• Theorem For every sub-group of the projective
  group the most general invariant curve
  deformation has the form
• Theorem In general dimensions, the most general
  invariant flow is given by
• u graph locally representing the surface
• g invariant metric
• E(g) variational derivative of g
• See Olver et al., Alvarez et al., Caselles-Sbert

22
General Geometric Framework For Object
Segmentation
23
Introduction

• Goal Object detection
• Approach Curve/surface deformation
• Geometry dependent regularization
• Image dependent velocity
• Characteristics
• Unifies previously considered independent
  approaches
• Relates segmentation with anisotropic diffusion
• General
• Any topology
• Any type of image data
• Any dimension
• Holds formal results

24
Notation

• Deforming curve
• Image

25
Basic active contours approach

• Terzopoulos et al., Cohen et al.
• Drawbacks
• Too many parameters
• Non-geometric
• Handling topology changes

26
Geodesic active contours (Caselles-Kimmel-Sapiro)

• Generalize image dependent energy
• Eliminate high order smoothness term
• Equal internal and external energies
• Maupertuis and Fermat principles of dynamical
  systems

27
Geodesic computation

• Gradient-descent
• Level-sets (Osher-Sethian)

28
Further geometric interpretation

• The geodesic flow

29
Model correctness

• Theorem The deformation is independent of the
  level-sets embedding function
• Theorem There is a unique solution to the flow
  in the viscosity framework
• Theorem The curve converges to ideal objects
  when present in the image
• Related work
• Kimia-Tannenbaum-Zucker
• Caselles et al.
• Malladi-Sethian-Vemuri
• Kichenassamy at al.
• Tek-Kimia, Whitacker
• New work
• Chan-Vese
• Paragios-Deriche
• Yezzi et al.
• Faugeras et al.

30
Extensions

• Gray-level values
• ds - length element (geodesics)
• Ordinary edge detector (gradient)
• Surfaces
• ds - area element (minimal surfaces)
• 3D edge detector
• Vector-valued images (color, texture, medical,
  etc)
• ds - length element
• Vector-valued edge detector (vector geodesics)
• Eigenvalues of the first fundamental form in
  Riemannian space
• Invariant detection (affine area geodesics)
• ds - affine length element (area related)
• Affine invariant edge detector
• Affine norm for gradient descent

31
Why color edges?
32
Notation

• Image
• Texture Gabor decomposition

33
Color edge computation

• Given a metric (Euclidean)
• Compute first fundamental form
• Compute eigenvectors and eigenvalues
• Edge maximal eigenvalue and its eigenvector
• Basic properties
• Eigenvectors are orthonormal
• Minimal eigenvalue is not zero

34
GBM Moving Images
35
Anisotropic diffusion
Isotropic vs. Anisotropic Smoothing
36
Isotropic diffusion (Koenderink, Witkin)

• All equivalent
• Gaussian filtering of the image
• Heat flow
• Minimize the L2 norm

37
Isotropic diffusion Good things

• Gaussian filtering if and only if
• Linear
• Shift-invariant
• No creation of zero crossings
• Gaussian filtering if and only if
• Linear
• Shift-invariant
• Semi-group property
• Scale-invariant (dimensionless)
• Unique linear filter that defines a scale-space
  Do not creates information at coarser scales
• Where everything started (Koenderink, Witkin)

38
Isotropic diffusion Bad things and possible
solutions

• Non-geometric
• Problems with implementations
• Who said linear? Replace heat flow by parabolic
  PDEs (Hummels original idea)
• Why parabolic? Because of the maximum principle.

39
Perona-Malik anisotropic diffusion

• Replace the L2 by a different norm (e.g., L1,
  Rudin-Osher-Fatemi Lorentzian, Black et. al.
  etc)

40
Selection of stopping term h

• How do we select h?
• hxx gt L2 gt linear gt Isotropic
  diffusion
• hx gt L1 (Rudin-Osher-Fatemi)

41
Robust anisotropic diffusion

• General theory for selection h, based on the
  theory of influence functions in robust
  statistics
• Edges should be considered outliers At certain
  point, h, the influence, should be zero.

42
Directional diffusion

• Diffuse in the direction perpendicular to the
  edges (Avarez et al.)

43
From active contours to anisotropic diffusion

• Replace embedding function in level-sets
  formulation by image itself

Shock-filters (Osher-Rudin)
Anisotropic diffusion (Alvarez et al.)
44
Relation with Perona-Malik anisotropic diffusion
Total variation, Robust estimation
Anisotropic diffusion
45
Concluding remarks
Terzopoulos snakes
Geometric interpretation Dynamical
systems Level-sets
Terms elimination
Curve evolution active contours
Geodesic active contours
Use image as embedding
Geometric diffusion
Self-snakes
Mumford-Shah
Add
Shock-filters
Divide by gradient
Perona-Malik flow
Variational interpretation
Total Variation
Robust Estimation
46
Anisotropic Diffusion of the Posterior
47
ADP in MRI
Review MAP Estimation

• 3 classes sulcus, gray matter, white matter
• Prior probability Pr(classC)
• Posterior probability Pr(classC data)
• MAP Choose class C that maximizes posterior
• C arg max Pr(classC data)
• C
• Bayes Rule
• Pr(classC data) Pr(data
  classC).Pr(classC)
• Pr(data)
• What is our prior, Pr(classC)?

48
ADP Common Techniques
MAP Estimation Uniform Prior
Classification
49
ADP Results
Anisotropic smoothing of posterior
(Teo-Sapiro-Wandell)
Smoothed Posterior
Classification
Posterior
50
ADP Comparisons
Comparison with manual segmentation
Automatic (2 min)
MR Image
Manual (18 hours)
51
SAR segmentation via vector probability
processing
With A. Pardo (see also Haker-Sapiro-Tannenbaum)
52
Anisotropic Diffusion in Vector Space
53
Goal and approach (Ringach-Sapiro)

• Goal
• Enhancement of vector valued data
• Extend classical theories of scalar PDEs in
  image processing
• Approach
• Work in vector space
• Compute vector edges
• Anisotropic diffusion
• Important Works for any vector data
• See also Cumani, Di Zenzo, Chambolle

54
Notation

• Image
• Texture Gabor decomposition

55
Color edge computation

• Given a metric (Euclidean)
• Compute first fundamental form
• Compute eigenvectors and eigenvalues
• Edge maximal eigenvalue and its eigenvector
• Basic properties
• Eigenvectors are orthonormal
• Minimal eigenvalue is not zero

56
Color anisotropic diffusion

• Direction Minimal change
• Strength

57
Level lines for vectorial images (Chung-Sapiro)
Vector and scalar representation sharing
level-lines
58
Contrast Enhancement (Sapiro-Caselles, and
Caselles-Lisani-Morel-Sapiro)

• Contrast enhancement via image deformations
• Approach Histogram modification
• Characteristics
• Simultaneous contrast enhancement and denoising
• First explanation of histogram modification in
  image domain
• Extended to local
• First semi-global partial differential equation
  in image processing
• Formal existence results

59
GBM Beyond the flat manifolds
60
The main problem and our goal (Tang-Sapiro-Caselle
s)

• Goal Enhancement and analysis of directional
  data (and data on non-flat manifolds)
• Problem Directions are unit vectors
• Regular images vs Directions
• Applications
• Optical flow, Gradients
• Vector data (normalized)
• Color image enhancement
• Surface normals and principal directions
• Flows in general manifolds

61
Average
62
Most popular previous approaches

• Work with angles Operations on the sphere
• Average, median, etc
• Statistics of directional data, Mardia
• Orientation Diffusion, Perona (1998)
• Tensor diffusion
• Weickert, Granlund-Knuttson
• See also Chan-Shen

63
Anisotropic Diffusion
Isotropic (Heat equation)
Anisotropic
64
What have we learned from images?
Robust Estimation
Robust function
Gradient Descent
Influence function (defines outliers)
Anisotropic Diffusion
65
Anisotropic Diffusion
Isotropic (Heat equation)
Anisotropic
66
Back to Directions Basic Idea

• Use the theory of harmonic maps
• Find a map I from two manifolds (M,g) and (N,h)
  such that
• In particular, liquid crystals

67
The Gradient-Descent Equations
General (p2)
Liquid crystals
68
A Few Theoretical Results (over hundreds relevant)

• For 2D unit vectors (n1), and p2, a unique
  solution exists and singularities are isolated
  points (if they exists at all). For smooth data,
  singularities do not occur.
• Singularities occur for 3D unit vectors (p2).
• Singularities well characterized for 1ltplt2.
• Energy well characterized for 1ltplt2.
• No singularities for manifolds with non-positive
  curvature.

69
Intermezzo Visualizing Directions

• Arrows
• Color Map
• Line Integral Convolution

70
Examples (Isotropic)
71
3D vector (Isotropic)
72
Denoising
73
(No Transcript)
74
(No Transcript)
75
Optical flow
76
Optical flow (cont.)
77
Gradient
78
Gradient (cont.)
79
Color Image Enhancement
80
Color image enhancement (cont.)
81
Color image enhancement (cont.)
82
Vector probability diffusion (with Alvaro Pardo)

• Perform diffusion on the hyperplane representing
  probabilities

83
Vector probability diffusion (cont.)

• The numerical implementation also stays on the
  hyperplane
• The numerical implementation also holds a maximum
  and minimum principle

84
Vector probability diffusion (cont.)

• Diffuse posterior probabilities (following
  Teo-Sapiro-Wandell and Haker-Sapiro-Tannenbaum)

85
Current Results and Future Research

• Novel framework for analysis of directional data
• Isotropic and anisotropic
• Works in any dimension
• Supported by theoretical results on existence,
  uniqueness, singularity classification
• See also Sochen-Kimmel-Malladi IEEE-IP 98,
  Chan-Shen UCLA 99, Osher-Vese UCLA 00.