Image processing and computer vision Chapter 7: Mean-shift and Cam-shift

1
Image processingand computer vision

• Chapter 7 Mean-shift and Cam-shift
• Ref
• 1 Dorin Comaniciu, Peter Meer,"Mean Shift A
  Robust Approach Toward Feature Space
  Analysis"Volume 24 , Issue 5 (May 2002),IEEE
  Transactions on Pattern Analysis and Machine
  Intelligence
• 2 web.missouri.edu/hantx/ECE8001/notes/Lect7_me
  an_shift.pdf

2
What is Mean-shift?

• Find the peak of a probability function by the
  change of the mean of the data
• Applications
• Segmentation
• Non-rigid object tracking

3
Applications segmentation of regions of images
in a movie

• Use color to segment the image into logical
  regions for analysis.
• If the regions are moving , mean-shift is useful.

http//www.youtube.com/watch?vn1W0IwipRBQfeature
related
4
Application tracking non-rigid object

• Human tracking

http//www.youtube.com/watch?vzLtjPfPP9HY
5
Intuition find the mode by mean-shift

• Target Find the modes (peaks) in a set of
  sample data.
• The mode of a continuous probability distribution
  is the peak.
• There may be multiple peaks.
• The method used is called mean-shift.
• By finding the shift of the mean, we can find the
  mode (peak)
• It can be used to segment an image into logical
  regions. (e.g. within each region, the color is
  the same.)

6
First we need to understand the Probability
Density Function PDF

• We use
• Kernel density estimation to find PDF

7
Motivation for Kernel density estimation to find
PDF

• The formula (parametric form) of the PDF
  (probability density function) is difficult to
  find.
• Use sampling method to estimate the P.D.F.
• That means Gaussian ( a parametric form with
  mean , standard deviation etc., is easy to use),
  but it is too simple to model real life problems.

PDF(x)
Too simple to model real life problems
x
0
An irregular shape PDF, the distribution Is
difficult to model using parameters --use
non-parametric methods instead
Gaussian distribution
8
Example

• Outbreak of flu in a year
• How do you model this PDF?

CUHK Clinic Patients Number Per day
100
month
3 6
9 12
9
Kernel density estimation KDE Demo

http//parallel.vub.ac.be/research/causalModels/tu
torial/kde.html
10
kernel density distribution function

• The general form of a kernel density distribution
  function
• The Kernel (K) has many choices
• Epanechnikov
• Uniform
• Normal (Gaussian)

11
Non-parametric methods --Histogram--Kernel
density estimation (discuss here)
General form of a Kernel density function

Add kernel functions to become the final P.D.F
Each one is a kernel function K(x-xi)/h)
Histogram
Kernel density
http//en.wikipedia.org/wiki/Kernel_density_estima
tion
12
The concept General form of a Kernel density
functionThe PDF summation of many small kernel
functions

The PDF
Kernel functions at each sample point
http//www.cs.cornell.edu/courses/cs664/2005fa/Lec
tures/lecture3.pdf
13
Kernel density in 2D (x1,x2 as axes) example

• E.g PDF of mosquitoes in CU.
• At time t, if you can find one mosiquito at a
  location xi (x1,x2)i, mark it green in the
  diagram.
• Each point is a density kernel function
• K(x-xi)/h)
• Say h1 meter
• You can build up the final PDF of mosquitoes by
  summing all kernel functions as in the above
  equation fh(x)

x2
Each green dot has such a circle
h
xi
x1
PDF of mosquitoes in CU
14
Illustration of the cam-shift (mean-shift)
ideaYou want to find the location of a circle
(fixed radius) to enclose the biggest number of
points. ? The more points, the bigger the PDF.

• Guess the peak is at xt0
• Draw a circle of radius h
• Find mean of points inside windowmean_loc(xt)
• Move the circle for xt1 at mean_loc(xt)
• m(t)mean_loc(xt)-xtmean-shift vector
• tt1
• Repeat 1-5 , till m(t) is small enough
• X is the result. (peak)

x2
Mean-shift-vector m(xt)
mean_loc(xt) of the green dots in the circle
h
x
x1
15
Example 1Show graphically of mean-shit

• Starting from x and the circle shown
• Find the peak of the PDF.
• Repeat the task if your first selection is at x

x2
x
h
x
PDF of mosquitoes in CU
x1
16
How about the kernels K (?? )?

• Building blocks

17
Kernel choices (??)

• Different radial symmetric kernel profiles

Epanechnikov Uniform Normal (Gaussain)
18
Exercise 1
Epanechnikov

• Draw KE(w), c1

KE(w)
w
-1 -0.75 -0.5 -0.25
0 0.25 0.5 0.75 1
19
Exercise 2

• Replace w in KE(w), by ((x-xi/h)
• Draw KE(x-xi/h), c1 when
• Xi0 and h2 when (x-xi/hlt1
• Xi3 and h3 when (x-xi/hlt1

KE(x-xi/h),
1
x
-6 -5 -4 -3 -2 -1
0 1 2 3 4 5 6
20
Exercise 3

• Each sample has an Epanechnikov distribution
• Estimate h?
• n?
• Draw the final PDF

Each sample xi is an Epanechnikov distribution
Only show 1-dimension
x
xi
Xi1
0 25 50 75
21
Go back to the PDF

• Find the peak of PDF by finding x when ?PDF(x)0

22
MotivationKernel Density (gradient) Estimation

• The peak of a Probability distribution (PDF) P(x)
  is difficult to find,
• so we find the gradient dP(x)/dx?P(x)
• And when ?P(x)0 , it is at the peak of P(x)

23
If you have the PDF, you can find the peak
(mode), how?

• If you know the formula of the PDF, you can find
  the mode (peak). I.e. peakmax (pdf)
• IF max (pdf) is difficult to find,
• find gradient (pdf)0 so ?(PDF)0

peak
peak
?(PDF)0
24
Step1 Model the PDF
25
Exercise 4 Derivation exercise

26
Derivation

27
Continue
28
Continue
29
Exercise 5Kernel Density (gradient) Estimation

Mean-shift-vector

30
Finding Mean-Shift-vector

Always Positive
Mean-shift-vector

• Simple Mean Shift procedure
• Compute mean-shift-vector
• Translate the Kernel window by m(x)

The proof of convergence can be found in 1
31
The gradient of PDF ? P(x) is proportional to
mean-shift vector m(x)

ve
mean shift vector m(x)
32
P(a) points to the increasing side of P(x)

• Taylor series, definition
• P(x)P(a)? P(a)(x-a)
• At the left hand side of the peak
• At P(a), bringing (a) to (x) will increase p(x).
• (x-a) is ve , and ? P(a) is ve and is pointing
  to the direction of increase of P(x)
• At the right hand side of the peak
• At P(a) , bringing (a) to (x) will increase
  p(x).
• (x-a) is -ve , and ? P(a) is -ve and is
  pointing to the direction of increase of P(x)
• So ? P(a) is pointing towards the increase of
  P(x)
• Since ? P(a) proportional to m(x)
• so m(x) points to the direction of increase of
  P(x).

Peak
P(x)
x
a
x
x a
left side of peak
right side of peak
33
Mean-shift

• Algorithm and procedure

34
Procedures To find the peak of PDF
Gradient of PDF

• For gradient of PDF ? P
• The first term is a PDF , it is positive.
• The second tern m(xt) always points to the
  maximum PDF direction (proof in previous slides
  1)
• So the peak can be found by
• First guess at x(t0)
• iterate followings until Xt1xt (meaning m(xt)
  too small)
• Find m(xt)
• Xt1xt m(xt)
• Increment t
• Xpeak xt
• Same as
• x(t1)x(t) ? P(x)
• Since ? P(x)? m(xt)

ve
P(x)
Peak
m(xt1)
m(xt)
x
Xpeak(tt)
xt
xt1
xt 2
1 Dorin Comaniciu, Peter Meer,"Mean Shift A
Robust Approach Toward Feature Space
Analysis" Volume 24 , Issue 5 (May 2002),IEEE
Transactions on Pattern Analysis and Machine
Intelligence
35
A target tracking tool using mean-shift Camshaft

• Demo opencv video

http//www.youtube.com/watch?viBOlbs8i7Og
36
Cam shift

• An algorithm for object tracking

37
Motivation

• For target tracking , mean shift can be used.
• Implementing the full mean shift algorithm is too
  complex
• It is found that the implementation can be found
  by zero-moment and first-moment
• We will show how
• Ref Gary R. Bradski, Computer Vision Face
  Tracking For Use in a Perceptual User,
  Microcomputer Research Lab, Santa Clara, CA,
  Intel Corporation

38
Overview

• Color features for tracking
• Object tracking by cam-shift
• Step1 find object color model (histogram)
• Step2 use the model to track using mean-shift,
  use histogram-back-project to identify the object
  for tracking
• Illustration of the cam-shift idea
• Demonstration
• Applications
• Face tracking
• Object tracking

39
(1) Color features for tracking

• Color processing
• Recall, hue ??
• From 0 to 360 degrees
• Encode values0?1

0 0.5
1
40
2) Object trackingStep1Procedure to find object
color model histogram, example

• Learn object color distribution e.g. face color,
  hue is enough to identify face
• Circle face by hand, keep hue image only Hue
  0?7 levels
• Find model face histogram of Hue color inside
• E.g. 8 bins

I(x,y) probility_of_face_color(Hue of the pixel)
counts
Hue
Histogram
41
Example to find modeland histogram back project
e.g. Pixel I(12,34) Hue is 3
x2
64x64

• Color histogram Model building
• Image size M64,N64. Total(MxN)
• For the hue image, each pixel value rk 0?7. (In
  practice, use 0?255)
• The number of pixels having hue_value rk is nk
• Use 8 bins (buckets). (In practice, use 32 bins)
• Histogram back project PhistI(x,y)
• You have a pixel rk I(x1,x2) (with hue color
  rk.)
• PhistI(x1,x2) is the probability this pixel
  belonging to a face
• E.g. rk I(12,34)3 (see next slide for the
  histogram)
• Read up Phist(rk 3) from histogram ,
• Phist3 0.160

x1
I(rk)
Hue
42
Example

• A numerical exercise

42
43
2) Object tracking step2 Recall mean-shift
procedures to find the peak of PDF
Gradient of PDF

• For gradient of PDF ? P
• The first term is a PDF , it is positive.
• The second tern m(xt) always points to the
  maximum PDF direction (proof in previous slides
  1)
• So the peak can be found by
• First guess at x(t0)
• iterate followings until Xt1xt (meaning m(xt)
  too small)
• Find m(xt)
• Xt1xt m(xt)
• Increment t
• Xpeak xt
• Same as
• x(t1)x(t) ? P(x)
• Since ? P(x)? m(xt)

ve
P(x)
Peak
m(xt1)
m(xt)
x
Xpeak(tt)
xt
xt1
xt 2
1 Dorin Comaniciu, Peter Meer,"Mean Shift A
Robust Approach Toward Feature Space
Analysis" Volume 24 , Issue 5 (May 2002),IEEE
Transactions on Pattern Analysis and Machine
Intelligence
44
Weighted kernel density for image tracking

45
What is moment in computer vision?

• In mechanics, it is momentlength mass.

3Kg
2Kg
3 m
2 m
Position o

• M1First Moment on this side about position ois
  2x3 Kgm
• M0Zero moment is 2Kg, no length involved

• M1First Moment on this side about position o
• is 3x2 Kgm
• M0Zero moment is 3Kg, no length involved

46
In computer vision it is the pixel intensity
length
I(x,y), the probability of this pixel has the
color of skin is PhistI(x1,I2)
x2
x1

• If only intensity is considered
• In our application, we want to know the
  probability of a pixel that has the color of
  skin, use histogram back project PhistI(x1,I2)
  instead of I(x1,I2)

47
Find first, zero moment in mean shift vector m(x)
Histogram_back_project PhistI(x1,x2)
First moment
Zero moment
m(x)Mean-shift-vector in the mean-shift algorithm
48
The Cam-shift algo. xt(x1,x2)t

• Recall m(x1,x2)(M10/M00), (M01/M00)-x1,x2
• ---The camshift algorithm --
• First guess at x(t0) x1,x2t0 .
• Form a search window centered at xt0
• Calculate M10,M01,M00 of the search window
• Find mean_loc(xt)(M10/M00),(M01/M00)t
• Find mean-shift-vector m(xt)mean_loc(xt)-x1,x2t
  
• New search window will center at mean_loc(xt)
• tt1
• Loop to 1 until m(xt) lt threshold
• Xpeak x1,x2t

• Recall the mean-shift algo.
• The peak can be found by
• First guess at x(t0)
• iterate followings until Xt1xt (meaning m(xt)
  too small)
• Find m(xt)
• Xt1xt m(xt)
• Increment t
• Xpeak xt

49
(3) Illustration of the cam-shift (mean-shift)
ideaYou want to find the location of a circle
(fixed radius) to enclose the biggest number of
points. ? The more points, the bigger the PDF.

• Guess the peak is at xt0
• Draw a circle of radius h
• Find mean of points inside windowmean_loc(xt)
• Move the circle for xt1 at mean_loc(xt)
• m(t)mean_loc(xt)-xtmean-shift vector
• tt1
• Repeat 1-5 , till m(t) is small enough
• X is the result. (peak)

x2
Mean-shift-vector m(xt)
mean_loc(xt) of the green dots in the circle
h
x
x1
50
(4) Demonstration of result Animation of tracking

• Demo

http//il.youtube.com/watch?vMfo_wE17bogfeature
related
51
Appendix
52
(5) Rotation tracking

53
(7) Applications Head motion tracking

54
Compare mean shift vector and zero, first moments

• The new Cam shift (mean shift using
  zero-first-moment)
• Recall in the previous slide
• M(x,y)(M10/M00), (M01/M00)-x,y
• ---The new camshift algorithm --
• First guess at x(t0)
• Iterate until m(xt)0
• Find m(xt)(M10/M00),(M10/M00)t-x,yt
• x,yt1x,yt m(xt)
• x,yt1x,yt (M10/M00),(M10/M00)t-x,yt
• x,yt1(M10/M00),(M10/M00)t
• Xpeak x,yt
• The centriod window is shifted by xc,yct
  (M10/M00),(M10/M00)t

• The general Mean shift algorithm
• The peak is found by
• First guess at x(t0)
• iterate until m(xt)0
• Find m(xt)
• Xt1xt m(xt)
• Xpeak xt

For Camshift X?x,y
55
Use excel to plot an Epanechnikov distribution
(cut and paste to excel to view plot)each cell
is(1/(23.1416A25A25))(1-(1(
(A2-B25)2 (B1-C25)2) /
(1(A25)(A25) )))

56
Use excel to plot a Gaussian (or normal)
distribution(cut and paste to excel to view
plot) each cell is (1/(23.1416A25A25))EX
P(-1( (A3-B25)2 (B2-C25)2)
/ (2(A25)(A25) ))

57
Answer Exercise 1

• Draw KE(w), c1

KE(w)
x
-1 -0.75 -0.5 -0.25
0 0.25 0.5 0.75 1
58
Answer Exercise 2

• Draw KE(x-xi/h), h2,c1 when
• Xi0 when (x-xi/hlt1
• Xi3 when (x-xi/hlt1

KE(x-xi/h),
1
x
-6 -5 -4 -3 -2 -1
0 1 2 3 4 5 6
59
Answer Exercise 4a Derivation

60
AnswerExercise 3

• Each sample has an Epanechnikov distribution
• Estimate h?
• n?
• Draw the final PDF

Each sample xi is an Epanechnikov distribution
Only show 1-dimension
x
xi
Xi1
0 25 50 75
61
Answer5a Exercise 5Kernel Density (gradient)
Estimation

62
Answer5b Exercise 5Kernel Density (gradient)
Estimation