Advanced Multimedia

1
Advanced Multimedia

• Image Content Analysis
• Tamara Berg

2
Announcements

• Start thinking about project ideas
• These can be related to text, sound, images,
  combinations of different media, interaction with
  digital media, social media,
• Next week on March 10 or 12 visit office hours to
  discuss your project ideas (20 points for coming
  with a well thought out idea or ideas).
• Can work alone or in pairs.
• Project proposal presentations March 17
• Reminder Assignment due March 12 (a week from
  today).

3
Possible Project Ideas

• Build a spam/ham detector
• Extend your pagerank algorithm to do web search
  by query
• Build a document topic classification system
• Implement a music retrieval system based on low
  level audio features or pitch class profile
• Implement an image retrieval system
• Projects related to combined sources of
  information images maps, images text
• Something cool with Twitter? Example twittering
  the superbowl, or twitter location
  information/maps.
• Projects related to interactivity
• Build a facebook app to do

4
How are images stored?
5
Reminder Images
Images are sampled and quantized measurements of
light hitting a sensor. What do we mean by
sampled? What is being quantized?
6
Images in the computer
7
Images in the computer
8
Color Images
Ellsworth Kelly Red Blue Green, 1963
img
9
Color Images
Ellsworth Kelly Red Blue Green, 1963
img
N
10
Color Images
M
Ellsworth Kelly Red Blue Green, 1963
img
N
11
Color Images
M
Ellsworth Kelly Red Blue Green, 1963
img
N

• Stored as 3d matrix of r, g, and b values at each
  pixel
• Image matrix would be size NxMx3
• R img(,,1) G img(,,2) B img(,,3)

12
Red component
img(,,1)
13
Green component
img(,,2)
14
Blue component
img(,,3)
15
Color Images
M
Ellsworth Kelly Red Blue Green, 1963
img
N

• Stored as 3d matrix of r, g, and b values at each
  pixel
• So img(i,j,) r g b

16
Color Images
M
j
i
Ellsworth Kelly Red Blue Green, 1963
img
N

• Stored as 3d matrix of r, g, and b values at each
  pixel
• So img(i,j,) r g b.
• In the case above this might be 255 0 0.

17
Useful Matlab image functions

• img imread(filename) read in an image
• imagesc(img) display an image
• imwrite(img,outfilename) write an image
• img(i,j,) indexes into the ith row, jth column
  of the image.
• subimg img(110,2030,) extracts part of img.

18
Matlab demo 1

• This will be very useful for homework 3!

19
How should we represent them?
20
Motivation

• Image retrieval
• We have a database of images
• We have a query image
• We want to find those images in our database that
  are most similar to the query

21
Motivation

• Image retrieval
• We have a database of images
• We have a query image
• We want to find those images in our database that
  are most similar to the query
• Similarly to text retrieval, music retrieval we
  first need a representation for our data.

22
How should we represent an image?
23
First try

• Just represent the image by all its pixel values

24
First try
Img1
Say we measure similarity as sim sum(abs(img1
img2))
Img2
25
First try
Img1
Say we measure similarity as sim average diff
between values in img1 and img2
Img2

• How similar are these two images? Is this bad?

26
Matlab demo 2
27
What do we want?

• Features should be robust to small changes in the
  image such as
• Translation
• Rotation
• Illumination changes

28
Second Try
Photo by marielito
Represent the image as its average pixel color
29
Second Try
Photo by marielito
Represent the image as its average pixel
color Pros? Cons?
30
Third Try
Photo by marielito
Represent the image as a spatial grid of average
pixel colors Pros? Cons?
31
QBIC system
QBIC link

• First content based image retrieval system
• Query by image content (QBIC)
• IBM 1995
• QBIC interprets the virtual canvas as a grid of
  coloured areas, then matches this grid to other
  images stored in the database.

32
Matlab demo 3
33
Color is not always enough!
The representation of these two umbrellas should
be similar. Under a color based representation
they look completely different!
34
What next?
Edges! But what are they how do we find them?
35
Reminder Convolution
36
Filtering
Alternatively you can convolve the input signal
with a filter to get frequency limited output
signal. Convolution
(convolution demo)
signal
f
filter
g
37
Filtering
Alternatively you can convolve the input signal
with a filter to get frequency limited output
signal. Convolution
(convolution demo)




38
Filtering
Alternatively you can convolve the input signal
with a filter to get frequency limited output
signal. Convolution
(convolution demo)




39
Filtering
Alternatively you can convolve the input signal
with a filter to get frequency limited output
signal. Convolution
(convolution demo)




40
Filtering
Alternatively you can convolve the input signal
with a filter to get frequency limited output
signal. Convolution
(convolution demo)




41
Filtering
Alternatively you can convolve the input signal
with a filter to get frequency limited output
signal. Convolution
(convolution demo)




42
Filtering
Alternatively you can convolve the input signal
with a filter to get frequency limited output
signal. Convolution
(convolution demo)



Convolution computes a weighted average.

43
Images -gt 2d filtering
44
Moving average

• Lets replace each pixel with a weighted average
  of its neighborhood
• The weights are called the filter kernel
• What are the weights for a 3x3 moving average?

Source D. Lowe
45
Moving average

• Lets replace each pixel with a weighted average
  of its neighborhood
• The weights are called the filter kernel
• What are the weights for a 3x3 moving average?

Source D. Lowe
46
Defining convolution in 2d

• Let f be the image and g be the kernel. The
  output of convolving f with g is denoted f g.

• Convention kernel is flipped
• MATLAB conv2 vs. filter2 (also imfilter)

Source F. Durand
47
g(x,y)
48
Moving Average In 2D
Source S. Seitz
49
Moving Average In 2D
Source S. Seitz
50
Moving Average In 2D
Source S. Seitz
51
Moving Average In 2D
Source S. Seitz
52
Moving Average In 2D
Source S. Seitz
53
Moving Average In 2D
What is this filter doing?
Source S. Seitz
54
Practice with linear filters
?
Original
Source D. Lowe
55
Practice with linear filters
Original
Filtered (no change)
Source D. Lowe
56
Practice with linear filters
?
Original
Source D. Lowe
57
Practice with linear filters
Original
Shifted left By 1 pixel
Source D. Lowe
58
Practice with linear filters
?
Original
Source D. Lowe
59
Practice with linear filters
Original
Blur (with a box filter)
Source D. Lowe
60
Gaussian Kernel
0.003 0.013 0.022 0.013 0.003 0.013
0.059 0.097 0.059 0.013 0.022 0.097
0.159 0.097 0.022 0.013 0.059 0.097
0.059 0.013 0.003 0.013 0.022 0.013
0.003
5 x 5, ? 1

• Constant factor at front makes volume sum to 1
  (can be ignored, as we should re-normalize
  weights to sum to 1 in any case)

Source C. Rasmussen
61
Example Smoothing with a Gaussian
source Svetlana Lazebnik
62
Edges
63
Edge detection

• Goal Identify sudden changes (discontinuities)
  in an image
• Intuitively, most semantic and shape information
  from the image can be encoded in the edges
• More compact than pixels
• Ideal artists line drawing (but artist is also
  using object-level knowledge)

Source D. Lowe
64
Origin of Edges
surface normal discontinuity
depth discontinuity
surface color discontinuity
illumination discontinuity
Edges are caused by a variety of factors
Source Steve Seitz
65
Characterizing edges

• An edge is a place of rapid change in the image
  intensity function

image
source Svetlana Lazebnik
66
Edge filters
Approximations of derivative filters
Convolve filter with image to get edge map
Source K. Grauman
67
Edge filters
Approximations of derivative filters
Respond highly to vertical edges
Source K. Grauman
68
Edge filters
Approximations of derivative filters
Respond highly to horizontal edges
Source K. Grauman
69
Edges example
source Svetlana Lazebnik
70
What about our umbrellas?
The representation of these two umbrellas should
be similar. Under a color based representation
they look completely different! How about using
edges?
71
Edges
Red umbrella
Gray umbrella
Edges extracted using convolution with Prewitt
filter
72
Edges
Edges overlayed from red and gray umbrellas. How
is this?
73
Edge Energy in Spatial Grid
Red Umbrella
Gray Umbrella
How is this representation?
74
Quick overview of other common kinds of Features
75
Important concept Histograms
Graphical display of tabulated frequencies, shown
as bars. It shows what proportion of cases fall
into each of several categories. The categories
are usually specified as non-overlapping
intervals of some variable.
76
Color Histograms
Representation of the distribution of colors in
an image, derived by counting the number of
pixels of each of given set of color ranges in a
typically (3D) color space (RGB, HSV etc). What
are the bins in this histogram?
77
Shape Descriptors shape context
Representation of the local shape around a
feature location (star) as histogram of edge
points in an image relative to that location.
Computed by counting the edge points in a log
polar space. So what are the bins of this
histogram?
78
Shape descriptors SIFT

• Descriptor computation
• Divide patch into 4x4 sub-patches
• Compute histogram of gradient orientations
  (convolve with filters that respond to edges in
  different directions) inside each subpatch
• Resulting descriptor 4x4x8 128 dimensions

David G. Lowe. "Distinctive image features from
scale-invariant keypoints. IJCV 60 (2), pp.
91-110, 2004.
source Svetlana Lazebnik
79
Texture Features
histogram
Convolve with various filters spot, oriented
bar. Compute histogram of responses.
Universal texton dictionary
Julesz, 1981 Cula Dana, 2001 Leung Malik
2001 Mori, Belongie Malik, 2001 Schmid 2001
Varma Zisserman, 2002, 2003 Lazebnik, Schmid
Ponce, 2003