Bayesian Content-Based Image Retrieval

1
Bayesian Content-Based Image Retrieval

• research with
• Katherine A. Heller
• based on (Heller and Ghahramani, 2006)
  part IB, paper 8, Lent

2
What is Information Retrieval?

• finding material from within a large unstructured
  collection (e.g. the internet) that satisfies the
  users information need (e.g. expressed via a
  query).
• well known examples
• but there are many specialist search systems as
  well

3
(No Transcript)
4
(No Transcript)
5
(No Transcript)
6
Universe of items being searched
Imagine a universe of items
The items could be images, music, documents,
websites, publications, proteins,
news stories, customer profiles,
products, medical records, or any other type of
item one might want to query.
7
Illustrative example
8
Generalization from a small set

• Query is a set of items
• Our information retrieval method should rank
  items x by how well x fits with the query set

9
Bayesian Inference Statistical Models

• Statistical model for data points
• with model parameters
• Prior on model parameters
• Dataset and model class
• Marginal likelihood (evidence) for model

10
Illustrative example
11
Ranking items

• Rank each item in the universe by how well it
  would fit into a set which includes the query
  set
• Limit output to the top few items

12
A Criterion?

• Having observed , belonging to some concept,
  how probable is it that an item also
  belongs to that concept ?
• What we really want to know is
  relative to ,
  , the probability of the item before
  observing the query

13
Bayesian Sets Criterion
So we compute

• Assume a simple parameterized model, ,
  and a prior on the parameters, .
• Since is unknown, to compute the score we
  need to average over all values of

14
Bayesian Sets Criterion(A Different Perspective)
We can rewrite this score as
15
Bayesian Sets Criterion (A Different Perspective)
This has a nice intuitive interpretation
16
Bayesian Sets Criterion
17
Bayesian Sets Algorithm

• For simple models computing the score is
  tractable.
• For sparse binary data computing all scores can
  be reduced to a single sparse matrix
  multiplication.
• Even with very simple models and almost no
  parameter tuning one can get very competitive
  retrieval results.

18
Sparse Binary Data
E.g
If we use a multivariate Bernoulli model
With conjugate Beta prior
We can compute
This daunting expression can be dramatically
simplified
19
Sparse Binary Data
Reduces to
where
and
20
Priors

• broad empirical priors from entire data set
  chosen before observing any queries
• prior proportional to mean feature frequency
• robust to changes in

21
Key Advantages of Our Approach

• Novel search paradigm for retrieval
• queries are a small set of examples
• Based on
• principled statistical methods (Bayesian machine
  learning)
• recent psychological research into models of
  human categorization and generalization
• Extremely fast
• search gt100,000 records per second on a laptop
  computer
• uses sparse matrix methods
• easy to parallelize and use inverted indices to
  search billions of records/sec

22
Applications

• Retrieving movies from a database of movie
  preferences
• EachMovie Dataset (person,movie) entry is 1 if
  the person gave the movie a rating above 3 stars
  out of a possible 0-5 stars
• Finding sets of authors who work on similar
  topics
• NIPS authors dataset (word,author) entry is 1 if
  the author uses that word more frequently than
  twice the mean across all authors
• Searching scientific literature
• NIPS dataset (word, paper) entry is 1 if the
  paper uses that word more frequently than twice
  the mean across all papers
• Image retrieval based on color and texture
  features only
• Corel dataset (image, feature) matrix contains
  240 binary features per image Gabor and Tamura
  texture features and HSV color features
• Searching a protein database
• UniProt database the worlds most comprehensive
  catalog of information on proteins. Binary
  features from GO annotations, PDB structural
  information, keywords, and primary sequences.
• Patent Search (Xyggy.com)

23
Retrieving MoviesEachMovie Data 1813 people by
1532 movies
24
Retrieving Movies
comparison to Google Sets
25
Retrieving Movies
comparison to Google Sets
26
(No Transcript)
27
(No Transcript)
28
Query Times
29
Content-Based Image Retrieval

• We can use the Bayesian Sets method as the basis
  of a content-based image retrieval system

30
The Image Retrieval Prototype System

• A system for searching large collections of
  unlabelled images.
• You enter a word, e.g. penguins, and it
  retrieves images that match this label, using
  only color and texture features of the images
• A database of 32,000 images (from Corel)
• Labelled Training Images 10,000 images with
  about 3-10 text labels per image
• Unlabelled Test Images 22,000 images
• For each training and test image we can store a
  vector of 240 binary color and texture features
• A vocabulary of about 2000 keywords
• For each keyword, we can compute a query vector q
  from the labelled training images, as is
  specified by the Bayesian Sets algorithm.

31
Image features

• Texture features (75)
• 48 Gabor features
• 27 Tamura features
• Color features (165)
• HSV histogram (8x5x5)
• Binarization
• Compute skewness of each feature
• Assign value 1 to images in heavier tail

32
The Image Retrieval Prototype System

• The Algorithm
• Input query word wpenguins
• Find all training images with label w
• Take the binary feature vectors for these
  training images as query set and use Bayesian
  Sets algorithm
• For each image, x, in the unlabelled test set,
  we compute score(x) which measures the
  probability that x belongs in the set of images
  with the label w.
• Return the images with the highest score
• The algorithm is very fast
• about 0.2 sec on this laptop to query 22,000
  test images

33
(No Transcript)
34
(No Transcript)
35
(No Transcript)
36
(No Transcript)
37
(No Transcript)
38
(No Transcript)
39
(No Transcript)
40
(No Transcript)
41
Results on all 50 queries
42
Results for Image Retrieval

NNall - nearest neighbors to any member of the
query set Nnmean - nearest neighbors to the mean
of the query set BO - Behold Search online,
www.beholdsearch.com A Yavlinsky, E
Schofield and S Rüger (CIVR, 2005)
http//www.inference.phy.cam.ac.uk/vr237/
43
(No Transcript)
44
Conclusions

• Given a query of a small set of items, Bayesian
  Sets finds additional items that belong in this
  set
• The score used for ranking items is based on the
  marginal likelihood of a probabilistic model
• For binary data, the score can be computed
  exactly and efficiently using sparse matrices
  (e.g. 1 sec for over 2 million non-zero entries)
• This approach can be extended to many
  probabilistic models and other forms of data
• Where applicable, results competitive with Google
  Sets
• Google Sets works well for lists that appear
  explicitly on the web
• Bayesian Sets works well for finding more
  abstract set completions
• We have built prototype movie, author, paper,
  image and protein search systems

45
Appendix
46
Image features
Texture features (75) We represented images
using two types of texture features, 48 Gabor
texture features and 27 Tamura texture features.
We computed coarseness, contrast and
directionality Tamura features, for each of 9
(3x3) tiles. We applied 6 scale sensitive and 4
orientation sensitive Gabor filters to each image
point and compute the mean and standard deviation
of the resulting distribution of filter
responses. Color features (165) Computed HSV
3D histogram with 8 bins for H and 5 each for
value and saturation. The lowest value bin was
not partitioned into hues since these are hard to
distinguish. Binarization Each feature was
binarized by computing the skewness of the
distribution of that feature and giving a binary
value of 1 to images falling in the 20 percentile
of the heavier tail of the feature distribution.