Automated Image Annotation Using Global Features and Robust Nonparametric Density Estimation

1
Automated Image Annotation Using GlobalFeatures
and Robust Nonparametric DensityEstimation

• Alexei Yavlinsky, Edward Schoeld and Stefan Ruger
• CIVR 2005

2
A simple framework for image annotation

• We want a representation for which the densities
  are as separable as possible for different
  annotation classes w
• Representations are dense enough for reliable
  inference from a small sample of images for each
  class

3
A simple framework for image annotation

• One method of inference is to specify a
  parametric form a priori for the true
  distributions of image features for the
  annotation class w
• Another method is to encode all our knowledge
  about the true distribution
• A third method is to adopt a nonparametric
  estimator of the true density

4
Nonparametric Density Estimation

• x is a vector x1xd
• The positive scalar h, called the bandwidth,
  reflects how wide a kernel is placed over each
  data point

5
Nonparametric Density Estimation

• d-dimensional Gaussian kernel
• Friedman et al. point out that kernel smoothing
  may become less effective in high-dimensional
  spaces

6
Earth Mover's Distance (EMD)

• A signature is a representation of clustered data
  dened as
• for a cluster i, ci is the cluster's centroid and
  mi is the number of points belonging to that
  cluster or its mass
• Given two such signatures, EMD is dened as the
  minimum amount of work necessary to transform one
  signature into the other

7
Nonparametric Density Estimation

• Use EMD for density estimation

8
Bayesian Image Annotation

• Model 1
• Model 2

9
Image Features

• Global Features
• The distribution of pixel colour in CIE space
• texture features proposed by Tamura
• 24-dimensional feature vector combining colour
  and texture.
• Locally Sensitive Features
• 9 equal rectangular tiles
• 108-dimensional feature vector

10
Image Features

• Image Signatures
• We used colour-only signatures for EMD
  computations
• applying simple k-means clustering to pixels in
  CIELab space and setting k to 16

11
Performance Evaluation

• The Corel Dataset
• 5,000 images, each image was assigned 15
  keywords from a vocabulary of 371 words
• 4,500 training images and 500 test images
• To optimise the kernel bandwidth parameters we
  divide the training set into 3800 training images
  and 700 images on which different bandwidth
  settings are evaluated

12
Performance Evaluation

• The Getty Dataset
• 7,560 images, 5000 training and 2560 test images
• Keywords for Getty images come in three different
  flavours subjects, concepts and styles. We use
  subject keywords only
• We restricted the range of keywords occur in
  fewer than 10 of the images and occur more than
  50 times
• We pruned specic locations, verbs and abstract
  nouns

13
Image Annotation
14
Ranked Retrieval

• For the Corel dataset all 1, 2 and 3 word queries
  were generated that would yield at least 2
  relevant images in the test set
• For the Getty dataset we required at least 6
  relevant images for any given query set), and
  generated all possible 14 word queries under
  this constraint

15
Ranked Retrieval (Corel)
16
Ranked Retrieval (Getty)
17
Kernel bandwidth optimisation
18
Conclusions and Future Work

• Presented a simple framework for automated image
  annotation based on nonparametric density
  estimation
• under this framework very simple global image
  properties can yield reasonable annotation
  accuracies
• We look forward to exploring global image
  features outside the colour domain