Implementation%20of%20a%20Visual%20Attention%20Model

1
Implementation of a Visual Attention Model

• Based on Itti, Koch and Nieburs
• A Model of Saliency-Based Visual Attention for
  Rapid Scene Analysis IEEE PAMI 1998

2
Overview

• Review of last presentation
• Details about individual steps
• Preprocessing
• Feature Maps
• Saliency Map
• Shifting Attention
• Analysis of the model and performance

3
Review

• Modelling the path of the focus of attention (FOA)

4
Review
5
Preprocessing

• Original image with red, green, blue channels
• Intensity as I (r g b)/3
• Broadly tuned color channels
• R r - (g b)/2
• G g - (r b)/2
• B b - (r g)/2
• Y (r g)/2 - r g/2 - b

6
Preprocessing
r, g, b
R, G, B, Y
Itti, Models of Bottom-Up and Top-Down Visual
Attention 2000
7
Preprocessing
Intensity
R
G
B
Y
8
Multi resolution Pyramids

• Repeated low-pass filtering
• W is the convolution kernel (Gaussian shape, s
  not stated)

G3 32 x 32
G2 64 x 64
G1 128 x 128
G0 256 x 256
9
Multi resolution Pyramids

• Achieve centre-surround difference through
  across-scale difference
• Denoted by Q
• Performed by interpolating courser scale
• Create one pyramid for each channelI(s), R(s),
  G(s), B(s), Y(s)where s Î 0..8 is the scale

10
Intensity Feature Maps

• I(c, s) I(c) Q I(s)
• c Î 2, 3, 4
• s c d where d Î 3, 4
• So I(2, 5) I(2) Q I(5) I(2, 6)
  I(2) Q I(6) I(3, 6) I(3) Q I(6)
  
• ? 6 Feature Maps

11
Colour Feature Maps

• Similar to double-opponent cells (Prim. V. C)
• Red-Green and Yellow-Blue
• RG(c, s) (R(c) - G(c)) Q (G(s) - R(s))
• BY(c, s) (B(c) - Y(c)) Q (Y(s) - B(s))
• Same c and s as with intensity

R-G
G-R
B-Y
Y-B
R-G
G-R
B-Y
Y-B
12
Orientation Feature Maps

• Create Gabor pyramids for q 0º, 45º, 90º,
  135º
• c and s again similar to intensity

13
Normalization Operator

• Promotes maps with few strong peaks
• Surpresses maps with many comparable peaks
• Normalization of map to range 0M
• Find all local maxima
• Find average m of all local maxima without the
  global maximum M
• Multiply the map by (M m)2

14
Normalization Operator
15
Conspicuity Maps
16
Saliency Map

• Average all conspicuity maps

17
Shifting Attention
18
Neural layers
S

• Saliency Map (SM) modeled as layer of leaky
  integrate-and-fire neurons
• SM feeds into winner-take-all (WTA) neural
  network
• Inhibition of Return as transient inhibition of
  SM at FOA(can have DOG shape)

SM
-
Inhibition of Return

WTA
FOA shifted to position of winner
19
Example
a Salient input location
b Location with half the saliency of a
Itti, Models of Bottom-Up and Top-Down Visual
Attention 2000
20
Analysis

• Perform analysis on multiple images
• Magazine covers, advertisements
• Try to find images where method fails
• If time permits
• Compare multiscale method to maintaining
  resolution but increasing variance of Gaussian
  (no interpolation)
• Compare original method to method without
  multiscale feature maps

21
Summary

• Model can be broken down into main steps
• Create pyramids for 5 channels of original image
• Determine feature maps then conspicuity maps
• Combine into saliency map (after normalizing)
• Use two layers of neurons to model shifting
  attention
• Plan to evaluate performance
• Study model by modifying parts of implementation
  and comparing results

22
References

• Engel, Zhang and Wandell Colour tuning in human
  visual cortex measured with functional magnetic
  resonance imagingNature, vol. 388, no. 6,637,
  pp. 68-71(July 1997)
• Greenspan, Belongie, Goodman, Perona, Rakshit and
  Anderson Overcomplete Steerable Pyramid Filters
  and Rotation InvarianceProc. IEEE Computer
  Vision and Pattern Recognition, pp. 222-228,
  Seattle Washington (June 1994)
• Itti Models of Bottom-Up and Top-Down Visual
  AttentionPhD Thesis, California Institute of
  Technology, Pasadena California (2000)
• Itti, Koch, and Niebur A Model of
  Saliency-Based Visual Attention for Rapid Scene
  AnalysisIEEE PAMI Vol. 20, No. 11, November
  (1998)
• Itti, Koch Computational Modeling of Visual
  AttentionNature Reviews Neuroscience Vol. 2
  (2001)
• Parkhurst, Law, Niebur Modeling the role of
  salience in the allocation of overt visual
  attentionVision Research 42 (2002)
• Tsotsos, Culhane, Wai, Lai, Davis and Nuflo
  Modelling Visual Attention via Selective Tuning
  Artificial Intelligence, vol. 78, no. 1-2, pp.
  507-545, (Oct. 1995)