Content-Based Image Retrieval: Reading One

1
Content-Based Image Retrieval Reading Ones Mind
and Making People Share

• Oral defense by Sia Ka Cheung
• Supervisor Prof. Irwin King
• 31 July 2003

2
Flow of Presentation

• Content-Based Image Retrieval
• Reading Ones Mind
• Relevance Feedback Based on Parameter Estimation
  of Target Distribution
• Making People Share
• P2P Information Retrieval
• DIStributed COntent-based Visual Information
  Retrieval

3
Content-Based Image Retrieval

• How to represent and retrieve images?
• By annotation (manual)
• Text retrieval
• Semantic level (good for picture with people,
  architectures)
• By the content (automatic)
• Color, texture, shape
• Vague description of picture (good for pictures
  of scenery and with pattern and texture)

4
Feature Extraction
5
Indexing and Retrieval

• Images are represented as high dimensional data
  points (feature vector)
• Similar images are close in the feature vector
  space
• Euclidean distance is used

6
Typical Flow of CBIR
Images
7
Reading Ones Mind

• Relevance Feedback

8
Why Relevance Feedback?

• The gap between semantic meaning and low-level
  feature ? the retrieved results are not good
  enough

DatabaseIndex and Storage
Feature Extraction
Lookup
Result
9
(No Transcript)
10
Problem Statement

• Assumption images of the same semantic
  meaning/category form a cluster in feature vector
  space
• Given a set of positive examples, learn users
  preference and find better result in the next
  iteration

11
Former Approaches

• Multimedia Analysis and Retrieval System (MARS)
• IEEE Trans CSVT 1998
• Weight updating, modification of distance
  function
• Pic-Hunter
• IEEE Trans IP 2000
• Probability based, updated by Bayes rule
• Maximum Entropy Display

12
Comparisons
Aspect Model Description
Modeling of users target MARS Weighted Euclidean distance
Modeling of users target Pic-Hunter Probability associated with each image
Modeling of users target Our approach Users target data point follow Gaussian distribution
Learning method MARS Weight updating, modification of distance function
Learning method Pic-Hunter Bayes rule
Learning method Our approach Parameter estimation
Display selection MARS K-NN neighborhood search
Display selection Pic-Hunter Maximum entropy principle
Display selection Our approach Simulated maximum entropy principle
13
Estimation of Target Distribution

• Assume the users target follows a Gaussian
  distribution
• Construct a distribution that best fits the
  relevant data points into some specific region

14
Estimation of Target Distribution

• Assume the users target follows a Gaussian
  distribution
• Construct a distribution that best fits the
  relevant data points into some specific region

15
Estimation of Target Distribution

• Assume the users target follows a Gaussian
  distribution
• Construct a distribution that best fits the
  relevant data points into some specific region

16
Expectation Function

• Best fit the relevant data points to medium
  likelihood region
• The estimated distribution represents users
  target

17
Updating Parameters

• After each feedback loop, parameters are updated
• New estimated mean mean of relevant data points
• New estimated variance ? found by differentiation
• Iterative approach

18
Display Selection

• Why maximum entropy principle?
• K-NN is not a good way to learn users preference
• The novelty of result set is increased, thus
  allowing user to browse more from the DB
• How to use maximum entropy?
• PicHunter Select a subset of images which
  entropy is maximized.
• Our approach data points inside boundary region
  (medium likelihood) are selected

19
Simulating Maximum Entropy Display

• Data points around the region of 1.18 d away from
  µ are selected
• Why 1.18?
• 2P(µ1.18 d)P(µ)

P(µ)
P(µ1.18 d)
20
Experiments

• Synthetic data forming mixture of Gaussians are
  generated
• Feedbacks are generated based on ground truth
  (class membership of synthetic data)
• Investigation
• Does the estimated parameters converge?
• Does it performs better?

Dimension No. of class No. of data points in each class Range of µ Range of d
4 50 50 -1,1 0.2,0.6
6 70 50 -1.5,1.5 0.2,0.6
8 85 50 -1.5,1.5 0.15,0.45
21
Convergence of Estimated Parameters

• More feedbacks are given, estimated parameters
  converge to original parameters used to generate
  mixtures

22
Precision-Recall

• Red PE
• Blue MARS
• More experiments in later section

23
Precision-Recall
24
Problems

• What if users target distribution forms several
  cluster?
• Indicated in Qcluster (SIGMOD03)
• Parameters estimation failed because single
  cluster is the assumption
• Qcluster solve it by using multi-points query
• Merge different clusters into one cluster !!

25
The Use of Inter-Query Feedback

• Relevance feedback information given by users in
  each query process often infer a similar semantic
  meaning (images under the same category)
• Feature vector space can be re-organized
• Relevant images are moved towards to the
  estimated target
• Similar images no longer span on different
  clusters
• Parameters estimation method can be improved

26
1st Stage of SOM Training

• Large number of data points
• ? SOM is used to reduce data size
• ? Each neuron represent a group of similar images
• ? original feature space is not changed directly

27
Procedure of Inter-query Feedback Updating

• User marked a set of images as relevant or
  non-relevant in a particular retrieval process
• The corresponding relevant neurons are moved
  towards estimated target
• Where
• MR set of relevant neurons
• c estimated target
• aR learning rate
• The corresponding non-relevant neurons are moved
  away from estimated target

28
SOM-based Approach
Neuron Class 1
Neuron Class 2
Neuron Class 3
29
SOM-based Approach

• After each query process

Relevant Neuron
Non- Relevant Neuron
30
SOM-based Approach
Estimated Target
31
SOM-based Approach

• Relevant neurons are moved towards estimated
  target

32
SOM-based Approach
33
SOM-based Approach

• Feature vector space re-organized

34
SOM-based Approach

• After several iterations (users queries)

35
SOM-based Approach
36
SOM-based Approach

• Similar images cluster together instead of
  spanning across different clusters in the new,
  re-organized feature vector space

37
Experiments

• Real data from Corel image collection
• 4000 images from 40 different categories
• Feature extraction methods
• RGB color moment (9-d)
• Grey scale cooccurence matrix (20-d)
• 80 queries are generated evenly among 40 classes
• Evaluations
• MARS
• PE without SOM-based inter-query feedback
  training
• PE with SOM-based inter-query feedback training

38
Precision vs Recall
39
Conclusion

• We propose a parameters estimation approach for
  capturing users target as a distribution
• A display set selection scheme similar to maximum
  entropy display is used to capture more users
  feedback information
• A SOM-based inter-query feedback is proposed
• Overcome the single cluster assumption of most
  intra-query feedback approach

40
Making People Share

• DIStributed COntent-based Visual Information
  Retrieval

41
P2P Information Retrieval
Images

Feature Extraction
Peer databases
Lookup
Query Image
Query Result
42
Contributions

• Migrate centralized architecture of CBIR to
  distribution architecture
• Improve existing query scheme in P2P applications
• A novel algorithm for efficient information
  retrieval over P2P
• Peer Clustering
• Firework Query Model (FQM)

43
Existing P2P Architecture

• Centralized
• Napster, SETI (Berkeley), ezPeer (Taiwan)
• Easy implementation
• Bottleneck, single point failure
• Legal problems

update
answer
query
transfer
44
Existing P2P Architecture

• Decentralized Unstructured
• Gnutella (AOL, Nullsoft), Freenet (Europe)
• Self-evolving, robust
• Query flooding

Peer
TCP connection
45
Existing P2P Architecture

• Decentralized Structured
• Chord (SIGCOMM01), CAN(SIGCOMM01), Tapestry
  (Berkeley)
• Efficient retrieval and robust
• Penalty in join and leave

Files shared by peers
Distributed Hash Table (DHT)
CAN model
TCP connection
Peer in the network
46
DISCOVIR Approach

• Decentralized Quasi-structured
• DISCOVIR (CUHK)
• Self-organized, clustered, efficient retrieval

attractive connections random connections
47
Design Goal and Algorithms used in DISCOVIR

• Peers sharing similar images are interconnected
• Reduce flooding of query message
• Construction of self-organizing network
• Signatures calculation
• Neighborhood discovery
• Attractive connections establishment
• Content-based query routing
• Route selection
• Shared file lookup

48
Construction of Self-Organizing Network

• Signatures calculation
• Signatures discovery of neighborhoods
• Comparison of signatures
• Attractive connection establishment

49
Signatures Calculation
Feature vector space
50
Signatures Calculation
Centroid of peer
Peer B
Peer A
51
Signatures Calculation
52
Signatures Calculation
Centriod of sub-cluster
Centroid of peer
Peer B
A1
A2
Peer A
B2
B3
B1
A3
53
Random connection
(2.0,5.8)
Attractive connection
6
(4.9,9.7)
(x,y)
Signature value
7
5
(2.7,6.0)
(2.6,5.9)
4
(4.7,9.3)
10
1
(5.2,8.5)
3
(1.5,1.2)
(1.3,2.4)
2
8
(1.6,1.8)
9
(5.6,8.8)
Existing clustered P2P network
11
(2.9,6.5)
54
Content-based Query Routing
Incoming query
Signaturevalue left?
N
Y
Similarity lt threshold?
Y
N
Forward toattractive link
Forward to random link if not forwarded before
End
55
Content-based Query Routing
Similarity lt threshold
Random connection
Attractive connection
Similarity gt threshold
56
Comparison of Content-based Query Routing and
Address-based Query Routing
Aspect Scheme Description
Application ABR Internet Protocol (IP), Domain Name System (DNS)
Application CBR Wide Area Information System (WAIS), DISCOVIR
Problem ABR We know where to go, but not the path
Problem CBR We dont know where to go, nor the path
Emphasis ABR Correctness, speed
Emphasis CBR Relevance of result retrieved
Goal ABR Avoid unnecessary traffic
Goal CBR Avoid unnecessary traffic
57
Experiments

• Dataset
• RBG color moment, 9-d 10000 images from Corel
  database, 100 classes
• Synthetic data, 9-d, 10000 points, 100 classes
• Operation
• Distribute data-points into peers (1 class per
  peer)
• Simulate network setup and query (averaged 50
  queries)
• Investigation
• Scalability (against number of peers)
• Property (against TTL of query message)
• Data resolution (different number of signatures
  per peer)

58
Network Model

• Small world characteristic, power-law
  distribution
• Few peers are connected with many peers
• Many peers are connected with few peers

59
Performance Metrics

• Relevance of retrieved result
• Recall
• Number of query traffic generated
• Query scope
• Effectiveness of query routing scheme
• Query efficiency

Number of retrieved relevant result Total number
of relevant result
Number peers visited by query message Total
number of peers
60
Recall vs Peers
61
Recall vs TTL
62
Query Scope vs Peers
63
Query Scope vs TTL
64
Query Efficiency vs Peers
65
Query Efficiency vs TTL
66
Difference Between Synthetic Data and Real Data
Inter-cluster distance Inter-cluster distance Inter-cluster distance Mean of variances Mean of variances Mean of variances
real synthetic real synthetic
max 1.4467 1.8207 max 0.0153 0.0128
min 0.0272 0.3556 min 0.0006 0.0042
avg 0.3298 1.1159 avg 0.0112 0.0086
67
Effects of Data Resolution

• Assign 2-4 classes of image to each peer
• High data resolution (use 3 signatures)
• Low data resolution (use 1 signatures)

68
Conclusion

• CBIR is migrated from centralized server approach
  to peer-to-peer architecture
• Efficient retrieval is achieved by
• Constructing a self-organizing network
• Content-based query routing
• Scalability, property and effects on data
  resolution are investigated
• Query efficiency are at least doubled under the
  proposed architecture

69
Questions and Answers
70
Precision, Recall, Novelty

• Precision ?range 0,1
• Recall ?range 0,1

Set of retrieved result
Set of relevant result
Set of retrieved knownto user before hand
71
Update Equation, Learning Rate

• Update equation of non-relevant neurons
• Update equation of neighboring neurons

72
Original SOM M
Modified SOM M
1-1 mappingfunction f -1
Table lookup
Retrieval process with SOM to capture feedback
information
Modified model vector spacedata-size reduced to
M
Original feature vector spacedata-size is I
73
Multiple Clusters Version
74
DISCOVIR System Architecture

• Built on LimeWire, Java-based
• Plug-in architecture for feature extraction
  module
• Query by example, sketch, thumbnail previewing

75
DISCOVIR Screen Capture
76
DISCOVIR-Protocol Modification
DISCOVIR Signature Query 0x80
Minimum Speed
DISCOVIRSIGNATURE
0
0
1
2
20
0
DISCOVIR Signature Query Hit 0x81
Number of Hits
Port
IP Address
Speed
Result Set
Servant Identifier
0
1
2
3
6
7
10
11

n
n16
Dummy
Dummy
Feature Extraction name
Signature value
0
0
0
3
4
7
8
Image Query 0x80
Minimum Speed
Feature Name
0
Feature Vector
0
1
2
0
Image Query Hit 0x81
Number of Hits
Port
IP Address
Speed
Result Set
Servant Identifier
1
2
3
0
6
7
10
11

n
n16
File Index
File Size
File Name
Thumbnail information, similarity
0
0
0
3
4
7
8
77
Query Message Utilization
78
Average Reply Path Length