Mutual reinforcement principle and applications

1
Mutual reinforcement principle
and applications

• Dhiraj Joshi
• Presentation for IST 597 C
• The Pennsylvania State University
• http//wang.ist.psu.edu

2
Outline

• Introduction and motivation
• Hubs and authorities Jon Kleinberg
• Google page rank Brin and Page
• Story picturing engine Joshi, Wang, and Li

3
Introduction and motivation

• Mutual reinforcement ranking scheme
• entities mutually reinforce each other based
  on some similarity
• between them

• has circular definition
• eqn. has iterative solution

4
Introduction and motivation

• Mutual reinforcement ranking scheme
• entities mutually reinforce each other based
  on some similarity
• between them
• Intuitive justification being reinforced by a
  high ranked entity should enhance the rank of
  another entity
• reinforcement could result from
• sharing a hyperlink in a Web-graph 2 web pages
• similar in language content - 2 documents
• similar in visual content - 2 images
• citation in a scientific document

5
Outline

• Introduction and motivation
• Hubs and authorities Jon Kleinberg
• Google page rank Brin and Page
• Story picturing engine Joshi, Wang, and Li

6
Hubs and authorities

• Model of the link structure of the Web
• Web modeled as a directed graph
• nodes web pages
• edges hyper-links between pages
• Web is a very sparse graph
• Basic hypothesis of this model
• If
• p and q are web-pages
• p has a hyperlink to q (p -gt q)
• Then
• p has conferred some authority to q

7
Hubs and authorities

• Model of the link structure of the Web
• Web modeled as a directed graph
• nodes web pages
• edges hyper-links between pages
• Web is a very sparse graph
• Web consists of two types of pages
• AUTHORITIES - pointed to by many pages
• have a high in-degree in
  the directed graph
• HUBS - point to many pages
• have a high out-degree in
  the directed graph

8
Hubs and authorities

• Search algorithm -
• STEP 1 - given a query q build a focused
  sub-graph
• begin with search results of common search
  engines
• follow hyperlinks to discover more pages

9
Hubs and authorities

• Search algorithm -
• STEP 1 - given a query q build a focused
  sub-graph
• STEP 2 - explore hyper-link structure of the
  sub-graph
• find hubs and authorities in sub-graph
• Hubs and Authorities exhibit mutually reinforcing
  relationship
• Good HUB points to many good AUTHORITIES
• Good AUTHORITY pointed to by many good
  HUBS

10
Hubs and authorities

• Search algorithm -
• STEP 1 - given a query q build a focused
  sub-graph
• STEP 2 - explore hyper-link structure of the
  sub-graph
• find hubs and authorities in sub-graph
• Iterative algorithm
• for each page p, maintain 2 weights
• normalize them
• iteratively update them

11
Hubs and authorities

• Search algorithm -
• STEP 1 - given a query q build a focused
  sub-graph
• STEP 2 - explore hyper-link structure of the
  sub-graph
• find hubs and authorities in sub-graph
• Iterative algorithm - graphically

12
Hubs and authorities

• Search algorithm -
• STEP 1 - given a query q build a focused
  sub-graph
• STEP 2 - explore hyper-link structure of the
  sub-graph
• find hubs and authorities in sub-graph
• Iterative algorithm - graphically
• A adjacency matrix of the graph
• updation operations can be represented as
• solution -
• x - principal Eigen-vector of AA
• y - principal Eigen-vector of AA

13
Outline

• Introduction and motivation
• Hubs and authorities Jon Kleinberg
• Google page rank Brin and Page
• Story picturing engine Joshi, Wang, and Li

14
Google page rank

• Model of the link structure of the Web
• Web modeled as a directed graph
• nodes web pages
• edges hyper-links between pages
• Web is a very sparse graph
• Every page is assigned a rank in graph
• Nature of page-rank
• page-rank of pages pointed to by high ranked
  pages should be high
• pages mutually reinforce each others rank
• circular definition

15
Google page rank

• Model of the link structure of the Web
• Web modeled as a directed graph
• nodes web pages
• edges hyper-links between pages
• Web is a very sparse graph
• PR(.) page rank
• C(.) out degree of node
• d damping factor
• Finally PR(.) are normalized to add up to 1

16
Google page rank

• PR(.) page rank
• C(.) out degree of node
• d damping factor
• Finally PR(.) are normalized to add up to 1
• Intuitively
• PR(.) represents the stationary distribution of a
  random walk in the Web-graph
• transitions decided by hyperlinks

17
Google page rank

• PR(.) page rank
• C(.) out degree of node
• d damping factor
• Finally PR(.) are normalized to add up to 1
• Random surfer model
• suppose a random surfer is following hyperlinks
  in the
• web-graph
• PR(A) probability of surfer hitting page A
• (1-d) damping factor to account for surfer
  getting bored

18
Outline

• Introduction and motivation
• Hubs and authorities Jon Kleinberg
• Google page rank Brin and Page
• Story picturing engine Joshi, Wang, and Li

19
The Story Picturing Engine Finding Elite Images
to Illustrate a Story

• Dhiraj Joshi
• Department of Computer Science and Engineering
• James Z. Wang
• School of Information Sciences and Technology
• Jia Li
• Department of Statistics
• The Pennsylvania State University
• http//wang.ist.psu.edu

20
Outline

• Introduction of the problem
• Related work
• Story Picturing Engine
• Experimental results
• Conclusions and future work

21
Story Picturing

• Scenario
• Pictures with manual annotations
• in a database
• Story with possible references to
  objects/events/people in pictures
• Problem - Finding suitable pictures to illustrate
  story

Story Picturing Engine
22
Story Picturing

• Typically performed by humans
• news picturing news readers/writers
• choosing best images to convey news
• documentary filming directors
• choosing best images to display
• educational story picturing teachers
• choosing best images to teach children
• story comics comic book writers
• picture based rendering more appealing

23
Story Picturing

• Why pictures?
• A picture is worth a thousand words
• Why computer intervention?
• amount of data becoming unmanageable
• large scale learning of concepts possible today
• high computational power available
• Why is story picturing challenging?
• human choice is very subjective
• based on past experience, knowledge, prejudices
• computerized content analysis still open problem

24
Story Picturing

• Choose the most representative image

Niagara Falls
25
Outline

• Introduction of the problem
• Related Work
• Story Picturing Engine
• Experimental results
• Conclusions and future Work

26
Related Work

• Computer Graphics domain
• WordsEye ATT Bell Labs
• converts English text into 3-D scenes
• AI for movie animation Chinese Academy of
  Sciences
• automatic text to scene conversion
• Computer Vision domain
• Auto-annotation and illustration, Berkeley
• learning based approach
• Past work of our group
• Automatic Linguistic Indexing of Pictures (ALIP)
• CLUster-based rEtrieval (CLUE)

Rockies snow glacier sky ski
27
Outline

• Introduction of the problem
• Related Work
• Story Picturing Engine
• Experimental results
• Conclusions and future Work

28
The Story Picturing Engine

• We propose -
• unsupervised approach to story picturing
• integration of text and image information
• criteria to quantify image importance
• Outline of the Story Picturing Engine
• Story processing - extracting keywords and noun
  descriptors from story
• Image selection - search database and form a pool
  of pictures
• Choosing elite pictures use pairwise
  similarities for ranking
• We use the Princeton WordNet for text processing

29
The Story Picturing Engine

• Story Processing
• Eliminate stopwords, adjectives and verbs
• Eliminate nouns with high polysemy count
• polysemy count- number of different forms of a
  word
• Identify proper nouns and keywords
• Image Selection
• Form a pool of possible candidates
• select images whose captions contain at least
  one proper noun and any one keyword

30
The Story Picturing Engine

• Choosing elite pictures A qualitative look
• Form pairwise image similarities
• Ranks using mutual reinforcement criteria
• images vote towards the rank
• of each other
• each vote enhances the rank
• of an image
• effect of a vote is determined
• by rank of voting image
• Display high ranked images

31
The Story Picturing Engine

• Choosing elite pictures A quantitative look
• is a collection of images,
• is a similarity measure between two images,
• Define rank of an image as the
• solution of the equation
• Iterative solution rank vector
• converges to the principal eigenvector
• of the image similarity matrix

32
The Story Picturing Engine

• Choice of
• Semantic similarity - similarity in content
• We combine both lexical and visual similarity
• Lexical similarity using WordNet topical
  hierarchy
• Visual similarity using Integrated Region
  Matching (IRM) measure

33
Wordnet - A Lexical Database

• Lexical arrangement of nouns
• inspired by psycho-linguistic theories of human
  lexical memory
• organized as topical hierarchies
• oak-gttree-gtplant-gtorganism
• WordNet definitions
• Synonym words that can be interchangeably used
• Meronym words which have object-part relation
  (beak, bird)
• Hypernym
• parent in lexical hierarchy

Diagram adopted from a paper on WordNet
34
The Story Picturing Engine

• Lexical similarity
• Keyword based similarity using WORDNET
• If and two keywords
• Case1 if and are same or synonyms
• Case 2- if and are meronyms or have
  same hypernym
• Case2 - if appears in s list of
  hypernyms at level t
• Case3 if unrelated
• Similarity defined as follows

35
The Story Picturing Engine

• Visual similarity
• Integrated Region Matching (IRM) measure
• extract wavelet features from images.
• segment images into a number of regions.
• calculate overall region based similarity.
• details can be found in IEEE PAMI 2001 (9)
• IRM distances converted into percentiles
• If is IRM distance between two
  images
• is fraction of all distances
  which are greater than
• Form a linear combination

36
Outline

• Introduction of the problem
• Related Work
• Story Picturing Engine
• Experimental results
• Conclusions and future Work

37
Experimental Setup

• Manually annotated image databases
• Terragalleria database (Q-T. Luong)
• annotated pictures from Luongs travels
• http//www.terragalleria.com
• Art Image Consortium (AMICO) database (J. Trant)
• consortium of museums from all over the world
• http//www.amico.org
• Stories
• Luongs travel stories
• obtained from Luongs Website
• Educational stories from ARTKids
• non-profit educational Website
• http//www.artfaces.com/artkids

Terragalleria pictures
AMICO pictures
38
Experimental Results
Story
Results
highest ranked lowest ranked
39
Experimental Results
Story
Results
highest ranked lowest ranked
40
Experimental Results
Story
Results
highest ranked pictures
41
Experimental Results
Story
Results
highest ranked pictures
42
Experimental Results
Story
Results
Only lexical content used
Both lexical and visual content used
Only visual content used
43
Outline

• Introduction of the problem
• Related Work
• Story Picturing Engine
• Experimental results
• Conclusions and future Work

44
Conclusions and Future Work

• Conclusions
• Introduced the problem of Story Picturing
• Proposed a computational solution
• Promising results have been shown
• Future work
• Integration of several databases
• Integrate Story Picturing Engine with commercial
  text based image search engines
• An online interface for Story Picturing Engine
• Providing real time solutions
• Comprehensive evaluation of the system
• Acknowledgments NSF ITR/Career/RI
• More details at http//wang.ist.psu.edu