What Are the High-Level Concepts with Small Semantic Gaps? - PowerPoint PPT Presentation

Loading...

PPT – What Are the High-Level Concepts with Small Semantic Gaps? PowerPoint presentation | free to download - id: 522e03-NWFlN



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

What Are the High-Level Concepts with Small Semantic Gaps?

Description:

What Are the High-Level Concepts with Small Semantic Gaps? CS 4763 Multimedia System, Spring 2008 Outline Introduction LCSS: a lexicon of high-level concepts with ... – PowerPoint PPT presentation

Number of Views:33
Avg rating:3.0/5.0
Slides: 26
Provided by: Yiju4
Learn more at: http://www.cs.utsa.edu
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: What Are the High-Level Concepts with Small Semantic Gaps?


1
What Are the High-Level Concepts with Small
Semantic Gaps?
  • CS 4763 Multimedia System, Spring 2008

2
Outline
  • Introduction
  • LCSS a lexicon of high-level concepts with small
    semantic gaps.
  • Framework of LCSS Construction
  • Methodologies
  • Experimental results
  • Conclusion

3
Introduction
  • Recent years have witnessed a fast development of
    Multimedia Information Retrieval (MIR) .
  • Semantic gap is still a fundamental barrier.
  • --Difference between the expressing power or
    descriptions of low-level features and high-level
    semantic concepts.

4
Introduction
  • To reduce the semantic gap, concept-based
    multimedia search has been introduced.
  • Select a concept lexicon relatively easy for
    computers to understand
  • Collect training data
  • Model high-level semantic concepts

5
Problem
  • Concept Lexicon selection is usually simplified
    by manual selection or totally ignored in
    previous works.
  • i.e. Caltech 101, Caltech 256, PASCAL
  • --Implicitly favored those relatively easy
    concepts
  • MediaMill challenge concept data (101 terms) and
    Large-Scale Concept Ontology for Multimedia
    (LSCOM) (1,000 concepts)
  • -- Manually annotated concept lexicon on
    broadcast news video from the TRECVID benchmark
  • All the lexica ignore the differences of semantic
    gaps among concepts.
  • No automatic selection is executed.

6
Problem
  • Concepts with smaller semantic gaps are likely to
    be better modeled and retrieved than concepts
    with larger ones.
  • Very little research is found on quantitative
    analysis of semantic gap.
  • What are the well-defined semantic concepts for
    learning?
  • How to automatically find them?

7
Objective
  • Automatically construct a lexicon of high-level
    concepts with small semantic gap (LCSS).
  • Two properties for LCSS
  • 1) Concepts are commonly used.
  • --The words have high occurrence frequency
    within the
  • descriptions of real-world images.
  • 2) Concepts are expected to be visually and
    semantically
  • consistent.
  • -- Images of these concepts have smaller
    semantic gaps
  • (easy to be modeled for retrieval and
    annotation)

8
Idea
  • Web images have rich textual features
  • --filename, title, alt text, and surrounding
    text.
  • Input titles and comments are good semantic
    descriptions of the images
  • These textual features are much closer to the
    semantics of the images than visual features.

9
Framework of LCSS Construction
10
Data Collection
  • 2.4 million web images from photo forums
  • --Photo.net, Photosig.com etc
  • Photos have high quality and rich textual
    information
  • 64 dimensional global visual feature
  • --color moments, color correlogram and
    color-texture moments

11
Framework of LCSS Construction
12
Nearest Neighbor Confidence Score (NNCS)
  • The higher the NNCS value, the smaller the
    semantic gap.
  • Calculate NNCS Score for all 2.4 million images
    with K500
  • Select 36231 candidate images with top NNCS
  • --its relatively large size and memory
    concern for the Affinity Propagation clustering
    algorithm.

13
Framework of LCSS Construction
14
Clustering Using Affinity Propagation
  • Fast for large scale data set and require no
    prior information (e.g., number of clusters).
  • 3623136231 content-context similarity matrix
    (CCSM) P

15
Framework of LCSS Construction
16
Text-based Keyword Extraction (TBKE)
  • The set of the related keywords of cluster
    is denoted as .
  • The relevance score of a keyword to cluster
    is denoted as
  • For each keyword , its relevance score to the
    whole cluster pool C can be denoted as

17
  • LCSS

Table 1. Top 50 keywords in the LCSS lexicon
18
Experiments
  • For each keyword w, randomly selected 500 titled
    photos with this keyword.
  • The confidence value decreases similarly to the
    keyword ranks depreciation.
  • Demonstrates that the image labeled with the top
    words have higher confidence value.

Figure 1 Distribution of average confidence
value. x-axis represents top 15 keywords (from
left to right) in LCSS.
19
Experiments
  • Apply the lexicon on the University of Washington
    (UW) dataset
  • (1109 images, 5 labeled ground truth
    annotations, 350 unique words)
  • Refine the annotation results obtained by the
    search-based image annotation algorithm (SBIA).

Figure 2 Image annotation refinement scenario .
20
Experiments
Figure 3 Annotation precision and recall of
different sizes of lexicon .
  • The refinement distinctively improves original
    annotations precision and recall when s becomes
    larger.
  • The performance of refinement keeps stable when s
    is equal or larger than 100.

21
Experiments
Figure 4 Annotation precision and recall of
different sizes of m .
  • When m is ranging from 3 to 7, the Precision and
    Recall of refined annotation are improved most .
  • The Precision and Recall of the annotation
    pruning remains same especially while m is larger
    than 7.
  • --most of top 7 annotation words fall into
    the LCSS.

22
Experiments
Figure 4 Annotation precision and recall of
different lexica.
  • LSCOM Large Scale Concepts Ontology for
    Multimedia.
  • WordNet A very large lexical database of
    English. (100,303)
  • LSCOM and WordNet do not improve the annotation
    precision.
  • --Many correct annotations are not included
    in these two
  • lexica thus are pruned.

23
Conclusion
  • Quantitatively study and formulate the semantic
    gap problem.
  • Propose a novel framework to automatically select
    visually and semantically consistent concepts.
  • LCSS is the first lexicon for concepts with small
    semantic gap
  • great help for data collection and concept
    modeling.
  • a candidate pool of semantic concepts
  • --image annotation, annotation refinement and
    rejection.
  • Potential applications in query optimization and
    MIR.

24
Future Work
  • Investigate different features to construct
    feature-based lexica
  • Texture feature
  • Shape feature
  • SIFT feature
  • Open questions
  • How many semantic concepts are necessary?
  • Which features are good for image retrieval with
    specific concept?

25
Reference
  • C. G. Snoek, M. Worring, J. C. van Gemert, J. M.
    Geusebroek, and A. W. Smeulders. The challenge
    problem for automated detection of 101 semantic
    concepts in multimedia. Proc. of ACM
    Multimedia, 2006.
  • C. M. Naphade, J. R. Smith, J. Tesic, and S. F.
    Chang, et al. Large-scale concept ontology for
    multimedia. IEEE MultiMedia, 13(3)8691, 2006.
  • B. J. Frey, and D. Dueck. Clustering by passing
    messages between data points. Science,
    315972-976, 2007.
  • X. J. Wang, L. Zhang, F. Jing, and W. Y. Ma.
    AnnoSearch image auto-annotation by search.
    Proc. of IEEE Conf. CVPR, New York, June, 2006.
  • C. Wang, F. Jing, L. Zhang, and H. J. Zhang.
    Scalable search-based image annotation of
    personal images. Proc. of the 8th ACM
    international Workshop on Multimedia information
    Retrieval, Santa Barbara, CA, USA, 2006. 10.
About PowerShow.com