Visual Information Systems - PowerPoint PPT Presentation


PPT – Visual Information Systems PowerPoint presentation | free to view - id: 72f279-MThiO


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation

Visual Information Systems


Visual Information Systems visual information retrieval – PowerPoint PPT presentation

Number of Views:74
Avg rating:3.0/5.0
Slides: 24
Provided by: CCSR8


Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Visual Information Systems

Visual Information Systems
  • visual information retrieval

Computational steps for visual retrieval systems
  1. image processing (colour, texture etc)
  2. human perception and computer perception
    (computer vision)
  3. Sensory gap
  4. features definition, extraction
  5. low-level and high-level
  6. content, semantics, and concepts
  7. small scale and large scale
  8. knowledge domain, knowledge elicitation,
    knowledge discovery and management
  9. Similarity measure, learn from feedback, and
    dynamic indexing
  10. Databases and system architecture
  11. Evaluation, not just system performance, but
    insights for the future

VIR and Traditional Database?
  • A traditional SQL database has as its basic
    element data items in a relation
  • select name
  • from employee, project
  • where employee.deptnumber 25 AND
  • project.number 100
  • databases exploit known structures and relations
  • DBMS retrieval is not probabilistic
  • How different from the WWW?
  • And from traditional IR?

VIR and Traditional IR systems?
  • IR systems can be considered the precursors to
  • The basic unit of a IR system is a document and
    the focus is on textual retrieval
  • exact matching - Boolean, text pattern searching
  • inexact matching - probabilistic, vector space,
  • Visual information has its own characteristics
    that traditional IR is incapable to handle

Recap IR Whats IR
  • Motivation
  • the larger the holdings of the archive, the more
    useful it is
  • however, it is harder to find what you want
  • IR is all about finding what you want when what
    you want is buried in a mass of what you dont

from Lesk, http//
Simple IR Model
Boolean Vector
Ranking Clustering Weighting
Stemming Thesaurus Signature
Pre- Processing
Post- Processing
Boolean Vector
Flat Files Inverted Files Signature Files PAT
Stemming Stoplist
Collection Processing
Recap IR Precision and Recall
  • Precision
  • ratio of the number of relevant documents
    retrieved over the total number of documents
  • how much extra stuff did you get?
  • Recall
  • ratio of relevant documents retrieved for a
    given query over the number of relevant documents
    for that query in the database
  • how much did you miss?

Recap IR Text Retrieval
  • The most popular approach is to extract keywords
    from each text document in the database to form
    the indices of the document.
  • The keyword extraction process may be divided
    into three major steps, stopwords removal,
    stemming and word weighting
  • stopwords removal a, an and the.
  • stemming removes the suffix and prefix of each
  • word weighting estimates the weighting of each

Recap IR Text Retrieval
  • Query will go through the same procedure
  • Similarity matching calculated from the
    pre-computed weighting of the matched keywords.
  • All documents with a similarity value higher than
    a certain threshold will be considered as
    relevant documents and returned to the user.
  • These relevant document may be ranked according
    to the similarity values when presenting to the
    user. (Most web search engines do this.)

Visual Information Retrieval-keyword
  • It is difficult for text to capture the
    perceptual saliency of some visual features
  • Pictures cannot speak, but they are stronger than
  • Text is not well suited for modelling perceptual
  • Subjective.
  • What is needed in these cases is the use of a
    more concrete description of visual content, one
    more closely related to human perception, and a
    new way of interaction that fully exploits human
    perception capabilities.

Visual information Retrieval content-based
  • Textual content free text search
  • image content image features, shapes, color,
    textures, spatial relationships
  • Video content motions, image features, scene
    composition, video semantics, audio, etc.

Content-Based Image Retrieval
  • As happens during the maturation process of many
    a discipline, after early successes in a few
    applications, research is now concentrating on
    deeper problems, challenging the hard problems at
    the crossroads of the discipline from which it
    was born (Arnold 2000)
  • computer vision, databases, and information
  • Deeper analysis is needed and semantics is more
    desirable make use of domain knowledge

Domain and Variability
  • A narrow domain has a limited and predictable
    variability in all relevant aspects of its
  • Semantics is well-defined, and unique.
  • A broad domain has an unlimited and unpredictable
    variability in its appearance even for the same
    semantic meaning
  • Semantics is more ambiguous, and partial
  • Need more contextual information

Domain and Variability
  • The notions of broad and narrow domains are
    helpful in characterizing patterns of use, in
    selecting features, and in designing systems.
  • For narrow, specialized image domains, the gap
    between features and their semantic
    interpretation is usually smaller, so
    domain-specific models may help.
  • In a broad image domain, the gap between the
    feature description and the semantic
    interpretation is generally wide
  • the required number of computational variables
    would be enormous.
  • Research issues raised

Research issues
  • How to handle variability?
  • Multiple processors and fusion process?
  • Inference engines?

Domain Knowledge
  • Laws of syntactic (literal) equality and
    similarity define the relation between image
    pixels or image features regardless of its
    physical or perceptual causes.
  • Laws describing the human perception of equality
    and similarity
  • Physical laws describing equality and difference
    of images under differences in sensing and object
    surface properties. The physics of illumination,
    surface reflection, and image formation have a
    general effect on images.
  • Geometric and topological rules describe equality
    and differences of patterns in space.
  • Category-based rules encode the characteristics
    common to class z of the space of all notions Z.
  • Finally, man-made customs or man-related patterns
    introduce rules of culture-based equality and

Difficulties in VIS
  • The sensory gap and the semantic gap

The Semantic Gap
  • A linguistic description is almost always
    contextual, whereas an image may live by itself.
  • associate higher level semantics to data-driven
  • labelling is seldom complete, context sensitive,
    and, in any case, there is a significant fraction
    of requests whose semantics can't be captured by
    labelling alone. Both methods will cover the
    semantic gap only in isolated cases.
  • This works well in narrow domain like I-Browse,
    though it is not the perfect solution

From broad domain to narrow domain
  • The challenge for image search engines on a broad
    domain is to tailor the engine to the narrow
    domain the user has in mind via specification,
    examples, and interaction.

Bridging the Gap
  • New challenges in content-based retrieval are the
    huge amount of objects to search among, the
    incomplete query specification, the incomplete
    image description, and the variability of sensing
    conditions and object states.
  • The aim of content-based retrieval systems must
    be to provide maximum support in bridging the
    semantic gap between the simplicity of available
    visual features and the richness of the user
  • The broader the domain, the more browsing or
    search by association can be the right solution.
    The narrower the domain, the more likely an
    application of domain knowledge will succeed

Video Retrieval
  • There are three major processes to prepare a
    video for retrieval, video segmentation, index
    extraction and keyframe extraction.
  • From another perspective, video retrieval could
    be considered simpler than image retrieval since
    video reveals its objects more easily as the
    points corresponding to one object move together.
  • In addition, video has a linear timeline, as
    important to the narrative structure of video as
    it is in text.

Video Retrieval
  • Video segmentation divides the video into a
    number of segments by detecting the camera
  • Index extraction manual indexing, image analysis
    and computer vision and object recognition
  • Keyframe extraction is to select representative
    image frames from each video segment to represent
    the segment. These keyframes may be used for
    browsing and for presentation.