CH 14 Multimedia IR - PowerPoint PPT Presentation

Loading...

PPT – CH 14 Multimedia IR PowerPoint presentation | free to download - id: 136c9-MzBkZ



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

CH 14 Multimedia IR

Description:

More suitable for modeling both multimedia data types and their semantic relationships ... GEMINI can be successfully applied to time series, and specifically ... – PowerPoint PPT presentation

Number of Views:398
Avg rating:5.0/5.0
Slides: 61
Provided by: deni2
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: CH 14 Multimedia IR


1
CH 14 Multimedia IR
2
Multimedia IR system
  • The architecture of a Multimedia IR system
    depends on two main factors
  • The peculiar characteristics of multimedia data
  • The kinds of operations to be performed on such
    data

3
Multimedia IR system
  • Support variety of data
  • Different kinds of media
  • Text, images (both still and moving), graphs, and
    sound
  • Mix of structured and unstructured data
  • Metadata
  • Semi-structured data
  • Data whose structure may not match, or only
    partially match, the structure prescribed by the
    data schema
  • The system must typically extract some features
    from the multimedia objects

4
Multimedia IR system
  • Data retrieval
  • Exploiting data attributes and the content of
    multimedia objects
  • Basic steps
  • Query specification
  • Fuzzy predicates, content-based predicates,
    object attributes, structural predicates
  • Query processing and optimization
  • Query is parsed and compiled into an internal
    form
  • Query answer
  • Query iteration

5
Multimedia IR system
  • Combine DBMS and IR technology
  • DBMS data modeling capabilities
  • IR system similarity-based query capabilities

6
Data modeling
  • Main tasks
  • A data model should be defined by which the user
    can specify the data to be stored into the system
  • Support conventional and multimedia data types
  • Provide methods to analyze, retrieve, and query
    such data
  • Provide a model for the internal representation
    of multimedia data

7
Object-oriented DBMS
  • Provide rich data model
  • More suitable for modeling both multimedia data
    types and their semantic relationships
  • Class
  • Attributes operations
  • Inheritance
  • Drawback
  • the performances of storage techniques, query
    processing, and transaction management is not
    comparable to that of relational DBMSs
  • Highly non-standard

8
Object-relational DBMS
  • Extend the relational model
  • Represent complex data types
  • Maintain the performance and the simplicity of
    relational DBMSs and related query languages
  • Define abstract data types
  • Allows one to define ad hoc data types for
    multimedia data

9
Internal representation
  • Using attributes is not sufficient
  • Feature
  • Information extracted from objects
  • Multimedia object is represented as a set of
    features
  • Features can be assigned manually, automatically,
    or using a hybrid approach

10
Internal representation
  • Values of some specific features are assigned to
    a object by comparing the object with some
    previously classified objects
  • Feature extraction cannot be precise
  • A weight is usually assigned to each feature
    value representing the uncertainty of assigning
    such a value to that feature
  • 80 sure that a shape is a square

11
SQL3
  • Support extensible type system
  • Provide constructs to define user-dependent
    abstract data types, in an object-oriented like
    manner
  • Collection data types
  • Sets, multisets, and lists
  • The elements of a collection must have compatible
    types

12
MULTOS
  • MULTimedia Office Server
  • Client/server
  • Support filing and retrieval of multimedia
    objects
  • Each document is described by a logical
    structure, a layout structure, and a conceptual
    structure
  • Documents having similar conceptual structures
    are grouped into conceptual types

13
MULTOS
  • Conceptual types are maintained in a hierarchy of
    generalization
  • Strong type
  • Completely specifies the structure of its
    instances
  • Weak type
  • Partially specifies the structure of its
    instances
  • Components of unspecified type (called spring
    component types) can appear in a document
    definition

14
spring component type
Conceptual structure of the type Generic_Letter
15
Complete conceptual structure of the type
Business_Product_Letter
16
Query languages
  • Relational/object-oriented database system
  • Exact match of the values of attributes
  • Multimedia IR system
  • Similarity-based approach (exact match)
  • Considers the structure and the content of the
    objects
  • Content-based query
  • Retrieve multimedia objects depending on their
    globe content

17
Query languages
  • In designing a multimedia query language, three
    main aspects require attention
  • How the user enters his/her request to the system
  • Which conditions on multimedia objects can be
    specified in the user request
  • How uncertainty, proximity, and weights impact
    the design of the query language

18
Request specification
  • Interfaces
  • Browsing and navigation
  • Specifying the conditions the objects of interest
    must satisfy, by means of queries
  • Queries can be specified in two different ways
  • Using a specific query language
  • Query by example
  • Using actual data (object example)

19
Conditions on multimedia data
  • Query predicates
  • Attribute predicates
  • Concern the attributes for which an exact value
    is supplied for each object
  • Exact-match retrieval
  • Structural predicates
  • Concern the structure of multimedia objects
  • Can be answered by metadata and information about
    the database schema
  • Find all multimedia objects containing at least
    one image and a video clip

20
Conditions on multimedia data
  • Semantic predicates
  • Concern the semantic content of the required
    data, depending on the features that have been
    extracted and stored for each multimedia object
  • Find all the red houses
  • Exact match cannot be applied

21
Uncertainty, proximity, and weights in query
expressions
  • Specify the degree of relevance of the retrieved
    objects
  • Using some imprecise terms and predicates
  • Represent a set of possible acceptable values
    with respect to which the attribute or he
    features has to be matched
  • Normal, unacceptable, typical
  • Particular proximity predicates
  • The relationship represented is based on the
    computation of a semantic distance between the
    query object and stored ones
  • Nearest object search

22
Uncertainty, proximity, and weights in query
expressions
  • Assign each condition or term a given weight
  • Specify the degree of precision by which a
    condition must be verified by an object
  • Find all the objects containing an image
    representing a screen (HIGH) and a keyboard
    (LOW)
  • The corresponding query is executed by assigning
    some importance and preference values to each
    predicate and term

23
SQL3 query language
  • Major improvements of SQL3
  • Functions and stored procedures
  • Allow users to integrate external functionalities
    with data manipulation
  • Active database facilities
  • Support active rules
  • The database is able to react to some system- or
    user-dependant events by executing specific
    actions
  • Limitation
  • No IR techniques are integrated into the SQL3
    query processor

24
MULTOS query language
  • General form
  • FIND DOCUMENTS VERSION version-clause
  • SCOPE scope-clause
  • TYPE type-clause
  • WHERE condition-clause
  • WITH component

25
MULTOS query language
  • Three main classes of predicates
  • Data attributes
  • Textual components
  • Images
  • The class to which an image should belong
  • Existence and the number of occurrences of an
    object within an image
  • Support imprecise query
  • Associating a preference and an importance value
    with the attributes in the query

26
MULTOS query language
  • FIND DOCUMENTS VERSION LAST WHERE
  • Document.Date 1/1/1998 AND
  • (Sender.Name Olivetti OR
  • Product_Presentation CONTAINS Olivetti) AND
  • Product_Description CONTAINS Personal Computer
    AND
  • (Address.Country Italy OR
  • TEXT CONTAINS Italy) AND
  • WITH Company_Logo

27
MULTOS query language
  • FIND DOCUMENTS VERSION LAST WHERE
  • (Document.Date BETWEEN (12/31/1998,1/31/98)
    PREFERRED BETWEEN (2/1/1998,2/15/98) ACCEPTABLE)
    HIGH AND
  • (Sender.Name Olivetti OR
  • Product_Presentation CONTAINS Olivetti) HIGH
    AND
  • (Product_Description CONTAINS Personal
    Computer) HIGH AND
  • (Product_Description CONTAINS good ergonomics)
    LOW AND
  • (Address.Country Italy OR TEXT CONTAINS
    Italy) HIGH AND
  • WITH Company_Logo HIGH
  • (IMAGE MATCHES
  • screen HIGH
  • Keyboard HIGH
  • AT LEAST 2 floppy_drives LOW) HIGH

28
Indexing and searching
  • Searching similar patterns
  • Distance function
  • Given two objects, O1 and O2, the distance
    (dissimilarity) of the two objects is denoted by
    D(O1,O2)
  • Similarity queries
  • Whole match
  • Sub-pattern match
  • Nearest neighbors
  • All pairs

29
Spatial access methods
  • Map objects into points in f-D space, and to use
    multiattribute access methods (also referred to
    as spatial access methods or SAMs) to cluster
    them and to search for them
  • Methods
  • R-trees and the rest of the R-tree family
  • Linear quadtrees
  • Grid-files
  • Linear quadtrees and grid files explode
    exponentially with the dimensionality

30
R-tree
  • R-tree
  • Represent a spatial object by its minimum
    bounding rectangle (MBR)
  • Data rectangles are grouped to form parent nodes
    (recursively grouped)
  • The MBR of a parent node completely contains the
    MBRs of its children
  • MBRs are allowed to overlap
  • Nodes of the tree correspond to disk pages

31
R-tree
  • Range query
  • Specify a region of interest, requiring all the
    data regions that intersect it
  • Retrieve
  • Compute the MBR of the query region
  • Recursively descend the R-tree, excluding the
    branches whose MBRs do not intersect the query
    MBR
  • The retrieved data regions will be further
    examined for intersection with the query region

32
(No Transcript)
33
Generic multimedia indexing approach
  • Whole match problem
  • A collection of N objects O1, O2,,ON
  • The distance/dissimilarity between two objects
    (Oi,Oj) is given by the function D(Oi,Oj)
  • User specifies a query object Q, and a tolerance
    e
  • Goal
  • Find the objects in the collection that are
    within distance efrom the query object

34
GEMINI
  • Generic Multimedia object INdexIng
  • Ideas
  • A quick-and-dirty test, to discard quickly the
    vast majority of non-qualifying objects
    (possibly, allowing some false alarms)
  • The use of spatial access methods, to achieve
    faster-than-sequential searching

35
GEMINI
  • Example
  • Database yearly stock price movements, with one
    price per day
  • Distance function
  • Euclidean distance
  • The idea behind the quick-and-dirty test is to
    characterize a sequence with a single number
    (feature), which help us discard many
    non-qualifying sequences
  • Average stock price over the year, standard
    deviation, some of the discrete Fourier transform
    (DFT) coefficients

36
GEMINI
  • Mapping function
  • Let F() be the mapping of objects to
    f-dimensional points, that is, F(O) will be the
    f-D point that corresponds to object O
  • Organize f-D points into a spatial access method,
    cluster them in a hierarchical structure, like
    the R-trees
  • Upon a query, we can exploit the R-tree, to
    prune out large portions of the database that are
    not promising

37
(No Transcript)
38
GEMINI
  • Search algorithm (for whole match query)
  • Map the query object Q into a point F(Q) in
    feature space
  • Using a spatial access method, retrieve all
    points within the desired tolerance efrom F(Q)
  • Retrieve the corresponding objects, compute their
    actual distance from Q and discard the false
    alarms

39
GEMINI
  • Lower Bounding lemma
  • To guarantee no false dismissals for whole-match
    queries, the feature extraction function F()
    should satisfy the following formula
  • Dfeature() distance of two feature vectors
  • (mapping F() from objects to points should make
    things look closer)

40
GEMINI
  • GEMINI algorithm
  • Determine the distance function D() between two
    objects
  • Find one or more numerical feature-extraction
    functions, to provide a quick-and-dirty test
  • Prove that the distance in feature space
    lower-bounds the actual distance D(), to
    guarantee correctness
  • Use a SAM (e.g., an R-tree), to store and
    retrieve the f-D feature vectors

41
GEMINI
  • Feature-extracting question
  • If we are allowed to use only one numerical
    feature to describe each data object, what should
    this feature be?
  • The successful answers to the question should
    meet two goals
  • They should facilitate step 3 (the distance
    lower-bounding)
  • They should capture most of the characteristics
    of the objects

42
One-dimensional time series
  • Search a collection of (equal-length) time
    series, to find the ones that are similar to a
    desirable series.
  • in a collection of yearly stock price movements,
    find the ones that are similar to IBM
  • Distance function
  • Euclidean distance

43
One-dimensional time series
  • Feature extraction
  • The coefficients of the Discrete Fourier
    Transform (DFT)
  • Lower-bounding
  • Parsevals theorem
  • The DFT preserves the energy of a signal, as well
    as the distances between two signals

44
One-dimensional time series
  • If we keep the first f (fn) coefficients of the
    DEF as the features, we lower-bound the actual
    distance
  • There will be no false dismissals

45
One-dimensional time series
  • DFT concentrates the energy in the first few
    coefficients, for a large class of signals, the
    colored noises. These signals have a skewed
    energy spectrum (O(F-b))
  • b2 random walks (brown noises)
  • Model stock movements and exchange rates
  • b2 black noises
  • Model water level of rivers and rainfall patterns
  • b1 pink noise
  • interesting signals musical scores and other
    works of art
  • (white noise is unpredictable, brown noise is
    too predictable)

46
(No Transcript)
47
One-dimensional time series
  • Experiments
  • Artificially generated random walks
  • Sequence length n1024
  • Database size N50400
  • GEMINI vs. sequential scanning
  • SAM R-tree
  • Response time

48
(No Transcript)
49
One-dimensional time series
  • GEMINI can be successfully applied to time
    series, and specifically to the ones that behave
    like colored noise
  • For signals with skewed spectrum, the minimum in
    the response time is achieved for a mall number
    of Fourier coefficients (f1,2,3). The minimum is
    rather flat, which implies that a suboptimal
    choice for f will give search time that is close
    to the minimum
  • The success in 1D series suggests that GEMINI is
    promising for 2D or higher-dimensionality
    signals, if those signals also have skewed
    spectrum

50
Two-dimensional color images
  • QBIC (Query By Image Content) (IBM)
  • Query large online image databases using the
    images content as the basis of the queries
  • Content
  • Color, texture, shape, position, and dominant
    edges of image items and regions
  • Applications
  • Medical
  • Give me other images that contain a tumor with a
    texture like this one
  • Photo-journalism
  • Give me images that have blue at the top and red
    at the bottom

51
Two-dimensional color images
  • Two datatypes
  • Image (scene)
  • Item
  • A part of a scene
  • A person, a piece of outlined texture, an apple,
  • Image feature
  • Focus on the color features
  • K-element color histogram for each item and
    scene, where k256 or 64 colors
  • Each component on the color histogram is the
    percentage of pixels that are most similar to
    that color

52
(No Transcript)
53
Two-dimensional color images
  • Distance function
  • A color-to-color similarity matrix
  • Obstacle of color indexing
  • Dimensionality curse
  • Quadratic nature of the distance function
  • Cross-talk among the features
  • Expansive
  • Precludes efficient implementation of commonly
    used spatial access methods

54
(No Transcript)
55
Two-dimensional color images
  • GEMINI
  • Feature extraction
  • Average amount of red, green, and blue in a given
    color image (RGB color space)

P is the number of pixels in the item R(P), G(p),
B(p) are the red, green, and blue components
respectively of the p-th pixel
56
Two-dimensional color images
  • Distance function
  • Lower-bounding
  • Quadratic Distance Bounding Theorem

57
Two-dimensional color images
  • Experiment
  • N924 color images
  • K256 colors
  • CPU time and disk accesses

58
(No Transcript)
59
Automatic feature extraction
  • GEMINI is useful for any setting that we can
    extract features from
  • Automatic feature extraction methods
  • Multidimensional Scaling (MDS)
  • FastMap
  • Extracting features not only facilitates the use
    of off-the-shelf spatial access methods, but it
    also allows for visual data mining we can plot a
    2D or 3D projection of the data set, and inspect
    it for clusters, correlations, and other patterns

60
(No Transcript)
About PowerShow.com