Multimedia Databases - PowerPoint PPT Presentation

Loading...

PPT – Multimedia Databases PowerPoint presentation | free to download - id: 7bbec6-ZTJjN



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

Multimedia Databases

Description:

Multimedia Databases Prepared by Pradeep Konduri Instructor: Dr. Yingshu Li Georgia State University – PowerPoint PPT presentation

Number of Views:11
Avg rating:3.0/5.0
Slides: 50
Provided by: Qing87
Learn more at: http://cs.gsu.edu
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Multimedia Databases


1
Multimedia Databases
  • Prepared by Pradeep Konduri
  • Instructor Dr. Yingshu LiGeorgia State
    University

2
Plan of Attack
  • Introduction
  • Architecture
  • Image Content Analysis
  • Modeling Constructs
  • Logical Implementation
  • Real-World Applications
  • Conclusion

3
Types of multimedia data
  • Text using a standard language (SGML, HTML)
  • Graphics encoded in CGM, postscript
  • Images bitmap, JPEG, MPEG
  • Video sequenced image data at specified rates
  • Audio aural recordings in a string of bits in
    digitized form

4
Nature of Multimedia Applications
  • Repositories central location for data
    maintained by DBMS, organized in storage levels
  • Presentations delivery of audio and video data,
    temporarily stored.
  • Collaborative complex design, analyzing data

5
Management Issues
  • Modeling complex objects, wide range of types
  • Design still in research
  • Storage representation, compression, buffering
    during I/O, mapping
  • Queries techniques need to be modified
  • Performance physical limitations, parallel
    processing

6
Research Problems
  • Information Retrieval in Queries Modeling the
    content of documents
  • Multimedia/Hypermedia Data Modeling and
    Retrieval Hyperlinks, Used in WWW
  • Text Retrieval Use of a thesaurus

7
Multimedia Database Applications
  • Documentation and keeping Records
  • Knowledge distribution
  • Education and Training
  • Marketing, Advertisement, Entertainment, Travel
  • Real-time Control, Monitoring

8
A Generic Architecture of MMDBMS
  • Media organization organize the features for
    retrieval(i.e., indexing the features with
    effective structures)
  • Media query processing accommodated with
    indexing structure, efficient search algorithm
    with similarity function should be designed

9
Multimedia Database Architecture
MM Data Pre- processor
lt!ELEMENT ..gt ..... lt!ATTLIST...gt
Meta-Data
Recognized components
Additional Information
Query Interface
MM Data
MM Data Instance
MM Data Instance
Users
Multimedia DBMS
Multimedia Data Preprocessing System
Database Processing
10
Document Database Architecture
lt!ELEMENT ..gt ..... lt!ATTLIST...gt
DTD Manager
DTD files
DTD Parser
DTD
Type Generator
Query Interface
lt!ELEMENT ..gt ..... lt!ATTLIST...gt
Document content
SGML/XML Parser
DTD
XML or SGML Document Instance
SGML/XML Documents
Parse Tree
C Types
Users
Multimedia DBMS
Instance Generator
C Objects
Document Processing System
Database Processing
11
Image Database Architecture
Semantic Objects
Syntactic Objects
Image Content Description
Meta-Data
Query Interface
Image Annotation
Image
Users
Image
Multimedia DBMS
Image Processing System
Database Processing
12
Video Database Architecture
Key Frames
Video Content Description
Meta-Data
Query Interface
Video
Video Annotation
Video
Users
Multimedia DBMS
Video Processing System
Database Processing
13
Image Content Analysis
  • Image content analysis can be categorized in 2
    groups
  • Low-level features vectors in a
    multi-dimensional space
  • Color
  • Texture
  • Shape
  • Mid- to high-level features Try to infer
    semantics
  • Semantic Gap

14
Image Content Analysis Color
  • Color space
  • Multidimensional space
  • A dimension is a color component
  • Examples of color space RGB, HSV
  • RGB space A color is a linear combination of 3
    primary colors (Red, Green and Blue)
  • Color Quantization
  • Used to reduce the color resolution of an image
  • Three widely used color features
  • Global color histogram
  • Local color histogram
  • Dominant color

15
Color Histograms
  • Color histograms indicate color distribution
    without spatial information
  • Color histogram distance metrics

16
Image Content Analysis Texture
  • Refers to visual patterns with properties of
    homogeneity that do not result from the presence
    of only a single color
  • Examples of texture Tree barks, clouds, water,
    bricks and fabrics
  • Texture features Contrast, uniformity,
    coarseness, roughness, frequency, density and
    directionality
  • Two types of texture descriptors
  • Statistical model-based
  • Explores the gray level spatial dependence of
    texture and extracts meaningful statistics as
    texture representation
  • Transform-based
  • DCT transform, Fourier-Mellin transform, Polar
    Fourier transform, Gabor and wavelet transform

17
Image Content Analysis Shape
  • Object segmentation
  • Approaches
  • Global threshold-based approach
  • Region growing,
  • Split and merge approach,
  • Edge detection app
  • Still a difficult problem in computer vision.
    Generally speaking it is difficult to achieve
    perfect segmentation

18
Salient Objects vs. Salient Points
  • Generic low-level description of images into
    salient objects and salient points

19
Modeling Images Principles
  • Support for multiple representations of an image
  • Support for user-defined categorization of images
  • Well-defined set of operations on images
  • An image can have (semantic, functional, spatial)
    relationships with other images (or documents)
    which should be represented in the DBMS
  • An image is composed of salient objects
    (meaningful image components)

20
Salient Object Modeling
  • Multiple representations of a salient object
    (grid, vector) are allowed
  • A salient object O is of a particular type which
    belongs to a user defined salient object types
    hierarchy
  • An image component may have some (semantic,
    functional, spatial) relationships with other
    salient objects

21
Semantic Gap
  • semantics-intensive multimedia systems
    applications

non-semantic multimedia data models
Semantic Gap
require
model
raw data,primitive properties (size, format,
etc)
semantic meaning of the data
22
Semantic modeling of multimedia -- Why hard?
  • Context-dependency
  • Semantics is not a static and intrinsic property
  • The semantics of an object often depends on
  • the application/user who manipulate the object
  • the role that the object plays
  • other objects in the same context

Example
Van Goghs paintings
flower
23
Why hard? (cont.)
  • Modality-independency
  • Media objects of different modalities may suggest
    the similar/related semantic meanings.
  • Example

Query
Results
Harry Potter has never been the star of a
Quidditch team, scoring points while riding a
broom far above the ground. He knows no spells,
has never helped to hatch a dragon, and has never
worn a cloak of invisibility.
image
video
text
24
MediaView A Semantic Bridge
  • An object-oriented view mechanism that bridges
    the semantic gap between multimedia systems and
    databases
  • Core concept media view (MV)
  • a customized context for semantic interpretation
    of media objects (text docs, images, video, etc)
  • collectively constitute the conceptual
    infrastructure of a multimedia system
    application

25
Architecture
MediaView Mechanism
26
Basic Concepts
  • A media view MVi is a virtual class that has a
    unique view name, a type description, and a set
    of objects associated with it.
  • A base class Ci is defined as a subclass of
    another base class Cj if and only if the
    following two conditions hold (1) properties(Cj)
    ? properties(Ci), and (2) extent(Ci) ?
    extent(Cj). If Ci is the subclass of Cj, we also
    say that there is an is-a relationship from Ci to
    Cj. A base schema (BS) is a directed acylic graph
    G(V, E), where V is a finite set of vertices and
    E is a finite set of edges as a binary relation
    defined on VV. Each element in V corresponds to
    a base class Ci. Each edge in the form of eltCi,
    Cjgt?E represents an is-a relationship from Ci to
    Cj (or Ci is a subclass of Cj).

27
Basic Concepts
  • A media view MVi is a subview of another media
    view MVj (or there is an is-a relationship from
    MVi to MVj) if and only if properties(MVj) ?
    properties(MVi) and extent(MVi) ? extent(MVj). A
    view schema (VS) is a directed acyclic graph
    GV, E, where a vertex in V corresponds to a
    media view MVi, and an edge eltMVi,MVjgt?E
    represents an is-a relationship from MVi to MVj
    (or MVi is a subview of MVj).

28
Basic Concepts
  • An example

29
Basic Concepts
  • Semantics-based data reorganization via media
    views

30
View Operators
  • A set of operators that take media views and view
    instances as operands.
  • Focus on the operators that are indispensable in
    supporting queries and navigation over multimedia
    objects.

31
View Operators
  • type-level
  • V-overlap
  • syntaxltbooleangt v-overlap (ltmedia view1, media
    view2 gt)
  • semantics true, if and only if (? o ?
    O)(o?extent(ltmedia view1gt) and o?extent(ltmedia
    view2gt))
  • Cross
  • syntaxltobjectgt cross (ltmedia view1, media
    view2 gt)
  • semanticsltobjectgt o ? O o ?
    extent(ltmedia view1gt) and o?extent(ltmedia
    view2gt)
  • Sum
  • syntaxltobjectgt sum (ltmedia view1,
    meida-view2 gt)
  • semanticsltobjectgt o ? O o ?
    extent(ltmedia view1gt) or o?extent(ltmedia view2gt)
  • Subtract
  • syntaxltobjectgt subtract (ltmedia view1, media
    view2gt)
  • semanticsltobjectgt o ? O o ? extent(ltmedia
    view1gt) and o?extent(ltmedia view2gt)

32
View Operators
  • instance-level
  • Class
  • syntaxltbase classgt class(ltview instancegt)
  • semanticsltview instancegt is a instance of ltbase
    classgt
  • components
  • syntaxltobjectgt components (ltview instancegt)
  • semantics ltobjectgt o?O o is a component
    (direct or indirect) of ltview instancegt
  • i-overlap
  • syntaxltbooleangt i-overlap (ltview instnace1gt,
    ltview instance2gt)
  • semantics true, if and only if (? o ? O) (o ?
    components (ltview instance1gt) and o ?
    components(ltview instance2gt))

33
View Algebra
  • Functions
  • -- derivation of new MVs from existing MVs
  • Heuristic Enumeration
  • Blind enumeration
  • Content-based enumeration
  • Semantics-based enumeration

34
View Algebra
  • Algebra Operators
  • select from src-MV where ltpredicategt
  • project ltproperty-listgt from src-MV
  • intersect (src-MV1, src-MV2)
  • union (src-MV1, src-MV2)
  • difference (src-MV1, src-MV2)

35
Comparison (vs. class)
media view object class
membership heterogeneous objects uniform objects
member acquisition dynamic inclusion/exclusion of existing objects of other classes creating new objects
mapping one object can belong to multiple media views one object has exactly one class
relationship inter-member semantic relationship N/A
36
Comparison (vs. traditional object view)
media view object view
membership heterogeneous objects uniform objects
relationship inter-member semantic relationship N/A
member properties instance-level properties (user-defined) inherited or derived properties (for view instances)
global properties MV-level properties (user-defined) N/A
37
Logical Implementation
  • MediaView Construction
  • MediaView Customization
  • MediaView Evolution

38
MediaViews Construction
  • Work with CBIR systems to acquire the knowledge
    from queries
  • Learn from previously performed queries
  • A multi-system approach to support multi-modality
    of media objects
  • Organize the semantics by following WordNet

39
Why WordNet?
  • Different queries may greatly vary with the
    liberty of choosing query keywords
  • We need an approach to organize those knowledge
    into a logic structure
  • A simple context a concept in WordNet
  • Common media views corresponds to simple
    contexts
  • We provide all common media views, based on which
    users can build complex ones.

40
Navigating the Multimedia Database
  • Navigating via semantic relationships of WordNet
  • Semantic Relationship Examples
  • Synonymy (similar) pipe, tube
  • Antonymy (opposite) fast, slow
  • Hyponymy (subordinate) tree, plant
  • Meronymy (part) chimney, house
  • Troponomy (manner) march, walk
  • Entailment drive, ride

41
Navigating the Multimedia Database
42
MediaViews Construction
43
Multi-dimensional Semantic Space
  • IS-A relationship in thesaurus
  • For example, Season has a 4-dimension semantic
    space spring, summer, autumn, winter

44
Encoding with Probabilistic Tree
  • A Probabilistic Tree specifies the probability of
    one media object semantically matching a certain
    concept in thesaurus.

45
Evolution through Feedback
  • A progressive approach
  • MediaView is accumulated along with the processes
    of user interactions
  • Two phases of feedback
  • System-feedback
  • User-feedback

46
Evolution through Feedback

47
Evolution through Feedback
  • Procedure
  • Record each feedback performed by users.
  • For each CBIR system i involved, calculate its
    accuracy rate of retrieval. That is, simply
    divide the total number of retrieved results by
    the number of correct results according to user
    feedback.
  • Reset the value of to its accuracy rate
    respectively.
  • Wait for next session of user feedback.

48
Fuzzy Logic based Evolution Approach
  • Due to the uncertainty of the semantics, can not
    make an absolute assertion that a media object is
    relevant or irrelevant to a context
  • A media object in a database may be retrieved as
    a relevant result to a context several times
    the more times a media object is retrieved, the
    more confidence it has to be considered as
    relevant to the context.

49
Fuzzy Logic based Evolution Approach
  • For a media object e, a context c,
  • - the accumulation of historial
    feedback information (from both system and
    users)
  • - the adjustment of after each feedback
    session


50
Inverse Propagation of Feedback
  • The drawback of up-down fashion of calculating
    the probability
  • E.g. Whether a media object matches season can
    not leverage from that the media object was a
    match of spring
  • Solution propagate the confidence value of a
    media object being relevant to a concept along
    the hierarchical structure from bottom-up

51
Inverse Propagation of Feedback
  • Procedure
  • Wait for a feedback session.
  • For each positive feedback, namely, stating a
    concept C is relevant to a media object.
    Following the thesaurus, trace from C to the root
    concept Root in thesaurus. Assume the path is
    ltC, C1, C2 , Root Cngt.
  • Append Ci as also positive feedback to that media
    object, where i1 to n.

52
MediaView Customization
  • Two level MediaView Framework

53
MediaView Customization
  • Dynamically construct complex-context-based media
    views based on simple ones
  • An example complex context the Grand Hall in
    City University
  • Several user-level operators are devised to
    support more complex/advanced contexts, besides
    the basic operators

54
User-level Operators
  • INHERIT_MV(N mv-name, NS set-of-mv-refs, VP
    set-of-property-ref, MP set-of-property-ref)
    mv-ref
  • UNION_MV(N mv-name, NS set-of-mv-refs) mv-ref
  • INTERSECTION_MV(N mv-name, NS set-of-mv-refs)
    mv-ref
  • DIFFERENCE_MV(N1 mv-ref, N2 mv-ref) mv-ref

55
Build a MediaView in Run-time
  • Example find out info about "Van Gogh"
  • Who is "Van Gogh"?
  • What is his work?
  • Know more about his whole life.
  • Know more about his country.
  • See his famous painting "sunflower"

56
Build a MediaView in Run-time
  • Who is Van Gogh?
  • INHERIT_MV(V. Gogh, ltpaintergt,nameVan Gogh
    ,)
  • What is his work?
  • INTERSECTION_MV(work, ltpaintinggt, vg)
  • Know more about his whole life.
  • INTERSECTION_MV(life, ltbiographygt, vg)
  • Know more about his country.
  • INTERSECTION_MV(country, ltcountrygt, vg)
  • See his famous painting sunflower
  • Set sunflower INTERSECTION_MV(sunflower,
    ltsunflowergt, ltpaintinggt)Set vg_sunflower
    INTERSECTION_MV(vg_sunflower, vg_work,
    sunflower)

57
Authoring Scenario
  • Creates a new media view named after the subject
  • All multimedia materials used in the document
    would be put into this MediaView for further
    reference.
  • To collect the most relevant materials for
    authoring, the user performs the MediaView
    building process.
  • Import suitable media objects by browsing media
    views
  • Reference the manner and style of authoring, to
    find other media views with similar topics.
  • Drag Drop
  • learning-from-references

58
Summary
  • Types of multimedia data Text, Audio, Video,
    Images.
  • Management issues Design, Storage, Modeling,
    Queries
  • Image Content Analysis Color, Texture, Shape
  • MediaView a semantic multimedia database
    modeling mechanism
  • to bridge the semantic gap between conventional
    database and semantics-intensive multimedia
    applications
  • A set of user-level operators to accommodate the
    specialization/generalization relationships among
    the media views

59
Summary (contd..)
  • MediaView promises more effective access to the
    content of media databases
  • Users could get the right stuff and tailor it to
    the context of their application easily.
  • Providing the most relevant content from
    pre-learnt semantic links between media and
    context
  • high performance database browsing and multimedia
    authoring tools can enable more comprehensive
    applications to the user.
  • Users could customize specific media view
    according to their tasks, by using user-level
    operators

60
Further Issues
  • The development and transition of MediaView to a
    fully-fledged multimedia database system
    supporting declarative queries
  • Intensive and extensive performance studies
  • Advanced semantic relations (eg. temporal and
    spatial ones) can also be incorporated in
    combining individual media views

61
  • Thank you!
  • Q A
  • Email pkonduri1_at_student.gsu.edu
About PowerShow.com