Using Several Ontologies for Describing AudioVisual Documents: A Case Study in the Medical Domain

1
Using Several Ontologies for Describing
Audio-Visual DocumentsA Case Study in the
Medical Domain
Antoine Isaac1 Raphaël Troncy2

• Sunday 29th of May, 2005

Multimedia and the Semantic Web
2
Describe AV documents

• Various uses / Different granularities
• Identification, feature extraction, structural
  decomposition, semantic description
• Description deep meaning cannot be accessed and
  processed by systems
• Knowledge is often implicit labels and comments
  in natural language
• Formal semantics should be interesting
• Reasoning with AV document descriptions
• Interoperability with formal domain-specific
  ontologies, allowing to mix AV and domain-related
  reasoning

? Use of Semantic Web technologies to better
retrieve, re-use and process AV content
3
Objectives

• Settle an mini-experiment to show the benefits of
  using semantic web technologies for annotating
  multimedia content
• Show that the use of
• formal ontologies and rules,
• inference capabilities,
• annotation design pattern
• are highly desirable for better accessing AV
  content

4
Agenda

• Corpus
• Ontological Resources
• AV Ontology
• Medical Ontology
• Annotating the Videos
• Querying the Knowledge Base
• Performing Reasoning
• Conclusion

5
Corpus

• Medicine-related TV documentaries
• 30 documents, about 30 hours
• 50 deals with heart and heart surgery theme
• Good examples of how AV features are used to
  popularize scientific notions
• Describe both the form and the content
• AV-oriented parts (documentary structure)
• Thematic-oriented parts (medicine notions)

6
Ontological Resources

• Building an Audio-Visual Core Ontology Isaac
  Troncy, 2004
• Characterization of programs and sequences (AV
  genre)
• Decomposition of programs and sequences
• Ability to introduce description of the
  activities that constitute the context of AV
  documents (roles of people involved, way
  production and broadcast are achieved, etc.)
• Legitimacy
• Grounding conceptualization by observed purposes
  and domain initiatives, study of 30 years of
  documentary practices
• Articulation with an upper-level ontology DOLCE
  Gangemi, 2002

7
Ontological Resources
8
Ontological Resources

• Extension of AV core with specific application
  notions
• Exemplification, demonstration, etc.
• Re-use of Medical Ontologies
• Menelas domain of coronary pathologies
• Concepts dealing with heart surgery
• Alternative choices are possible
• Galen (concepts dealing with surgical procedures)
• Articulation between the ontologies
• No use of automatic alignment methods or tools
• State by hand OWL axioms (equivalentClass)

9
Description Process

• Segmentation of the AV material
• Selection of relevant documentary items
• Knowledge-based AV description
• Documentary structure characterization
• Segment description

10
Segmenting the Videos
11
Describing the Videos

• Annotation Mechanism
• The structure is described at the knowledge level
• Concepts and relations from the AV ontology are
  manually introduced in the description
• Content description
• Link to external world themes and entities
• Documentary patterns
• Layered approach Troncy, 2003
• AV description language Troncy Carrive, 2004

12
Describing the Videos

• Relational Indexing Pattern
• Help for user specify how concepts and relations
  have to be used
• Important for ontology conception and use (with
  reasoning knowledge)

• Simple pattern that can lead to complex
  descriptions
• Recursive relational structure

13
Describing the Videos
14
Querying the Knowledge Base

• Example
• retrieve the programs that explain a disease
  and show one of its causes
• Need for the following inferences
• Subsumption
• Composition

15
Performing reasoning

• A layered complexity approach
• RDFS subsumption
• OWL DL complex definitions algebraic
  properties
• Rules horn clauses
• Concrete implementation
• RDFS Sesame Architecture Broekstra, 2002
• OWL DL BOR Reasoner Simov, 2002
• OWL-DLP Grosof, 2003 Rules Sesame custom
  inference module

16
Examples

• DL definition
• Composition rule

17
Summary
18
Conclusion

• This experimentation
• Uses Semantic Web languages and tools for
  describing AV contents
• Uses several ontologies to capture both the
  structure and the content of the documents
• Uses relational indexing patterns for the
  annotation
• Future work thorough evaluation of those
  techniques involving real users
• A problem that cannot be generally solved fixing
  a trade-off between expressivity and tractable
  computation
• Ad hoc, according to the needs of the application
  targeted