Integration of DomainSpecific and DomainIndependent Ontologies for Colonoscopy Video Database Annota

1
Integration of Domain-Specific and
Domain-Independent Ontologies for Colonoscopy
Video Database Annotation

• Jie Bao1, Yu Cao2, Wallapak Tavanapong2, and
  Vasant Honavar1
• Artificial Intelligence Research Laboratory1
• Multimedia Research Laboratory2
• Department of Computer Science
• Iowa State University, Ames, IA USA 50010
• baojie, yucao, tavanapo, honavar_at_cs.iastate.edu

2
Outline

• Background of Colonoscopy Video
• Ontology for Video Annotation
• Construction of Domain Ontology and Video
  Ontology
• Discussion

3
Colonoscopy

• Colon is part of intestine and Colonoscopy is an
  important endoscopic screening modality for
  colorectal cancer
• Colonoscopy allows inspection of the entire colon
  and provides the ability to perform a number of
  therapeutic operations (e.g., polyp removal)
• A sequence of images of the colon generated
  during endoscopic procedures provides invaluable
  information for colorectal research.

4
Colonoscopy Analysis System
Semantic units of the colonoscopy video, such as
scene and operation shots, are recognized by
automatically parsing video and audio contents.
Background work The experimental system for
parsing colonoscopy videos into semantic units
5
Outline

• Background of Colonoscopy Video
• Ontology for Video Annotation
• Construction of Domain Ontology and Video
  Ontology
• Discussion

6
Semantics in Colonoscopy Videos

• Integration of
• Multimedia Description
• Structure Units Scenes, Shots
• Visual Features Color, Texture
• Audio Features Dictations of the endoscopist
• Domain Description
• Reasons, Procedures, Findings, Diagnosis,
  Anatomical Sites, etc.

• Useful for
• Standard description for diagnosis
• Automated retrieval and inference
• Data-driven knowledge acquisition, such as
  machine learning

7
Semantic Units in Colonoscopy Videos

• View is part of a video for one continuous
  inspection
• A scene corresponds to an important part of
• the colon such as Rectum and Sigmoid
• An operation shot corresponds to a biopsy or
  therapeutic operation such as polyp removal

View
8
Colonoscopy Video Ontology

• Those semantics could be represented by Ontology
• Ontology provides a formal conceptualization of a
  given domain
• In the context of knowledge modeling, an ontology
  is a description of the concepts and
  relationships of interest
• Example Frame isPartOf Shot Congested equals
  Edematous

9
Ontology Design Considerations

• Expressiveness Sufficiently expressive to permit
  precise specification of the entities of the
  domain and the relations and constraints among
  the entities
• Formal with precise syntax and semantics
• Inferentially adequate Permit automated
  inferences
• Example Enable an agent to infer that if a Video
  is about Colon, all scenes in that video are
  likely to be about Colon.
• Unified framework Describe both the structure of
  the video as well as domain-specific terms and
  relationships among terms within a single unified
  framework

10
How to represent OntologySyntax Level

• XML
• Examples MPEG7, Dublin Core
• Has a limited utility as a knowledge
  representation language
• Semantic Ambiguity
• partOf or subClassOf ?
• Query over syntax, not over semantics
• XQuery

11
How to represent OntologySemantic Level

• Ontology Language
• Use DAMLOIL or OWL (Web Ontology Language)
• Extension of RDF(Resource Description Framework)
• Usually use XML as syntax
• Standard technique used in Semantic Web
• Ontology entities class, property, instance
• Class (concept) Colon subClassOf Site
• Property (role) domainVideoClip hasID
  rangeinteger
• Instance () Video_010 typeVideo
• Can be mapped to description logic, enabling
  automatic and tractable inferences
• DL is a subset of first order logic
• Inference engines available FACT, RACER

12
Outline

• Background of Colonoscopy Video
• Ontology for Video Annotation
• Construction of Domain Ontology and Video
  Ontology
• Discussion

13
Domain Ontology - MST

• Minimal Standard Terminology (MST) a controlled
  vocabulary for colonoscopy and endoscopy
  reporting
• The terms and their relations in MST
  documentation are given in a set of tables
• Class in taxonomy (subclass) of terms, like site,
  findings, reasons, and diagnosis.
• Terms listed in the tables as Attributes and
  Attributes Values are modeled as properties of
  corresponding classes

14
Domain Ontology - Part
15
Video Ontology

• Extend the set of MPEG-7 descriptors
• Semantic units organized in partOf hierarchy
• hasStart, hasEnd
• Controlled inheritance of properties - common
  properties like hasSite, and hasID are modeled in
  VideoClip
• Includes video and audio features

16
Video Ontology

• rdfabout"hasStart"/

17
Integration of Domain Ontology and Video Ontology

• Instances of MST ontology are added as features
  to the instances of video ontology
• Annotate different levels of video ontology with
  different MST annotations
• Example Reasons and Examination properties are
  only meaningful at the Video level, but not at
  the Frame or Shot levels
• Cardinalities Multiple annotation of them are
  legal

18
Integrated Ontology for Video Annotation (IOVA)
19
Integration of Domain Ontology and Video Ontology
(Part)
20
Example instance of this ontology
21
Example instance of this ontology (Part)
22
Example instance of this ontology (Part)
23
Outline

• Background of Colonoscopy Video
• Ontology for Video Annotation
• Construction of Domain Ontology and Video
  Ontology
• Discussion

24
Conclusions

• In this paper, we present an approach to
  annotation of a video database using a domain
  specific ontology, a domain independent video
  ontology that encodes the structure and
  attributes of video data.
• The two ontologies are integrated using
  domain-specific semantic linkage. The result
  (IOVA) is represented in OWL.
• Annotation of video data using IOVA constitutes
  an important first step towards flexible and
  fully automated indexing, retrieval, inference,
  and data-driven knowledge discovery using video
  data in a broad range of applications including
  colonoscopy video analysis.

25
Questions

• Thank you!

26
VIDAI -video database annotation using IOVA
27
VIDAI

• Video analyzer Do semantic segmentation and
  labeling based on video and audio processing.
• Domain Term Annotator Semantic finding from the
  video analyzer is created as instance of MST
  terms and be annotated with appropriate video
  entities.
• Ontology Editor and Browser expert knowledge
  about the inspected video, such as examination,
  complications, reasons, finding. Diagnosis and
  therapy, could be added into the video case,
  resulting in fully annotated video
• Video Repository Annotated video is saved to the
  video repository, with support from some backend
  DBMS.
• Query Interface an interface that could be
  visited by agent or expert. Could based on RQL or
  RDQL
• Browsing interface A interface for user to view
  the query result