From video segmentation to semantic indexing: the Prestospace approach

1
From video segmentation to semantic indexing
the Prestospace approach
Basili Roberto(1), Marco Cammisa(1), Laurent
Boch(2), Alberto Messina(2), Giorgio Dimino(2),
Valentin Tablan(3), Boslav Popov(4), Werner
Bailer(5), Walter Allasia(6), Michele
Vigilante(6) (1) University of Roma, Tor
Vergata (Italy) (2) RAI Centro Ricerche e
Innovazione Tecnologica (Italy) (3) University of
Sheffield (UK) (4) Sirma Lab., Ontotext
(Bulgaria) (5) Johannneum Research Center,
(Germany) (6) Eurix (Italy)

• 4th Conference on IMAGE INFORMATION MINING
• EUSC Torrejon air base - Madrid (Spain)

2
Outline

• Prestospace introduction and motivation of MAD
• MAD documentation
• Content and Semantic Analysis
• MAD publication
• Information Retrieval in MAD
• Conclusions

3
Prestospace

• Scientific and technological objective
• to develop an integrated approach to audiovisual
  preservation and access, to produce sustainable
  assets with high cultural value and potential
  commercial use.
• Metadata Access and Delivery (MAD)
• to generate, validate and deliver to the archive
  users metadata created by automatic and
  semi-automatic information extraction processes
• 32 partners, more than 8 EU member states
• Funding about 3.3 M

4
MADan architectural Overview
Large Storage Public Access Search
5
The MAD Data model

• Supports interoperability with existing standards
• EBU P/META, ISO MPEG7, Dublin Core, SMPTE MXF and
  DMS-1
• Results from the analysis of existing users data
  models
• Imposes a clean separation between editorial and
  material related information
• Definition of the basic entities and relations
• Supports incremental refinements

6
Schema of PrestoSpace document format (XML)
Ad hoc structures
Root element / wrapper
Identification and
Language information
P_META sets
Material realisations
Editorial partitions and views
Content related information
MPEG7 profile nodes
Enrichment information
Ancillary Data
7
Outline

• Prestospace introduction and motivation of MAD
• MAD documentation
• Content and Semantic Analysis
• MAD publication
• Information Retrieval in MAD
• Conclusions

8
Documentation process

• Automatic metadata extraction
• Legacy metadata import
• Audiovisual content analysis
• Semantic analysis
• Web mining and alignment
• Manual annotation and validation
• Current prototype features
• AV data news and documentaries
• Languages English and Italian

9
Core Documentation Platform
10
Content Analysis in MAD

• Currently available GAMPs
• Welcomer demux MXF (RAI)
• Content Analysis Speech to text (RAI)
• Content Analysis Shots finder (RAI)
• Content Analysis Video segmenter (RAI)
• Content Analysis Media analiser (JRS)
• Content Analysis Stripe images (JRS)
• Content Analysis Camera motion (JRS)
• Semantic Analysis Text segmenter (Sheffield)
• Semantic Analysis (Eng) Classifier, Named Entity
  extraction (Univ. of Sheffield)
• Semantic Analysis (Ita) Classifier, Named Entity
  extraction (Univ. of Tor Vergata)
• Semantic Analysis Web Aligner (Univ. of Tor
  Vergata)

11
Semantic analysis (ItaSA GAMP)

• Aim
• to process textual transcriptions of multimedia
  content (e.g. radio and TV broadcast)
• To provide information useful for retrieval, i.e.
  semantic metadata
• Target Semantic Phenomena
• Topical categories
• Recognition/Classification of (internal) Named
  Entities (NE)
• Alignment with external (Web) material
• Recognition of events and NE participants

12
ItaSA GAMP Architectural Overview
13
GUI Overview
14
GUI Overview (2)
15
GUI Overview (3)
16
Conceptual retrieval GUI
17
Retrieved Transcriptions
18
(No Transcript)
19
(No Transcript)
20
Outline

• Prestospace introduction and motivation of MAD
• MAD documentation
• Content and Semantic Analysis
• MAD publication
• Information Retrieval in MAD
• Conclusions

21
Publication Platforman Overview
Web interface
Rich Content
Publication Platform
http
Key Frames View
MCP
Speech to text display
Multimedia Contents Publisher
Full motion Video preview
Semantic Search (KIM Platform)
Topic Search (Full text)
22
Publication Platform
23
(No Transcript)
24
Multiarchive scenario
Query processor
UI
Query translation Context extraction
Publication Platforms
Distributed query middleware
25
IR Interface
26
Retrieval of AV data in MAD

• Different types of retrieval are required
• Legacy Information (structured queries)
• Full Text
• Ontology-driven browsing
• Natural language queries
• No constraint on the nature of the query (non NL
  queries are also managed)
• NERC
• Cross-lingual
• Query Classification for domain-specific retrieval

27
Simple Ontology-driven browsing Highlight,
Hyperlink,
28
Explore and Navigate
29
, Entity Pattern Search
30
Cross-language Retrieval (CLIR)

• Archive AV data span across different languages
• Retrieval should be concept rather than text
  oriented
• Source language (of queries) can be different
  from the target language (characterizing metadata)

31
CLIR in MAD

• Techhnology
• Domain modeling via LSA (query expansion)
• Word sense disambiguation (Wordnet)
• Sense based translations
• Ontological Ids
• Merging different evidencies (query translation)
• Strength
• Fully automatic
• Good accuracy
• Portable across domains
• Current limitations
• Resource Coverage (Multiwordnet)

32
(No Transcript)
33
CLIR GUI Settings
34
CLIR GUI translations
35
CLIR GUI translation (2)
36
CLIR GUI SA GAMP role
37
CLIR GUI Results

• Translated Query
• PersonBlair, OrganizationNato,
• (n.a.t.o "organizzazione del trattato
  nordatlantico"),
• membro, truppe, arma

38
CLIR GUI Results (2)
39
MAD Future steps

• Documentation platform
• Complete the annotation GAMP
• Optimise the system performance on specific
  content types
• Compile operational guidelines
• Publication platform
• Full Integration of the conceptual search engines
• Completion of the user interface
• Experimental Evaluation of cross language IR
• Turnkey system
• Define use cases
• Select the components (from Documentation and
  Publication)

40
Conclusions

• Prestospace realizes an advanced form of
  automatic extraction of semantic information from
  AV content
• Content and Semantic Analysis
• Web mining for collateral information is also
  applied
• Publication platform
• Different retrieval functionalities over the AV
  archives enabled Ontology-based browsing and
  robust Cross-language IR
• Links
• http//www.prestospace.org/
• RitroveRAI a Web application for semantic
  indexing and hyperlinking of multimedia news
  International Semantic Web Conference (ISWC05),
  Galway, Ireland