Semantics in Multimedia Content for Multimedia Use

1
Semantics in Multimedia Content for Multimedia Use

• Steffen Staab
• http//isweb.uni-koblenz.de

2
Semantics What for?

• Tasks Purpose
• Find me pictures about a boy on the beach
• Find me cute pictures about Nico to show to my
  Mum
• Find me pictures from Turkey
• Find me pictures from our common holiday
• with a resolution of .
• with no other person in the foreground

3
Multimedia Semantics

• Content semantics (general knowledge) boy
  beach jet ski sea sky
• Content semantics (private knowledge) Nico
  digging
• Retrieval semantics cute picture to show
• Situational semantics Side, Turkey, 2005
• Social semantics Vacation with friends 05

4
Multimedia Semantics
Human perspective
Technical perspective
common vacation
Multimedia social semantics
Multimedia retrieval semantics
cute
Multimedia situational semantics
Turkey
boy
Multimedia content semantics
And semiotics will tell you that this list is
incomplete
5
Multimedia Semantics
common vacation
Manual Class. Mgmt
Multimedia social semantics
Feedback AutoClassify
Multimedia retrieval semantics
cute
Capture (during creation)
Multimedia situational semantics
Turkey
Extract from (image) data
boy
Multimedia content semantics
6
Find me a cute picture of Nico digging during our
common vacation
Manual Class. Mgmt
Feedback AutoClassify
Capture (during creation)
myCal
Extract from (image) data
7
Ontology
sand texture
common vacation
cute
Turkey
boy
myCal
Core Question for Multimedia Use How should this
ontology look like? Is it (like) MPEG-7 ?
8
What is the Problem with MPEG-7?
ltMpeg7gt ltDescription xsitype"ContentEntityType"
gt ltMultimediaContent xsitypeImageType"gt
ltImagegt ltSpatialDecompositiongt
ltStillRegion idSR1gt ltTextAnnotationgt
ltKeywordAnnotation xmllang"en"gt
ltKeywordgtSkylt/Keywordgt lt/KeywordAnnotationgt
lt/TextAnnotationgt lt/StillRegiongt
ltStillRegion idSR2gt ltSemanticgt
ltLabelgt ltNamegtSkylt/Namegt ltLabelgt
lt/Semanticgt lt/StillRegiongt
ltStillRegion idSR3gt ltSemanticgt
ltDefinitiongt lt!-- Also TextAnnotation!! --gt
ltStructuredAnnotationgt ltWhatObjectgt
ltName xmllang"en"gtSkylt/Namegt
lt/WhatObjectgt lt/StructuredAnnotationgt
lt/Definitiongt lt/Semanticgt
lt/StillRegiongt ...
How do you formulate a query to get all segments
that show Sky?
First Shot XQL //StillRegion.//KeywordSky
9
Conclusion
Sharing Tagging systems
Multimedia social semantics
Semantic querying
Multimedia retrieval semantics
Process support
Multimedia situational semantics
Constraint reasoning
Multimedia content semantics
10
ISWeb - Information Systems Semantic
Web University of Koblenz ? Landau, Germany
Thank You