Some ideas around PIADA: (Picture Indexing: Affect, Description and Availability)

1
Some ideas around PIADA (Picture Indexing
Affect, Description and Availability)

• Diana Santos

2
PIADA in a nutshell

• Context making sense of images
• High-level concerns
• Purpose
• Interaction with text/context
• Cultural factors
• Jokes, irony, metaphor, affect, humour
• Practical setup Web archive, image repositories,
  Wikimedia
• Expected outcomes
• Picture ontologies (as well as integration with
  other NLP ontologies)
• Reasoning with images
• Cross-cultural understanding of image differences

3
Presentation outlook

• A short description of the main ideas of PIADA
• 5 things not yet solved
• How to go about it? Data 3 kinds of reasoning
• SINTEFs LEG (Language Engineering Group)
• PIADA in a Norwegian context

4
1. Find images with feelings

• Most image descriptions are about objective
  features like who (names), where (places), dates,
  colours and objects
• Asterix sad no way, and we know there exist
  these pictures
• Try Hillary happy and Hillary angry and you get
  mostly the same pictures those of Hillary
  Clinton together with texts that may report
  whatever about happiness or sadness of her
  competitors or fans

5
2. Find culturally implicit information

• Japanese people in the underground (tube) most
  probably posted by Japanese people, not with
  captions saying that people are Japanese...
• Try Google images and give up
• Man reading. This can be a good enough caption in
  an European museological context, but certainly
  not in an Asian, or African context

6
3. Find synergy between images and text

• why this caption or this illustration? why do
  they work together? creative use of pictures in
  text or how many of these images of Sócrates
  are jokes?
• from the obvious a happy baby in a diapers
  advertisement to the provocative or very very
  subtle humour (was it?)

7
4. Find the pictures purpose
http//staff.science.uva.nl/marx/

• Why is this picture here?
• Reasoning about a multimedia world... finding the
  cause for humour and the important details
• Vi roser Anne!

How my students end up....
8
5. Help discuss or describe pictures for
different purposes

• Properties of pictures are often mentioned in
  some contexts
• (didactical, antropological, documentary,
  scientific) to focus on particular details see
  the character behind Jesus, see the tree on the
  left, note the tool on his hand, look at the back
  wings, at the dark clouds, at the tumor ...
• (police or law courts) to explain why the picture
  was taken revolver under the table, near the
  corpse
• (artistic setting) in sunlight, in rainy wather,
  with a special lins...

9
In a nutshell, we believe that

• there are many aspects of picture description and
  reasoning that have not received any or enough
  attention, namely
• emotions
• creativity and humour
• crosscultural differences
• intertextuality with pictures
• a natural language processing angle is the right
  way to attack them

10
Real applications

• Significantly enhancing image bank providers
  activity
• Helping professionals that need images and text
• teachers
• museum staff
• encyclopedia authors (the Wikipedia community)
• other multimedia content providers for games,
  educational CDs, textbooks
• advertisers
• historians and biographers
• Common multilingual image search

11
How to go about it? Data

• Image collections to study
• Image ontologies and folksonomies available
• Text and image collections
• Wikipedia
• Web pages (Internet archive)
• Special sites and or multimedia products (guided
  tours)
• Image search logs
• Elicitation collections sets of stories about
  image search
• Game results eliciting similarities or
  associations among images

12
How to go about it? Reasoning 1

• If you want to find a picture of a strong healthy
  man, or of a genius, you probably find an
  instance of a person that illustrates these
  qualities (such as Johny Weissmuller or Einstein
  and look for them)
• If you want to find a picture of Asterix angry,
  you can look for more objective descriptions
  such as Asterix shouting or Asterix beating
• If you want to illustrate the property black you
  probably look for concepts where black is
  stereotypical, such as coffee...

13
How to go about it? Reasoning 2

• You need to know the context of the annotation or
  text to know what should be implicit and what
  should be probably commented out
• searches in .pt for Sócrates are probably about
  the prime-minister but in .no, after the
  philosopher, comes the Brazilian football player
  ?
• the stereotypical image for man or woman is
  obviously different depending on the gender of
  the beholder, no matter sexual orientation, and
  the same for places and cultures
• a typical restaurant (and its food) varies
  widely
• pictures captioned Lisbon (or Dublin, or
  Helsinki) are most often by tourists. People who
  live in Lisbon give the precise names of what
  they take the picture of, or dont even bother to
  specify location

14
How to go about it? Reasoning 3

• Why are the images chosen?
• What is the kind of connection?
• What is their import?
• Which other associations -- interpictuality --
  they bring?
• Why can they be considered offensive or funny?
• Do they feel old-fashioned? Do they feel modern?

15
Language engineering at SINTEF

• Question answering
• Ontologies
• Geographical reasoning
• Contrastive studies
• Information extraction
• Corpus search
• We believe all these pieces will help us to
  address the image search and indexing issue.

16
PIADA in a Norwegian context

• We want to develop specific knowledge on images
  in Norwegian
• The vocabulary of images and image search in
  Norway
• Demo collections
• Study of user behaviour what do people ask for,
  what do they want to see?
• Picture reasoning ontologies
• We hope that by cooperating with commercial
  actors we will do something useful not only for
  research purposes

17
Specific proposal

• brukerstyrte innovasjonsprosjekter (BIP)
• OR
• kompetanseprosjekter med brukermedvirkning (KMB)
• Scanpix Norge as prime contractor
• SINTEF writes most of the proposal
• ABM-utvikling is also involved
• Other Norwegian actors also contacted