ECE160 / CMPS182 Multimedia

1
ECE160 / CMPS182Multimedia

• Lecture 20 Spring 2009
• Image Recognition and Retrieval

2
National Research Priorities

• Energy Technologies
• Fuel efficient engines
• Replacement energy to fossil fuels
• Lighter, longer-duration batteries
• Bioengineering/Bioinformatics
• Genes ? disease
• Disease ? medicine
• Search with Multimedia Content
• Video surveillance
• Photo interpretation

3
Multimedia Recognition

• Video surveillance
• Photo interpretation

4
Wide-area Surveillance
advertisement of objectvideo.com
5
Surveillance Scenarios
6
Multimedia Recognition

• Video surveillance
• Photo interpretation

7
How to Organize these Photos?
8
Image Organization Retrieval

• Keyword-based
• Manual labeling is subjective, cumbersome
• The aliasing problem
• Content-based
• Promising for general semantics outdoor,
  landscape, flowers, people, etc.
• Not enough for wh-queries (where, who, when, or
  what)

9
EXTENTTM contEXT contENT

• Context
• Spatial (location)
• Temporal
• Social
• Others
• Content
• Perceptual features, such as color, texture, and
  shape
• Holistic features and local features

10
EXTENTTM
11
Augmented Images


Cameraphones with high-quality lens can record
location, time, camera parameters, and voice
12
Context from Space/Time

• GPS or CellID data
• Into place names
• Time-based grouping
• Into meaningful events
• From place names and time
• Time of day
• Weather

13
Example of Using Three Pieces of Information
14
Maui Sunsetscan be obtained from Space/Time
15
Use content for verification
16
Use content to transfer metadata
17
Summarize of the example

• Derived from Context
• Derive time of the day
• Obtain weather
• Verify content
• Use of Content
• Verify context
• Transfer context
• Much more

18
Are They Similar?
19
Are They Similar?
20
Are They Similar?

• In terms of what?
• What is the users perception?

21
Conveying Perception

• Image Databases
• Conveyed via Examples
• Use a sunset picture (or pictures) to find more
  sunset images
• Where does the perfect example come from?

22
Conveying Perception

• Internet Searches
• Conveyed via Keywords

23
Keyword Retrieval

• Pros
• A user-friendly paradigm
• Cons
• Annotation is a laborious process
• Annotation quality can be subpar
• Annotation can be subjective
• Synonyms

24
Conveying Perception

• Image Databases
• Conveyed via Examples
• Use a sunset picture (or pictures) to find more
  sunset images
• Where does the perfect example come from?

25
Are They Similar?
26
Are They Similar?
27
Are They Similar?

• In terms of what?
• What is the users perception?

28
Recogintion of Content
Blue
Green
Blue
29
Recognition
Sea
Whale
Sea
30
Recognition
Sky
Mountain
Lake
31
clouds vs. waves
32
Web 1.0 vs. Web 2.0
Content (text)
33
Web 2.0

• Content Users Interactions
• Collect rich, organized content
• Attract users interactions
• To provide metadata
• To provide new content
• Improve search quality
• With new metadata and data
• Via social-network structure

34
Fotofiti
User management
Metadata collection - contextual - content
Photo uploading
Metadata fusion
Social networks
Annotate photos
Event management
Photo search