Photo Identity, Tagging, and Multimedia Logging Applications

1
Photo Identity, Tagging, and Multimedia Logging
Applications

• Presented by Robert Chen

2
Leveraging Context to Resolve Identity in Photo
Albums

• Mor Naaman, Ron B. Yeh, Hector Garcia-Molina,
  Andreas Paepcke
• JCDL 05
• Animations from authors related slides

3
Motivation

• Photos most commonly/effectively identified by
  people in them
• Automatic facial detection/recognition difficult
• Current solution manual ID
• Long list annotation
• Limited retrieval

4
Novel Idea

• Leverage context from photo metadata and user
  input
• Predict which people are likely to appear in a
  photo based on context
• No facial detection/recognition

5
System Overview
Tonya
Mattan
Ima Zafi Mor Tonya Etty
Ima Zafi Mor Tonya Etty
Ima Zafi Mor Etty Maria
Ima Zafi Mor Etty Maria
6
System Overview (ii)

• Predictions are a likelihood score for a person
  to appear in a photo
• Derived from PhotoCompas system
• Every photo contains
• Time
• Lat/long
• May contain annotation (names)

7
Types of Parameters

• Location set of photos in a country and
  setting
• Events set of related photos
• Usually related by time
• Neighboring location set of photos taken within
  some radius
• Neighboring time set of photos taken within
  some time period

8
Basic Estimators

• Event E(s)
• Location L(s)
• Neighboring Nloc(s)
• Time Neighboring Ntime(s)
• Global S (all photos)
• All combine to create final estimator Q(s)

9
Example Event Estimator
P1
New Picture
P2
?
?

• P1Kimya, Dylan
• P2Dylan
• Dylan 2/2 100
• Kimya ½ 50

10
Co-Occurrence PeopleRank

• People usually appear together in groups
• Create a graph of people
• Nodes are people
• Links are of times a pair of people are
  together in a photo

11
PeopleRank Graph
12
PeopleRank Example

• Dylan lt-gt Nick 4 photos
• Kimya lt-gt Nick 8 photos
• Kimya 8/12 66
• Dylan 4/12 33

13
Aggregation

• Goal Generate a 5-person candidate list
• Padding use one estimator at a time until 5
  people selected
• Weighting weighted average of estimator values

14
Methodology

• Simulate process of identifying
• 4 sets of photos
• Casual vs. Industrious users
• Industrious record everything, all IDs
• Casual record a certain of IDs
• Prediction performances are equivalent after
  about 10 of pictures have been annotated

15
Results

• Location estimator is decent, 90
• PeopleRank (on the Event level) with basic
  estimators helps, slightly gt 90
• More people less accuracy
• More annotations more accuracy

16
Towards Automatic Extraction of Event and Place
Semantics from Flickr Tags

• Tye Rattenbury, Nathaniel Good, Mor Naaman
• SIGIR 07

17
Goal Motivation

• Goal
• Given photo with geo-location and time
• Given tag(s) for photo
• Determine whether photo represents an event or
  place
• Event time, Place location
• Event lecture, Place CS building
• Why do this?
• Improve image search
• If missing time/place info, can guess using tags
• Aggregate/gather photos better

18
Methods

• Important tags seen as bursts in time/space
• Naïve Scan
• Spatial Scan
• Scale-Structure Identification
• Look at tags in different scales
• Cluster of tags should exist consistently at each
  scale
• Works best, 64-70 accurate

19
Some Mobile Multimedia Products
20
Purpose

• Explore some applications which record media
  along with associated data
• Flickr
• Zurfer
• Symbian-Life Blog

21
Flickr

• Each user has account
• Upload pictures
• Share with other users
• Social groups with common interests

22
Flickr Organization

• Sets groups of related pictures, defined by
  user
• Tags text attributes given to pictures and/or
  sets
• Groups created by users
• Map links to Yahoo!Maps, can place photos on
  the map and associate them with lat/long

23
Flickr Home
24
Flickr Map
25
Zurfer

• Phone application
• Uses Flickr database, but does not need to
• Goal show pictures of interesting items to the
  user
• What is interesting?
• Channels

26
Channel Types

• Nearby Photos
• Cell tower ID
• Local highlights
• Social channels
• Recent Comments
• Contacts Photos
• Group channels
• My Stuff linked to Photo Wallet set in Flickr
• Interesting Today Flickr views and comments
• Can also perform Flickr tag search

27
Zurfer Basic Channels Interesting Today
28
Zurfer Group Channel
29
Symbian LifeBlog

• Phone application
• Blog on the phone
• Records everything you do
• Should be able to upload to a web server
• Text, audio, pictures, text messages