Tag Suggestr: Automatic Photo Tag Expansion using Visual Information for Photo Sharing Websites

1
Tag Suggestr Automatic Photo Tag Expansion using
Visual Information for Photo Sharing Websites

• Onur Kucuktunc, Sare G. Sevil, A. Burak Tosun,
  Hilal Zitouni, Pinar Duygulu, and Fazli Can
• Bilkent University,
• Department of Computer Engineering.
• Ankara, Turkey

2
Motivation

• Organization and retrieval problems

2
SAMT 2008 Tag Suggestr
3
Motivation

3
SAMT 2008 Tag Suggestr
4
Motivation
4
SAMT 2008 Tag Suggestr
5
Purpose

• Assist users in order to
• Ease the process
• Reduce noise in tags
• Provide more descriptive/meaningful tags.
• TagSuggestr suggesting tags at upload time

5
SAMT 2008 Tag Suggestr
6
Approach
6
SAMT 2008 Tag Suggestr
7
Approach Initial Tagging

• Require initial set of tags
• Initial tags
• used for retrieving relevant
  images.

7
SAMT 2008 Tag Suggestr
8
Approach Relevant Photos

• Photos including all initial tags..
• Retrieve corresponding tags.
• Form set of distinct tags.

8
SAMT 2008 Tag Suggestr
9
Approach Computing Weights

• We need to assign weights to tags.
• Visual similarity between a given image and
  target image is important.

9
SAMT 2008 Tag Suggestr
10
Approach Computing Weights

• Visual similarity computed as
• Two different feature extraction methods were
  evaluated
• Color Histograms
• Interest Points

10
SAMT 2008 Tag Suggestr
11
Approach Computing Weights

• weight of tag i.

11
SAMT 2008 Tag Suggestr
12
Approach

• Sort tags according to weights.
• Tags with higher weights are suggested.

12
SAMT 2008 Tag Suggestr
13
Experimental Work

• Applied to Flickr Java API FlickrJ.
• 100 photos selected 66 target photos
• For each target photo
• 100 relevant photos
• 7000 images

13
SAMT 2008 Tag Suggestr
14
Experimental Work

• Analysis Number of required initial tags
• 2-3 tags

14
SAMT 2008 Tag Suggestr
15
Evaluation Methods

• How do we decide a suggested tag is valid?
• Use an existing Flickr photo
• Manually pick initial tags from its tag list.
• Performance how many of the original tags could
  we suggest?

15
SAMT 2008 Tag Suggestr
16
Evaluation Methods

• Computed precision/recall values and accuracy.
• 3 approaches evaluated
• Tag Frequency
• Color Similarity
• SIFT Similarity
• Frequency approach is the baseline.

16
SAMT 2008 Tag Suggestr
17
Evaluation Methods

• Two experiments performed
• Suggest-All-Tags
• Suggest-Top-5

17
SAMT 2008 Tag Suggestr
18
Evaluation Methods

• Suggest-All-Tags
• Suggest as many tags as were present in the
  original list.

18
SAMT 2008 Tag Suggestr
19
Evaluation Methods

Gray Color Histogram, Black SIFT, Dashed
Frequency
19
SAMT 2008 Tag Suggestr
20
Evaluation Methods

• Gray Color Histogram, Black SIFT, Dashed
  Frequency

20
SAMT 2008 Tag Suggestr
21
Evaluation Methods

• Gray Color Histogram, Black SIFT, Dashed
  Frequency

21
SAMT 2008 Tag Suggestr
22
Evaluation Methods

22
SAMT 2008 Tag Suggestr
23
Evaluation Methods

23
SAMT 2008 Tag Suggestr
24
Evaluation Methods

• Suggest-Top-5
• Suggest only the top 5 tags.

24
SAMT 2008 Tag Suggestr
25
Evaluation Methods

25
SAMT 2008 Tag Suggestr
26
Evaluation Methods
26
SAMT 2008 Tag Suggestr
27
Evaluation Methods

• Over all accuracy values

27
SAMT 2008 Tag Suggestr
28
Discussion

• Difficulty of selecting a ground truth
• Tag lists are
• Noisy
• incomplete
• Limited dataset
• Results are data dependent

28
SAMT 2008 Tag Suggestr
29
Future Work

• Larger dataset
• Evaluate different visual features and their
  combinations
• Extensive user-study

29
SAMT 2008 Tag Suggestr
30
Demo

30
SAMT 2008 Tag Suggestr
31
31
SAMT 2008 Tag Suggestr
32
32
SAMT 2008 Tag Suggestr
33
33
SAMT 2008 Tag Suggestr
34
34
SAMT 2008 Tag Suggestr
35
Conclusion
35
SAMT 2008 Tag Suggestr
36
Thank You for Listening.

• Questions?

37
References

• www.wikipedia.org
• www.flickr.com