Workshop on Social Events in Web Multimedia, ICMR 2014 - PowerPoint PPT Presentation

Loading...

PPT – Workshop on Social Events in Web Multimedia, ICMR 2014 PowerPoint presentation | free to download - id: 71413d-MmNlZ



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

Workshop on Social Events in Web Multimedia, ICMR 2014

Description:

Title: PowerPoint Presentation Author: Akis Last modified by: gpetkos Created Date: 1/1/1601 12:00:00 AM Document presentation format: On-screen Show – PowerPoint PPT presentation

Number of Views:6
Avg rating:3.0/5.0
Slides: 25
Provided by: Aki105
Learn more at: http://mklab2.iti.gr
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Workshop on Social Events in Web Multimedia, ICMR 2014


1
Social Event Detection at MediaEval a
three-year retrospect of tasks and
results Georgios Petkos, Symeon Papadopoulos,
Vasileios Mezaris, Raphael Troncy, Philipp
Cimiano, Timo Reuter, Yiannis Kompatsiaris
Workshop on Social Events in Web Multimedia, ICMR
2014
2
Overview
  • The problem of social event detection.
  • The social event detection task.
  • Evolution of the task and datasets.
  • Overview of approaches pursued by participants
    and results.
  • Outlook.

3
Social events?
Attended by people and represented by multimedia
content shared online
news
demonstration / riot / speech
personal
wedding / birthday / drinks
entertainment
concert / play / sports
4
Pope Benedict
2007 iPhone release
2008 Android release
2010 iPad release
Pope Francis
http//petapixel.com/2013/03/14/a-starry-sea-of-ca
meras-at-the-unveiling-of-pope-francis/
5
Social event detection
  • Social event detection involves the discovery
    and retrieval of social events in collections of
    multimedia.

COLLECTION
EVENT SET
E1
E2
SOCIAL EVENT DETECTION
EN
5
6
The social event detection task
  • As part of the well-known MediaEval benchmarking
    activity, a task on social event detection has
    been running for 3 years (2011-2013).
  • The interest on the task has grown significantly
  • 2011 7 participants
  • 2012 5 participants
  • 2013 11 participants

6
7
SED 2011 The task
  • Two challenges were defined in 2011.
  • In both, participants were provided with a set of
    images collected from Flickr and were asked to
    surface events of a particular type at particular
    locations
  • Soccer matches in Barcelona and Rome.
  • Concerts in Paradiso and Parc Del Forum.
  • Two differences between the two challenges
  • In the first, both a topical and a location
    criterion are defined, whereas in the second only
    a location criterion is defined.
  • The specificity of the location of interest is
    different.

7
8
SED 2011 The dataset, ground truth and evaluation
  • 73,645 photos collected from Flickr.
  • All photos were originally geo-tagged and were
    taken at 5 different cities in May 2009 (geo-tags
    were removed for 80 of the pictures in the
    provided dataset).
  • The ground truth was generated by utilizing
    machine tags provided by event directories as
    well as an automatic cluster-based framework.
  • Evaluation measures
  • F-score
  • NMI

8
9
SED 2012 The task
  • Three challenges, similar to those of the first
    year, were defined in 2012.
  • Again, participants were provided with a set of
    images collected from Flickr and were asked to
    surface events of a particular type at particular
    locations
  • Technical events (e.g. exhibitions and fairs)
    that took place in Germany.
  • Soccer events in Hamburg and Madrid.
  • Demonstration and protest events of the
    Indignados movement in Madrid.
  • Characteristic of the challenges
  • Theme and location of queries quite different.
  • Notion of technical events somewhat fuzzy.
  • Indignados events are spontaneously organized.

9
10
SED 2012 The dataset, ground truth and evaluation
  • 167,332 photos collected from Flickr.
  • Again, all photos were originally geo-tagged but
    geo-tags were removed for 80 of the pictures in
    the provided dataset.
  • The ground truth was again generated by utilizing
    machine tags provided by event directories as
    well as an automatic cluster-based framework.
  • Evaluation measures
  • F-score
  • NMI

10
11
SED 2013 The task
  • Two completely new challenges were defined
  • Produce a complete clustering of the image
    dataset according to events.
  • Extension Assign a set of videos to the clusters
    generated from the image dataset.
  • Classify event media as either representing a
    social event or not and for those that do
    represent a social event identify the type of
    event (eight event types were defined).

11
12
SED 2013 The dataset, ground truth and evaluation
  • Separate dataset, ground truth and evaluation
    for each challenge
  • Challenge 1
  • Dataset 427,370 pictures from Flickr and 1,327
    videos from YouTube corresponding to 21,169
    events.
  • Ground truth obtained from last.fm and upcoming
    machine tags.
  • Evaluation F-score and NMI.
  • Challenge 2
  • Dataset 27,754 training images and 29,411 test
    images collected from Instagram.
  • Ground truth obtained by manual annotation.
  • Evaluation NMI.

12
13
Evolution of the task
  • Two distinct eras of the task
  • First two years. Datasets contained both event
    and non-event images and the task was to retrieve
    sets of images matching these criteria.
  • Third year. Broken into two subtasks
  • Full clustering.
  • Detection of event type in individual images.
  • (no filtering subtask though)
  • Also, datasets have become larger and richer.

13
14
Approaches First era, 2011-2012 (1/4)
  • At a very high level there are two types of
    approaches
  • Matching images to event descriptions retrieved
    from online event directories.
  • Applying a sequence of filtering or
    classification and clustering steps on the
    datasets.

14
15
Approaches First era, 2011-2012 (2/4)
  • Methods in the first class differ in the way that
    matching is carried out
  • Indexing and querying in Lucene.
  • Probabilistic matching.
  • Methods in the second class are much more popular
    and
  • Some utilize external sources, e.g. DBPedia or
    the Google geocoding API to enrich the matching
    criteria.
  • For most of them time and location (sometimes
    inferred by the textual metadata when geo-tags
    are not available) are the primary criteria for
    clustering.
  • Alternatively, some approaches treat the problem
    as a multimodal clustering problem utilizing a
    learned similarity metric

15
16
Approaches First era, 2011-2012 (3/4)
2011 Challenge 1 Challenge 1 Challenge 2 Challenge 2
2011 F-score NMI F-score NMI
Brenner et al. 68.70 0.410 33.00 0.500
Hintsa et al. - - 68.67 0.678
Liu et al. 59.13 0.247 68.95 0.617
Nguyen et al. 10.13 0.026 12.44 -0.01
Papadopoulos et al. 77.37 0.630 64.00 0.379
Ruocco et al. 58.65 0.475 66.05 0.644
Wang et. al 64.90 0.236 50.44 0.448
  • For challenge 1 best approach performs early
    classification of images into cities and then
    groups images into buckets containing same day
    and city photos.
  • For challenge 2, two matching-based approaches
    achieved the best results (most likely because
    the type of event makes it more likely to find
    relevant info in online directories).

16
17
Approaches First era, 2011-2012 (4/4)
2012 Challenge 1 Challenge 1 Challenge 2 Challenge 2 Challenge 2 Challenge 2
2012 F-score NMI F-score NMI F-score NMI
Zeppelzauer et al. 2.15 0.020 29.99 0.200 47.58 0.310
Vavliakis et al. 84.58 0.724 90.76 0.850 89.83 0.738
Schinas et al. 18.66 0.187 74.64 0.674 66.87 0.465
Brenner et al. - - 72.66 0.65 - -
Dao et al. 70.15 0.601 - - 60.96 0.446
  • The best approach by Vavliakis et al. involves
    the following steps
  • City classification.
  • For the images of each city, topic modeling
    using LDA is performed.
  • The topic model is used to match the photos that
    are relevant to the queries.
  • Events are identified by finding for each topic
    and city of interest the days for which there a
    number of images above some threshold.

17
18
Approaches Second era, 2013 (1/4)
  • For the first challenge, there are two main
    types of approaches
  • Sequence of unimodal clustering operations.
  • Multimodal clustering using a learned similarity
    measure.
  • However, there are also some rather distinct
    approaches, e.g.
  • An approach that applies a Chinese Restaurant
    Process to perform a stochastic clustering of
    images.
  • An approach that utilizes WordNet to compute
    appropriate semantic similarity measures.

18
19
Approaches Second era, 2013 (2/4)
2013 Challenge 1 2013 Challenge 1 2013 Challenge 1
F-score NMI
Rafailidis et al. 0.570 0.873
Samangooei et al. 0.946 0.985
Schinas et al. 0.704 0.910
Vizuete et al. 0.883 0.973
Nguyen et al. 0.932 0.984
Zeppelzauer et al. 0.780 0.940
Sutanto et al. 0.812 0.954
Wistuba et al. 0.878 0.965
Papaoikonomou et al. 0.236 0.664
Gupta et al. 0.142 0.180
Brenner et al. 0.780 0.712
  • Results are better that in the 2 previous years
    (probably because a filtering step is not
    required)
  • The best approach computes one affinity matrix
    per modality, averages them and uses the average
    for clustering as part of a DBScan or spectral
    clustering procedure.

19
20
Approaches Second era, 2013 (3/4)
  • For the second challenge, all approaches adopt a
    classification procedure.
  • They differ in the set of features that they
    utilize. For instance
  • One approach utilizes scalable Laplacian
    Eigenmaps to obtain in a semi-supervised manner,
    an appropriate representation of the images.
  • Another approach used semantic similarity
    features based on WordNet.

20
21
Approaches Second era, 2013 (4/4)
2013 Challenge 2 2013 Challenge 2 2013 Challenge 2
F-Score (per category) F-Score (Event/Non-event)
Schinas 0.334 0.716
Nguyen 0.449 0.854
Sutanto 0.131 0.537
Brenner 0.332 0.721
  • The best performing approach uses an SVM
    classifier and a very rich set of textual
    features, including a set of ontological features
    (visual features are not used).

21
22
Outlook for the SED task
  • Remarkable number of participants in the last
    year and appearance of quite novel approaches.
  • The SED task is organized in 2014 as well! Three
    challenges this year
  • Full clustering.
  • Retrieval / filtering.
  • Summarization / labelling of events.
  • Registration opens soon!

22
23
Outlook for the problem of social event detection
  • We havent seen any approach for dealing with the
    problem of social event detection into the
    wild
  • Examined image collections so far had a high
    ratio of event to non-event photos the
    application to a random collection of images
    would most likely produce poor results.
  • Classification of images as event or non-event
    related is important for dealing with the more
    general scenario.
  • Additionally, accurate event/non-event
    classifiers may assist for obtaining more focused
    crawling mechanisms.
  • Combination of agnostic approaches (such as
    clustering) and approaches that utilize event
    directories.
  • More extensive usage of visual content, rather
    than mostly of metadata.

23
24
Acknowledgments
24
About PowerShow.com