CS257 Modelling Multimedia Information LECTURE 1

1
CS257 Modelling Multimedia InformationLECTURE 1

• Dr Andrew Salway, (a.salway_at_surrey.ac.uk)
• Tuesday 17th January

2

• What does Multimedia mean?

3
(No Transcript)
4
(No Transcript)
5
What is so special about DIGITAL multimedia?

• After all, cave paintings, writing, singing,
  opera, libraries, photography, cinema, radio,
  television, shoeboxes of photos, video rental
  stores, etc. have been around for quite some
  time

6
What is special about DIGITAL multimedia?

• SOME ANSWERS
• Different kinds of information are stored and
  shared in the same way ? facilitates the
  combination of different kinds of media
• The multimedia information can be computed, i.e.
  systems can act on the information
• Multiple and remote access to information is
  possible
• Multiple indices can be maintained vs. static
  categorisation on a library shelf
• Personalised presentation of and interaction with
  information
• It is easier and easier to create media producer
  ? more and more multimedia information is being
  created

7
Overview of LECTURE 1

• Introduction to Multimedia Information Modelling
• Example applications for indexing, retrieving and
  browsing multimedia data
• Why do we need to add metadata to multimedia
  data?
• Basic modelling examples analysis of some simple
  multimedia scenarios
• Module Overview how CS257 will run

8
Multimedia Data Includes

• Text (email, txts, news, scientific papers,
  technical documents, literature, etc.)
• Image (personal photos, news, paintings, medical
  images, etc.)
• Video (home video, TV, film, etc.)
• Audio (speech, radio, music, etc.)
• Graphics / Virtual Reality
• PLUS, combinations of all of the above
  PowerPoint presentations, CD-ROMs, web-pages,
  interactiveTV, DVDs, etc.

9
Where are we at?

• Its becoming easier and easier to create,
  distribute and view multimedia data
• BUT
• As the amount of multimedia data increases, it
  becomes harder to find what you want
  information explosion
• Multimedia information only has value if it can
  be found (retrieved) and used

10
Key Terms

• Indexing literally pointing to an index
  points to where information
• User information needs the reason someone is
  looking for multimedia information
• Retrieval accessing multimedia data by making a
  query and getting back a set of results
• Browsing accessing multimedia data by perusing
  results sets / summaries and by following
  hypermedia links between documents

11
Multimedia applications for indexing, retrieving
and browsing multimedia data

• Personal Media Collections
• aka Memories for Life
• Media in the Home
• Media on Intranets
• Media on the Internet
• For entertainment and leisure, arts and culture,
  science, education, business, etc

12
(No Transcript)
13
Retrieving Multimedia Data

• Can be hard to keep track of your own multimedia
  data files, never mind accessing collections
  compiled by other people
• How do you find the text / image / video / audio
  files that you want?

14
Personal Image Collections
Picassa allows you to add keywords to your
images, for keyword-search, www.picasa.com
15
Online Music Collections
iTunes lets you search through and organise music
files by Song Names, Artists, Albums, Genres -
www.apple.com/itunes/
16
Searching an Organisations Information

• Autonomy provides technology to index/retrieve,
  browse and summarise multimedia documents held by
  an organisation www.autonomy.com.

17
Media across the Internet
18
Media across the Internet
19
Multimedia applications for indexing, retrieving
and browsing multimedia data

• Picassa, iTunes, Autonomy, Google and SingingFish
  are examples of the many systems now available
  for indexing / retrieving / browsing multimedia
  data
• They all allow users to make queries which are
  then matched against stored descriptions of the
  media, aka metadata
• Why do we need metadata?

20
Digitisation

• rich variety of sensory inputs that make up
  images, text, moving pictures and sounds must be
  reduced to patterns of binary digits inside a
  computer
• Chapman and Chapman (200034)

21
Digitisation

• The power of multimedia systems originates in
  their ability to represent disparate information
  as a bit stream every form of representation,
  from video to text, can be stored, processed and
  communicated using the same device a computer.
• Jain (1994), IEEE Multimedia vol. 1(2), p.3

22
Devices for Digitisation

• Increasingly media is being created digital in
  the first place but if you have some non-digital
  media then
• Text scanners OCR ? .txt, .html
• Images scanners cameras Photoshop ? .gif,
  .jpg
• Audio microphones MIDI devices ? .wav, .midi,
  .mp3
• Video video cameras capture cards Premiere ?
  .mpg, .avi
• (For further reading see Covell 2000139-154
  Chapman and Chapman 200053)
• Data displayed on
• PCs Digital TVs media centres DVD / MP3
  players 3G / WAP phones, etc

23
(No Transcript)
24
11101011000010101001010000101000010000100100000101
00010101000010000001110010100100111010110000101010
01010000101000010000100100000101000101010000100000
01110010100100111010110000101010010100001010000100
00100100000101000101010000100000011100101001001110
10110000101010010100001010000100001001000001010001
01010000100000011100101001001110101100001010100101
00001010000100001001000001010001010100001000000111
00101001001110101100001010100101000010100001000010
01000001010001010100001000000111001010010011101011
00001010100101000010100001000010010000010100010101
00001000000111001010010011101011000010101001010000
10100001000010010000010100010101000010000001110010
10010011101011000010101001010000101000010000100100
00010100010101000010000001110010100101110101100001
01010010100001010000100001001000001010001010100001
00000011100101001110101100001010100101000010100001
00001001000001010001010100001000000111001010010011
10101100001010100101000010100001000010010000010100
01010100001000000111001010010011101011000010101001
01000010100001000010010000010100010101000010000001
11001010010011101011000010101001010000101000010000
10010000010100010101000010000001110010100100111010
11000010101001010000101000010000100100000101000101
01000010000001110010100100111010110000101010010100
00101000010000100100000101000101010000100000011100
10100100111010110000101010010100001010000100001001
00000101000101010000100000011100101001001110101100
00101010010100001010000100001001000001010001010100
00100000011100101001001110101100001010100101000010
10000100001001000001010001010100001000000111001010
01011101011000010101001010000101000010000100100000
10100010101000010000001110010100111010110000101010
01010000101000010000100100000101000101010000100000
01110010100100111010110000101010010100001010000100
00100100000101000101010000100000011100101001001110
10110000101010010100001010000100001001000001010001
01010000100000011100101001001110101100001010100101
00001010000100001001000001010001010100001000000111
00101001001110101100001010100101000010100001000010
01000001010001010100001000000111001010010011101011
00001010100101000010100001000010010000010100010101
00001000000111001010010011101011000010101001010000
10100001000010010000010100010101000010000001110010
10010011101011000010101001010000101000010000100100
00010100010101000010000001110010100100111010110000
10101001010000101000010000100100000101000101010000
10000001110010100101110101100001010100101000010100
00100001001000001010001010100001000000111001010011
10101100001010100101000010100001000010010000010100
01010100001000000111001010010011101011000010101001
01000010100001000010010000010100010101000010000001
11001010010011101011000010101001010000101000010000
10010000010100010101000010000001110010100100111010
11000010101001010000101000010000100100000101000101
01000010000001110010100100111010110000101010010100
00101000010000100100000101000101010000100000011100
10100100111010110000101010010100001010000100001001
00000101000101010000100000011100101001001110101100
00101010010100001010000100001001000001010001010100
00100000011100101001001110101100001010100101000010
10000100001001000001010001010100001000000111001010
01001110101100001010100101000010100001000010010000
01010001010100001000000111001010010111010110000101
01001010000101000010000100100000101000101010000100
00001110010100111010110000101010010100001010000100
00100100000101000101010000100000011100101001001110
10110000101010010100001010000100001001000001010001
01010000100000011100101001001110101100001010100101
00001010000100001001000001010001010100001000000111
00101001001110101100001010100101000010100001000010
01000001010001010100001000000111001010010011101011
00001010100101000010100001000010010000010100010101
00001000000111001010010011101011000010101001010000
10100001000010010000010100010101000010000001110010
10010011101011000010101001010000101000010000100100
00010100010101000010000001110010100100111010110000
10101001010000101000010000100100000101000101010000
10000001110010100100111010110000101010010100001010
00010000100100000101000101010000100000011100101001
01110101100001010100101000010100001000010010000010
10001010100001000000111001010011101011000010101001
01000010100001000010010000010100010101000010000001
11001010010011101011000010101001010000101000010000
10010000010100010101000010000001110010100100111010
11000010101001010000101000010000100100000101000101
01000010000001110010100100111010110000101010010100
00101000010000100100000101000101010000100000011100
10100100111010110000101010010100001010000100001001
00000101000101010000100000011100101001001110101100
00101010010100001010000100001001000001010001010100
00100000011100101001001110101100001010100101000010
10000100001001000001010001010100001000000111001010
01001110101100001010100101000010100001000010010000
01010001010100001000000111001010010011101011000010
10100101000010100001000010010000010100010101000010
00000111001010010111010110000101010010100001010000
1000010010000010100010101000010000001110010100

25
The need for metadata

• You dont want to search for a painting, or other
  kinds of multimedia data, with a query like
  00110010001000001
• We need metadata to describe the multimedia data
  users queries will then be matched against
  metadata (which is associated with the multimedia
  data)

26
Metadata

• Metadata data about data
• In this case, data about the bit-streams that are
  texts, images, video, audio,etc.
• By including metadata then multimedia
  applications can know about media content, as
  well as about other things like who owns it, when
  it was made, who made it, etc.

27
Modelling Multimedia Information
Modelling Multimedia Information Analysing a
multimedia scenario (data and user information
needs) to determine what metadata needs to be
associated with the multimedia data for users to
find the information they need.
Design and Implementation of Multimedia
databases and applications for retrieving and
browsing multimedia data
Multimedia Data Text, image, video, audio data
files
Users and User Information Needs
28
A simple multimedia scenario

• Think about a video-on-demand (VoD) system allows
  users to find films they want to watch and
  downloads them to their homes
• What metadata should be attached to each video
  data file, i.e. to each film, to enable searching
  in this scenario?

29
Modelling Films for VoD

• They are many things you can describe about a
  film, so think about the kinds of queries a VoD
  user might want to make
• Find me films made by Steven Spilberg
• Find me horror films
• Find me films made in 2003
• Find me films starring Brad Pitt
• These suggest that metadata attributes such as
  Director, Genre, Year and Actors will be
  important in this scenario

30
Modelling Films for VoD

• We could use an E-R diagram as part of the
  modelling

FILM
Title Director Genre Year Actors
31
A Simple Database for VoD

• Note, the filename is included as an attribute
  in the Table in order to associate the metadata
  with the multimedia data, in this case video data
  stored as .mpgs
• The user can search on Title / Director / Genre
  / Year / Actors attributes and the system can
  return them the appropriate video data

32
Analysis for Multimedia Applications

• Generally, you need to ask
• What kinds of multimedia data?
• What are the users information needs? What
  kinds of queries are they likely to want to make?
  
• What attributes can be described for this kind of
  multimedia data? Which of these attributes would
  be useful as metadata given the user information
  needs?

33
Analysis for Multimedia Applications

• As with other kinds of modeling, there is
  probably never one correct way to do it you
  should always be prepared to explain and justify
  the modeling decisions you make

34
Set Exercise 1-1 another multimedia scenario

• Guildford art gallery has digitised its
  collection of paintings and wants to make them
  available for searching online.
• What metadata would you specify for
• The general public to search for paintings?
• Art experts to search for paintings?

35
Set Exercise 1-2what is metadata?

• Skim read the following webpages and extract /
  summarise 2-3 definitions of what metadata is and
  3-4 reasons why people say metadata is important.
• http//dublincore.org/documents/usageguide/whatis
  metadata
• www.tasi.ac.uk/advice/delivering/metadata.html
• www.openp2p.com/pub/a/p2p/2001/01/18/metadata.html
  
• You do NOT need to read these articles in detail

36

• We will discuss your answers to exercises 1-1
  and 1-2 at the start of Lecture 2.

37
LECTURE 1LEARNING OUTCOMES

• After this lecture, and set exercises, you should
  be able to
• Give examples of multimedia data and its uses in
  todays world
• Explain the need for metadata to support a
  variety of multimedia applications, especially
  for retrieval and browsing of multimedia data
• Analyse / model a simple multimedia scenario,
  i.e. given some multimedia data and user
  information needs, you should specify appropriate
  metadata and explain / justify your decisions

38
Overview of CS257

• Aim
• Learning Outcomes
• How the module will run
• Lectures and labs
• Assessment
• Reading
• Outline of content
• Module web-page

39
Aim of CS257

• This module aims to provide students with a
  understanding for the role of modelling
  multimedia information (text, image, video and
  audio data) in a variety of multimedia
  information retrieval and browsing applications,
  including personal media collections,
  organisation-wide media archives, and web-search
  engines. The module also aims to give students
  practical skills for modelling multimedia items
  appropriately for a given application.

40
Learning Outcomes

• At the end of the module students should be able
  to
•  
• 1.   Apply a standard metadata set to describe
  media items in an archive
• 2.   Apply appropriate techniques for modelling
  text documents
• 3. Compare and apply different kinds of
  metadata for image data
• 4.   Distinguish and apply appropriate data
  models for temporal media (video/audio)
• 5.   Describe video content formally, in terms of
  objects, events and temporal relationships
  between events
• 6.    Design and implement a synchronised
  multimedia presentation including a variety of
  media types and temporal relationships
• 7.       Explain the theory behind hypermedia
  systems and assess how this is put into practice
  in current hypermedia applications
• 8. Explain, contrast and evaluate the modelling
  of multimedia information to support media
  access, especially retrieval and browsing, in a
  variety of applications - these include personal
  media collections, organisation-wide media
  archives and web-based search engines

41
How the module will run

• Lectures 9-11am on Tuesdays. Reading the slides
  should not be a substitute for attending.
• Labs split into groups 1 hr per group for
  some practical exercises, and for feedback on
  your coursework
• lab exercises will give good preparation for
  your coursework

42
How the module will run

• Assessment
• 60 coursework the coursework will relate to
  Lectures 1-6
• 40 unseen written exam the exam will relate to
  all the module
• lab exercises will give good preparation for
  your coursework

43
How the module will run

• Resource based, independent learning
• Consider spending about 2 hours per week revising
  lecture notes and doing exercises / reading
• Resources include
• Lectures and lecture slides, and additional notes
• Exercises
• Labs and lab exercises
• Optional reading
• Your lecturer (a.salway_at_surrey.ac.uk office
  hours 4-6pm Tuesday)
• Teaching assistant - Andrew Vassiliou
  (csp1av_at_surrey.ac.uk) who will be available
  one-hour per week for questions

44
How the module will run

• Optional Reading
• There is not a set textbook recommended for the
  module, and there is very little essential
  reading beyond the lecture notes. However, there
  will be some optional reading given to help you
  get a deeper understanding of particular topics.
  This reading will normally be available online,
  from the Library Article Collection, or in the
  Librarys Main Collection
• What does optional reading mean?
• It means that you will find more detail, and / or
  alternative explanations for the topics covered
  in the lecture.
• The alternative explanations may be useful to
  help you understand the lecture better.
• The extra detail may be appropriate if you are
  doing coursework in this area, or if you are just
  interested!
• But, be careful not to overload yourself with
  Optional Reading

45
Outline of Module Content

• Week 1 Introduction and Module Overview
• Week 2 Multimedia Data and Generic Metadata
  Standards
• Week 3 Modelling Text Data and Keywords
• Week 4 Modelling the Content of Images
• Weeks 5-6 Modelling the Content of Temporal
  Media Video Data and Audio Data
• Week 7 Combining Media Items Synchronised
  Presentations
• Week 8-9 Combining Media Items Hypermedia
• Weeks 10 REVISION lecture

46
Module web-page

• www.computing.surrey.ac.uk/personal/pg/A.Salway/cs
  257/cs257_home.htm
• Slides will be available
• in advance of the lecture
• Printing slides might
• work best printed as
• Grayscale

47
NOTE

• No labs today
• We will discuss Set Exercises 1-1 and 1-2 at the
  start of the next lecture