IEG7007 Multimedia Coding and Processing

1
IEG7007 Multimedia Coding and Processing

• Tuesday 645 pm to 945 pm
• Innovation Centre Room 23

2
Instructors

• Lecturer Dr. Philip Tse
• Email kctse_at_ie.cuhk.edu.hk
• Tutor Mr. Lee Kai Ki
• MSN leekaiki_at_hotmail.com

3
About You

• Yourself
• First year MSc students
• Work
• Come after work gt tired and hungry, need some
  rest
• Expert experience in your working area gt
  diversity
• Family friends
• To take care gt go home ASAP
• Expectation
• Earn the MSc in two years gt dont fail
• Learn new technologies gt advance in career
• Make new friends gt social network

4
About this course

• Course content
• Source of Materials
• Assessments
• Course Information
• http//www.ie.cuhk.edu.hk/courses/IEG7007/index.h
  tml
• Lecture style

5
Course Contents

• Introduction (1 week)
• Characteristics of multimedia data
• Multimedia applications
• Multimedia request streams
• Coding techniques (2 weeks)
• Data representations
• Sampling and quantization
• Processing techniques (3 weeks)
• Image processing
• Video processing
• Compression techniques (3 weeks)
• Image compression JPEG, JPEG2000
• Video compression MPEG, H.261
• Advanced topics (2 weeks)
• Multimedia storage organizations
• Multimedia web caching
• Summary (1 week)

6
Source of Material (1)

• Tim Morris, Computer Vision and Image Processing,
  Palgrave Macmillan, New York, 2004.
• K.R. Rao, Z.S. Bojkovic, D.A. Milovanovic,
  Multimedia Communication Systems Techniques,
  Standards, and Networks, Prentice-Hall, 2002.
• Yao Wang, Jorn Ostermann, Ya-Qin Zhang, Video
  Processing and Communications, Prentice-Hall,
  2002.

7
Source of Material (2)

• Ian Witten, Alistair Moffat, Timothy Bell,
  Managing Gigabytes Compressing and Indexing
  Documents and Images, 2nd Ed., Morgan Kaufmann,
  1999.
• Harald Kosch, Distributed Multimedia Database
  Technologies Supported by MPEG-7 and MPEG-21, CRC
  Press, 2004.
• Dinkar Sitaram and Asit Dan, Multimedia Servers
  Application, Environment, and Design, Morgan
  Kaufmann, 2000.
• Research papers in journals and international
  conferences.

8
Assessments

• 10 on assignment 1
• Release on week 3
• Due on week 5
• 10 on midterm test
• Test on week 7
• 10 on assignment 2
• Release on week 9
• Due on week 11
• 70 on final written examination
• 2 hours
• One A4 open note

9
No Plagiarism

• No Plagiarism
• Dont do it.
• You come to school to learn, not to cheat.
• Penalty includes failing the course and receiving
  demerits
• Read this web site
• http//www.cuhk.edu.hk/policy/academichonesty/

10
Lecture Style

• Presentation
• Explanations
• Examples
• Small group discussions
• Exercises
• Time division
• 5 minutes relaxation
• 60 minutes lecture
• 10 minutes break
• 60 minutes lecture
• 5 minutes break
• 10-20 minutes discussions, or exercises

11
Course Progress

• Introduction
• Multimedia (MM) Data
• Multimedia Applications
• Coding Techniques
• Processing Techniques
• Compression Standards
• Advanced Topics

12
What are computer data?

• Computer process data in binary bits of 1 or 0
  ONLY.
• Any other data must be converted to binary bits
  for any processing.
• High level processing must be performed on the
  binary bits level.
• E.g. 23 00102 00112 01012.
• E.g. INTO 494E16 544F16 494E544F16
  INTO.

13
What are multimedia data?

• Traditional data represent the logical meaning
  only of real world entities in computers
• Numbers (1,2,3,4, ) to represent values
• Text (A,B,C,D, ) to represent textual
  information
• Drawings to represent information graphically

14
What are multimedia data?

• Records of real world entities
• Images are recorded on films and handled by
  photographic equipments
• Sound is recorded on cassette tapes and CD-ROM,
  transmitted by phones
• Moving images, video, is recorded on tapes and
  transported physically.
• Everything is fine except that these are ANALOG
  signals

15
What are multimedia data?

• Multimedia (multiple media) data provide a direct
  representation of real world entities in digital
  format
• Photograph, video
• Recorded speech, music
• X ray image
• Ultrasound images, Ultrasound video
• MRI images
• Artificial data that are represented similarly
• Drawings, charts, animations

16
Why digital?

• Digital data are 100 reproducible
• Precise
• Inadvertent to making copies
• Independent of the storage media
• Can be processed by computers to produce new
  software effects (blur, sharpen, change colour,
  portrait, object creation, )
• Microsoft imaging, Photoshop
• Macromedia Director
• Can be transmitted via networks

17
Multimedia Objects

• A recording of multimedia data as a unit to
  represent an entity
• E.g. image object, video object
• Example multimedia objects

18
Example Objects

• Still images
• photographs

• charts

19
Example Objects

• Audio
• speech, voice

• music, audio

20
Example Objects
Animation graphics
Video
21
Multimedia Data Summary

• Multimedia data can directly represent real world
  entities in the digital format
• Digital multimedia data can be processed by
  computer programs to produce software effects
  that are not possible before.

22
Content

• Introduction
• Multimedia (abbrev. MM) Data
• Multimedia Applications
• Coding Techniques

23
Multimedia Applications

• Digital TV
• Digital movies
• Digital music
• Digital library
• Virtual Museum
• Virtual Musician
• Sports on demand
• News on demand
• Education on demand

• Video phone
• Video conference
• Collaborative computing
• Video emails
• Internet radio
• Patient monitoring
• Security monitoring
• Earth observatory
• Quality control

24
Broadcasting

• Broadcast Video transmit audio and video
  programs via network
• Cable TV
• E.g. iCable in HK, OptusVision in Australia
• Internet live radio
• E.g. www.live-radio.net, www.rthk.org.hk
• Internet live TV program
• E.g. US Universities include Maryland, Princeton,
  Stanford, Duke, MIT,and Pennsylvania produce
  research channel on http//www.researchchannel.com
  /onair/

Trademarks
25
Video on demand (VOD)

• Sometimes called Interactive TV
• Transmit selected video and audio objects
  according to user selection
• movie-on-demand
• www.now.com, yahoo movie (trailers only)
• Educational VOD
• Virtual University, Open University
• News on demand
• CNN.com

26
Communications

• Remote communication and cooperation using video
  and audio
• Video phone (e.g. 3 uses video phone over
  broadband)
• Video Conferencing (e.g. Microsoft NetMeeting,
  CUSeeMe)
• Collaborative Computing
• Video emails
• IDD calls (e.g. www.skype.com)
• Video sharing (movie or family video)
• Napster, BitTorrent, youtube.com, etc

Trademarks
27
Monitoring and Control

• Make a 24 hours by 7 days record of a scene
• Security monitoring
• Patient monitoring
• Earth observatory by satellite
• Use the recorded information to control the
  production process
• Quality control using video monitoring
• E.g. Arnott biscuit production

Trademarks
28
Multimedia Applications

• Visual Information Systems
• Image databases,
• Video databases,
• Virtual museum, digital library e.g.
• Medical imaging system
• Games
• Interactive story telling
• Enrichments to current computer applications
• Beauty enhancement preview

29
Major Configuration Concerns

• Delivery network
• Dedicated cable
• Internet
• Delivery scheduling
• Broadcasting
• Video on demand (VOD)

• Data storage and distribution system
• Centralised
• Storage Area Network (SAN)
• Content Distribution Network (CDN)
• Serverless, Peer-to-peer (P2P)

30
Broadcasting over dedicated cable

• User subscribes to an Interactive TV (ITV)
  company
• ITV Company broadcast a number of channels of
  video contents via a cable and a dedicated Set
  Top Box
• User selects channel to watch via remote control
  of the Set Top Box

ITV video server
STB
Cable network
STB
31
Broadcasting over the Internet

• User subscribes to a content provider
• User joins a live video/audio channel
• Streaming server delivers the live object to all
  users
• Browser receives and plays the object

Streaming server
Internet
32
VOD over dedicated cable

• User subscribes to an ITV company
• ITV company downloads a movie list to the Set Top
  box
• User selects a movie from the list using remote
  control of set top box
• ITV Company broadcast the movie in a new channel
• New user may join an existing channel to watch

ITV video server
STB
Cable network
STB
33
VOD over the Internet

• User subscribes to a content provider
• User selects a video/audio object from the
  content provider
• Content provider test the streaming ability to
  the users computer
• Server delivers the low or high resolution object
  suitable for the delivery to the user. Browser
  receives and plays the object.

VOD streaming server
Internet
34
4 Types of Video-on-demand Systems

• Near Video-on-demand (NVOD)
• True Video-on-demand (TVOD)
• Partitioned Video-on-demand (PVOD)
• Dynamically Allocated Video-on-demand (DAVOD)

35
True Video-on-demand

• User has complete control of a multimedia program
  (functionsforward, reverse, fast forward, random
  positioning, pause, resume)
• Each user is allocated a unique channel during
  the total duration
• Allows complete user interactivity
• The number of concurrent users is limited by the
  number of available channels. Many potential
  users cannot access the system.

36
Near Video-on-demand

• Provides video distribution at relatively low
  cost
• One copy of each popular video is started every
  time interval, say 5 minutes.
• All the streams of the same video are broadcast
  at the same time on several channels
• When the user requests this video, his access
  will be delayed until the start of the next
  stream
• Limited user interactivity

37
Near Video-on-demand
start
start
end
Stream 1
Stream 2
T
Stream 3
T

Stream n
Video length
source Visual96
38
Partitioned Video-on-demand

• Combines the advantages of both NVOD and TVOD
  systems. User interactivity at the capacity of
  the system
• Digital channels are partitioned into 2 groups
  for NVOD and TVOD services
• NVOD channels for broadcasting the most popular
  video with limited user control
• TVOD channels will complete user control

39
Partitioned Video-on-demand
50 broadcast channels
450 interactive channels
NVOD 250 channels
TVOD 200 channels
source Visual96
40
Dynamically Allocated VOD

• An extension of PVOD scheme
• The user, watching a video from the NVOD list of
  most popular videos, can request the
  interactivity with the video at any time
• If a channel is available, the user will be
  switched to the TVOD group of channels which
  allows complete control.
• E.g. Split-And-Merge (SAM) protocol (Liaos
  paper).

41
Video Conferencing

• Computers are each installed with a video camera,
  microphone, and connected to the network
• One user initiates and hosts a meeting
• Other users joins the meeting
• All of them send their own video and audio
  signals to all the other users
• Users may speak, type, or draw on whiteboard.

Network
42
Video Conferencing
Place call
End call
Directory
Presentation
List of users
Whiteboard
Share program
Transfer files
Chat
43
IBM QBIC system

• The user searches for an image from an image
  database, but how can he input the image?
• Select a colour
• Sketch a shape
• Select a texture
• The database will then searches for the images
  with the closest match to the input
• User picks a few relevant images
• The database then selects the images with
  features like the relevant images

44
Multimedia Applications Summary

• Multimedia can be used in many types of
  applications such as
• Broadcasting, video on demand,
• Communications, monitoring and control
• Information systems
• The design of multimedia system needs to consider
  the storage system, delivery network, and
  scheduling approach
• Most of these systems require the storage of
  large multimedia objects in a storage system for
  future retrievals

45
Break

• Know your classmates
• Name
• Current job
• Contact
• Career direction
• Technical, management, run a company, etc
• Your expectations from this course
• Personal understanding
• Academic degree
• Job promotion, new career
• All of the above
• Others email to me (does not mean it will be
  fulfilled)

46
Course Progress

• Introduction
• Data Representation
• Processing Techniques

47
Data Representation

• Number and text
• Graphics and animations
• Sound and Audio
• Images Representation
• Video Representation

48
Data Representations

• Multimedia data types include
• Numbers and text
• Graphic and animation
• Image
• Sound
• Video
• others.

49
Data Representations

• A computer can only store process data of
  binary bits
• A bit can represent a 1 or 0.
• How to represent each media type as digital data?
• More bits can represent more information

50
Numbers

• Numbers
• For positive integers, use the base 2 instead of
  the base 10 to handle numbers
• For negative integers, use 1s complement.
• More bits can represent larger integer numbers

51
Numbers

• Integers
• Number 6 is represented as 0110 to the base 2.
• Number -6 is represented as 10011 1010.
• 6(-6) can directly be added to get 6(-6)
  11111 0000 and ignoring the overflow.

52
Numbers

• Integers
• 4 bits can represent the range 0, 15 or -8,
  7.
• 8 bits can represent range 0, 255 or -128,
  127.
• 16 bits can represent the range 0, 65535 or
  -32768, 32767.
• When the number of bits is increased by 1, the
  range of numbers is doubled.

53
Numbers

• Real Numbers
• Represented by mantissa and exponent.
• The mantissa is a number between 0 and 1.
• The exponent raises the number to the power of 2.
• More bits in the mantissa increases the
  precision.
• More bits in the exponent increases the range of
  numbers.

54
Numbers

• Real numbers
• Each digit at the base of 2 represents the digit
  multiplied by the digit position at the power of
  2.
• E.g. 11.012
• 121 120 02-1 12-2
• 2 1 0 0.25
• 3.25

55
Numbers

• Real numbers
• By multiplying the decimal number by 2 repeatedly
  and take the unit digit, we can get the real
  number representation.
• E.g. pi (p)3.1415926.
• 3 is represented as 11.
• 0.14159262 0.2831852
• 0.28318522 0.5663704
• 0.56637042 1.1327408
• 0.13274082 0.2654816
• Thus, the mantissa is 110010 and the exponent is
  10.

56
Numbers

• Real numbers
• A real number represented with mantissa110010
  exponent10 is
• Thus, this real number is 3.125.

57
Text

• Text
• Use a binary number for each character
• ASCII code and ECBDIC code
• ASCII code is the most commonly used in computers
  nowadays
• The English characters and symbols are encoded in
  7-bits only.
• The Latin characters can also be encoded in 8
  bits.
• Up to 256 characters can be encoded using 1 byte
  (8 bits).

58
Text

• Text
• In ASCII code, 0100 0001 means A, 0100 0010
  means B, , 0101 1010 means Z
• Upper case characters
• A is hex 41, B is hex 42, space is 20
• Lower case characters
• a is hex 61, b is hex 62
• Numbers
• 0 is hex 30, 1 is hex 31
• Symbols
• Check the symbol table.

59
Text

• Text
• Some languages like Chinese uses more than 256
  characters. Thus, more bits are needed to encode
  each character.
• For multilingual purpose, the Universal Codes
  (Unicode) uses double bytes to encode each
  character or symbol.
• Up to 65536 characters or symbols can be encoded
  in 16 bits.

60
Text

• Text
• A word is a string of characters. Thus, it is
  represented as a list of ASCII codes.
• E.g. Hello is represented as hex 48 65 6C 6C
  6F

61
Operations on Text

• Concatenation
• Hello , world Hello, world
• Change case
• Hello, world is changed to HELLO, WORLD
• Convert to number
• 123 hex 31 32 33
• is converted to the number 12310 or hex 7F.

62
Graphics

• Each position on the screen is specified as a
  coordinate (x,y).
• X-axis from left to right and y-axis from top to
  bottom.
• E.g. In a 800 x 600 screen,
• the top left corner is (0, 0)
• the top right corner is (800, 0)
• the bottom right corner is (800, 600)
• the bottom left corner is (0 ,600)

63
Simple Graphics

• A line A pair of coordinates.

(x2, y2)
(x1,y1)
(x3, y3)

• A curve A list of coordinates of start, turning
  points, and end.

(x2, y2)
(x4, y4)
(x1,y1)
64
Simple Graphics
length
(x1,y1)

• A rectangle The coordinates of the top left
  corner, length and width of the rectangle.

width
65
Animation

• A computer uses graphic tools to provide visual
  effects in a frame buffer
• The frame buffer is changed continuously by the
  program. These changes are repeatedly drawn on
  the display to appears as continuous motion

Display
Update Frame
Frame Buffer
66
Animation Speed

• Animation 15-20 changes/sec
• approx. 50 millisec.
• Animation may run too fast or jerky on computers