Discussion on Video Analysis and Extraction, MPEG-4 and MPEG-7 Encoding and Decoding in Java, Java 3D, or OpenGL

1
Discussion on Video Analysis and Extraction,
MPEG-4 and MPEG-7 Encoding and Decoding in Java,
Java 3D, or OpenGL

• Presented by
• Emmanuel Velasco
• City College of New York

2
Video Analysis and Extraction

• As more videos are being created, digitized and
  archived, the need for content-based search and
  retrieval is necessary. This involves analyzing a
  video and extracting its contents.
• The videos are cut into frames. The frames are
  analyzed and the objects can be extracted using
  image processing techniques.

3
Video Analysis and Extraction

• Temporal Video Segmentation
• Cut detection The changes in the contents are
  visible and occur instantaneously between
  consecutive frames.
• Gradual transition detection The image
  transition makes gradual changes. This requires
  multiple frames to be analyzed. Gradual
  transitions include fade in, fade out, wipe and
  dissolve.

4
Video Analysis and Extraction

• Examples
• Cut transition
• Gradual transition

5
Video Analysis and Extraction

• The cut transition is easier to detect. We check
  the frame differences between two consecutive
  frames and see if the difference is greater than
  a certain threshold. If it is, then a cut is
  determined.
• Gradual transitions are harder to detect. There
  are several methods, which include the
  twin-comparison algorithm. This works by noticing
  that the first and last transition frames are
  different, and any consecutive frames between
  them are similar.

6
Video Analysis and Extraction

• Twin-Algorithm Results

7
Video Analysis and Extraction

• Scene and Object Detection
• We want to identify objects in a video. One
  method of finding this is the opposite of
  transition detection. Instead of finding the
  differences between frames above a threshold, we
  want to find image regions below a certain
  threshold.
• Another method is to take an image and try all
  possible transformations between the edges of the
  two images.

8
Video Analysis and Extraction

• Text Extraction
• We want to retrieve the captions in an video.
  While most text segmentation is done on high
  resolution media, video is low resolution.
• One method is to assume that the gray levels of
  the text is lighter or darker than the
  background. Using a minimum difference with the
  background, the text can be extracted.

9
Video Analysis and Extraction

• Example of Text Extraction

10
Video Analysis and Extraction

• So we see that video analysis and extraction is
  useful in our projects.
• The Classroom Project
• Object detection is used for finding the
  location of the professor.
• Text extraction is useful for capturing text in
  the PowerPoint slides shown in a video.

11
Video Analysis and Extraction

• The NYC Traffic Project
• Object detection is used for detecting how heavy
  or light the traffic is.
• Transition detection is used to see if we are
  looking at the same view, or if the view has
  changed.

12
MPEG-4

• Is an ISO/IEC compression standard created by the
  Moving Pictures Expert Group (MPEG).
• Has been successfully used in
• digital television
• interactive graphics applications
• interactive multimedia

13
MPEG-4

• Can bring multimedia to new networks such as
  mobile networks.
• Media objects are audio, video, or audiovisual
  contents and can be natural (recorded using a
  camera and/or microphone) or synthetic (generated
  using a computer).

14
MPEG-4

• An example of an MPEG-4 scene.

15
MPEG-4

• The media objects are independent from their
  background. This allows easy extraction of the
  object and easier editing of an object.
• The objects are synchronized by time and space.

16
MPEG-4

• With a set of media objects, MPEG-4 allows us to
• place objects anywhere in a given coordinate
  system.
• apply transforms to change an visual object
  geometrically or change an audio object
  acoustically.
• group objects together (such as the visual image
  of the person, and their voice).
• apply streamed data to media objects to modify
  their attributes.
• change the users viewpoint or listening point
  anywhere in the scene.

17
Encoder / Decoder Definitions

• Encoder To format (electronic data) according to
  a standard format.
• Decoder to recognize and interpret (an
  electronic signal)

18
MPEG-4 Encoder / Decoder

• While many MPEG-4 encoders and decoders exists
  as standalone applications, we want to be able to
  encode and decode using Java, Java 3D, or OpenGL.

19
MPEG-4 Encoder / Decoder

• IBM Toolkit for MPEG-4 is a set of Java classes
  and API with five applications.
• AVgen a simple, easy-to-use GUI tool for
  creating audio/video-only content for ISMA- or
  3GPP-compliant devices
• XMTBatch a tool for creating rich MPEG-4 content
  beyond simple audio and video
• M4Play an MPEG-4 client playback application
• M4Applet for ISMA a Java player applet for
  ISMA-compliant content
• M4Applet for HTTP a Java applet for MPEG-4
  content played back over HTTP.

20
MPEG-4 Encoder / Decoder
Add media object

• IBM MPEG-4 XMT Editor Tool

Object Attributes
Time Frame
21
MPEG-4 Encoder / Decoder

• IBM MPEG-4 Demos
• http//www.research.ibm.com/mpeg4/Demos/DemoSyste
  ms.htm
• SKLMP4 Encoder / Decoder
• is a C library that is capable of encoding and
  decoding MPEG-4
• http//skal.planet-d.net/coding/mpeg4codec.html

22
MPEG-4

• MPEG-4 can make it easier for us to extract the
  objects since each object is independent of each
  other.
• The Classroom Project
• The professor is an image object, separated from
  the PowerPoint background.

23
MPEG-4

• The NYC Traffic Project
• The background (roads) are separate from the
  objects (cars).
• The interactivity that MPEG-4 allows can make
  the user interface easier to interact with. They
  can point and click on the map and view the
  cameras in that location.

24
MPEG-7

• Since audiovisual data is increasing and coming
  from many different sources, searching for a
  certain type of media content will be more
  difficult. Therefore we need a way to search the
  data quickly and efficiently. The solution is
  MPEG-7.
• MPEG-7 is a standard for describing media
  content. Unlike MPEG-1, MPEG-2, and MPEG-4,
  MPEG-7 is not a standard for the actual coding of
  moving pictures and audio.

25
MPEG-7

• MPEG-7 uses XML Schema as the language of choice
  for content description.
• These descriptions may include information
  describing the creation of the content (title,
  author). It may include the storage features of
  the content (storage format, encoding). It can
  contain low level features in the content (color,
  texture, shape, motion, audio).

26
So what will MPEG-7 standardize?

• A set of descriptors (D) Descriptors define the
  syntax and the semantics of each feature
  (metadata element).
• A set of description schemes (DS) A description
  scheme specifies the structure and semantics of
  the relationships between its components.

27
So what will MPEG-7 standardize?

• Description Definition Language (DDL) to define
  the syntax of the descriptors and description
  schemes.

28
Some possible MPEG-7 Applications

• Audio play a few notes on the keyboard, and it
  will return musical pieces with similar tunes.
• Graphics sketch a few lines on a screen and get
  a set of images containing similar graphics or
  logos .
• Images define objects, color patterns or
  textures and retrieve images that look like the
  image described.

29
MPEG-7 Encoder / Decoder

• MPEG-7 Library is a set of C classes,
  implementing the MPEG-7 standard.
• http//iis.joanneum.at/mpeg-7/overview.htm
• Java MPEG-7 Audio Encoder is a java library that
  provides a MPEG-7 audio encoder to describe an
  audio content with some descriptors of the MPEG-7
  standard.
• http//www.ient.rwth-aachen.de/team/crysandt/soft
  ware/mpeg7audioenc/

30
MPEG-7

• Once we have a lot of media contents, MPEG-7
  allows us to search through them easier.
• The Classroom Project
• If we have a lot of videos, sound, or both. We
  can find the content we need quickly.
• The NYC Traffic Project
• If there are many cameras at several locations,
  finding a specific location can be easier.

31
Discussion on Video Analysis and Extraction,
MPEG-4 and MPEG-7 Encoding and Decoding in Java,
Java 3D, or OpenGL

• Presented by
• Emmanuel Velasco
• City College of New York