The Twilight of Videotape: New Ways to Migrate Video

1
The Twilight of Videotape New Ways to Migrate
Video

• Uncompressed, Lossy, and Lossless Compression in
  Digital Media Formats

Jim LindnerManaging MemberMedia Matters, LLC
2
Video as Data

• This session focuses not on the how of storage,
  but the what of storage.
• Results of Study
• Dance Heritage Coalition project Digital Video
  Preservation Reformatting Project

3
What is the Impact of Compression on Video

• Technical issues relating to Compression have the
  potential to
• Cause changes (artifacts) to the piece that may
  or may NOT be within the artistic intent of the
  piece
• Have the potential to create new classes of
  masters and copies at different resolutions and
  quality levels

4
There is No Free Lunch

• Compression saves valuable resources but there
  are almost always tradeoffs
• Loss of Information / Quantitative
• How critical was the loss?Loss of QualityDid
  anyone Notice?Time / Processing Power to
  Encode/DecodeSoftware Encoders are slower then
  Hardware EncodersGenerally real time is fast
  enough but may not be as efficient

5
Case Study for Video Preservation File Format

• Dance Heritage CoalitionPreservation File Format

6
Dance Heritage CoalitionTowards a Preservation
File Format

• Partially Funded by a Grant from the Mellon
  Foundation
• Discussion of Electronic Media Preservation
  Specifically Video and Audio

7
Test Objectives

• Quality
• Usability
• Preservability

8
Test Objective Quality

• The quality of the picture and sound, including
  resolution, croma bandwidth, luminance, sync, and
  a lack of phase shifts. A copy will pass the
  quality test if the measurement of these elements
  shows little or no diminishment or degradation
  when compared to the measurements of the original.

9
Test Objective Usability

• The usability of the end product or resulting
  preservation master copy or working copies made
  from that master must support the following
  performance measures
• a. It must be possible to edit the copy.
• b. The copy must retain any information that
  allows users
• to run processes on the footage, such as
  search
• engines.
• c. The copy must allow output that can produce an
  HDTV
• copy.
• d. The copy must permit tape-to-film transfer,
  and must
• allow freeze framing. (Freeze frame
  capability is
• important for the dance community, where
  users must
• be able to view single frames clearly to
  study
• choreographic details.)

10
Test Objective Preservability

• Preservability of the end product (i.e., end
  product must be migratable and must avoid
  technical protection, such as encryption). The
  format must be open source, public, well
  documented, and should carry no fee or very low
  fees.

11
Method

• Selected 22 Clips for Analysis
• Represent Diverse Motion and footage with that we
  believe may be problematic
• Prepare for Analysis
• Transferred uncompressed to .AVI using strict
  control over levels
• Compress each clip using 6 different Codecs
• Analyize using 14 Different Metrics

12
Exhaustive Analysis

• Several Weeks of Computer Time
• 22 Clips x
• 6 Compression Types x
• 14 Different Metrics
• 1848 Different Clips Analyzed

13
Clips
14
Clips

• Jacobs Pillow
• 1992 Gala Ted Shawn Thtr. Presentation
• Hi-8

15
Clips

• NYPL
• CATHY WEIS PROJECTS
• NOVA PRODUCTIONS FROM SKOPJE, MACEDONIA
• NOT SO FAST, KID!
• Excerpt from Show MeThe Kitchen, New York City,
  January 11, 2001
• DVCam

16
Clips

• NYPL
• Choreography and Text by NEIL GREENBERG
• Performed by ELLEN BARNABY, CHRISTOPHER
  BATENHORST,NEIL GREENBERG, JUSTINE LYNCH, JO
  MCKENDRY
• NOT-ABOUT-AIDS-DANCE Excerpt Performed by Dance
  by Neil Greenberg
• The Kitchen, New York, December 15, 1994
• ¾" Umatic

17
Compression Types
18
.mov files

• sorenson video 3, 640 x 480, millions, 29.97
  fpsinterlaced bottom field firstKey frame every
  300 framesaspect ratio 43bitrate limit 1200
  kbpsspatial quality 50image smoothing on.

19
.mp4 files

• MPEG-4 Video, 640 x480, millions of colors29.97
  fpsInterlaced Bottom Field FirstKey Frame Every
  300 Framesaspect ratio 43bitrate 1229kbps

20
.rm files

• Real Media 9 640x480 millions of colorsbitrate
  1067kbps constant bit rate29.97 fps43 aspect
  ratioprogressive (no option for interlaced)key
  frame every 300 frames

21
.wmv files

• Windows Media Video 9 Professionalbitrate 1340
  Variable bit rate29.97 fps43 aspect
  ratioInterlaced Bottom Field firstkey frame
  interval 300 frames

22
mpeg-2

• 20 Megabit 640x48029.97 fpsInterlaced Bottom
  Field firstconstant bit rate43 aspectGOP
  Pattern IPBBIPBBLong GOPSequence Headers for
  each GOPHigh Motion Search Range

23
Jpeg2000

• Motion JPEG2000 Kakaduvariable bit rate
  (lossless)29.97 fpsinterlaced bottom field
  first43 aspect ratio5/3 Reversiblemillions of
  colors

24
Video Quality Metrics

• Genistas Media Optimacy

25
Analysis Tools

• Tools were needed to examine the files on the
  signal level in order to establish where and when
  in a file artifacts appear as result of
  compression.
• Perceptual quality measurement tools, such as
  Genistas Media Optimacy, have enabled content
  providers to develop associated network delivery
  mechanisms for the best possible audience
  experience.

26
Video Quality Metrics

• Genista has developed a set of metrics for
  measuring the quality of digital video and still
  images. Genista's quality metrics measure the
  typical artifacts introduced by processing
  (notably compression) and transport of digital
  video. Additionally, a metric exists to make a
  prediction of Mean Opinion Score (MOS), (i.e.,
  reproducing the results of human subjective tests
  on overall image quality).

27
Video Quality Metrics

• Metrics are not merely based on network
  statistics or network performance parameters such
  as packet loss.
• Take into account the image content and frame
  data of the video resulting from the given coding
  and transmission conditions.
• The metrics can be divided into spatial and
  temporal metrics.

28
Spatial Metrics

• Spatial metrics, such as blockiness, perform
  their measurements on a frame-by-frame basis,
  returning a result for each frame measured.
  Temporal metrics, such as jerkiness, look at two
  or more consecutive frames simultaneously to
  obtain a measurement. MOS prediction takes into
  account both spatial and temporal aspects.

29
Relative and Absolute Metrics

• Video quality measures can be divided into
  relative (full-reference, FR) metrics and
  absolute (non-reference, NR) metrics. FR metrics
  compare a compressed or otherwise processed video
  directly with the original whereas NR metrics
  analyze any video without the need for a
  reference, using only the data contained in the
  clip under test.

30
Non-reference Metrics

• Non-reference metrics target real-time
  measurement of streaming video.
• Useful for monitoring quality variations due to
  network problems, as well as for applications
  where service level agreements and quality
  control are required.
• Characterization of the reference content prior
  to encoding or processing.
• Currently non-reference metrics exist to measure
  jerkiness, blockiness, Blur, and MOS.

31
Metrics Categories

• Fidelity metrics
• measure the mathematical difference between
  processed and reference video.
• Spatiotemporal metrics
• as defined by the ANSI standard
• Perceptual metrics
• includes a prediction of MOS
• provides an overall perceptual quality in MOS
  scale. Each of

32
Fidelity Metrics

• These metrics are widely used and represent
  arithmetic measures of the distance between
  processed and reference video.
• They are full-reference metrics by definition.
• Do not take into account human perception

33
Fidelity Metrics

• PSNR FR, spatial Peak signal to noise ratio
  (luminance).
• SNR FR, spatial Signal to noise ratio
  (luminance).
• RMSE FR, spatial Root mean square error
  (luminance).
• Color PSNR FR, spatial PSNR from CIE

34
Metric Type Description

• Motion energy difference
• FR, temporal
• Added motion energy indicates error blocks,
  noise.
• Repeated frames
• FR, temporal
• Lost motion energy indicates jerkiness.
• Edge energy difference
• FR, spatial
• Indicates dropped or repeated frames.
• Horizontal and vertical edges
• FR, spatial
• Added edge energy indicates edge noise,
  blockiness, and noise
• Spatial frequencies difference
• Lost edge energy indicates Blur.

35
Spatiotemporal Metrics

• Rely on algorithms defined by recommendations
  from the American National Standards Institute
  (ANSI). This recommendation represents an attempt
  by a standards body to define objective measures
  that serve as a basis for the measurement of
  video quality.

36
Perceptual Metrics

• Perceptual quality metrics measure specific
  artifacts introduced into the video as perceived
  by a human viewer. These artifacts are well
  known, and are easily recognized even by
  non-experts.
• Provide an automatic measure of those artifacts
  that viewers will perceive, in a way that is
  correlated with human perception. Additionally, a
  metric exists to make a prediction of Mean
  Opinion Score (MOS), (i.e. reproducing the
  results of human subjective tests).

37
Jerkiness

• Perceptual measure of frozen pictures or motion
  that does not look smooth. The primary causes of
  jerkiness are network congestion and/or packet
  loss.
• It can also be introduced by the encoder dropping
  or repeating entire frames in an effort to
  achieve the given bit-rate constraints.
• A reduced frame rate can also create the
  perception of jerky video.

38
Blockiness

• Perceptual measure of the block structure that is
  common to all DCT-based image compression
  techniques.
• The DCT is typically performed on 8x8 blocks in
  the frame, and the coefficients in each block are
  quantized separately, leading to artificial
  horizontal and vertical borders between these
  blocks.
• Blockiness can also be caused by transmission
  errors, which often affect entire blocks in the
  video. Genista has developed both FR and NR
  blockiness metrics.

39
Blur

• Perceptual measure of the loss of fine detail and
  the smearing of edges in the video. It is due to
  the attenuation of high frequencies at some stage
  of the recording or encoding process.
• It is one of the main artifacts of wavelet-based
  compression techniques, such as JPEG2000, where
  transmission errors or packet loss can also
  induce Blur.
• Other important sources of Blur are low-pass
  filtering (e.g,. analog VHS tape recording),
  out-of-focus cameras, or high motion (leading to
  motion blur).

40
Noise

• Perceptual measure of high-frequency distortions
  in the form of spurious pixels. It is most
  noticeable in smooth regions and around edges
  (edge noise).
• Can arise from noisy recording equipment (analog
  tape recordings are usually quite noisy), the
  compression process, where certain types of image
  content introduce noise-like artifacts, or from
  transmission errors (especially uncorrected bit
  errors).

41
Ringing

• Perceptual measure of ripples typically seen
  around high-contrast edges in otherwise smooth
  regions (the technical cause for this is referred
  to as Gibb's phenomenon).
• Ringing artifacts are very common in
  wavelet-based compression schemes (e.g,
  JPEG2000), but also appear to a slightly lesser
  extent in DCT-based compression techniques (e.g.
  JPEG, MPEG).

42
Colorfulness

• Describes the intensity or saturation of colors
  as well as the spread and distribution of
  individual colors in the image.
• The range and saturation of colors often suffers
  due to compression.

43
Mean Opinion Score

• MOS Prediction. MOS is the Mean Opinion Score
  obtained from experiments with human subjects.
• Metrics correlate with human perception of video
  quality and thus with the output of subjective
  test results, a metric that represents the
  perceived quality of video content.

44
What Did We Learn?
45
Almost impossible to predict Codec performance in
advance

• There was little consistancy in Codec Performance
  from clip to clip or within a clip
• Sometimes performance is divergent and other
  times

46
Performance is Bad for all
47
Smooth Sailing Followed by Disaster
48
In this example MPG2 is clearly better then WM
but is the large variation in quality distracting
and actually worse then a more even performance?
49
There was no clear leader over a wide variety of
material

• Each Codec has its own problems
• Bitrate is not a good overall predictor of
  quality
• Artifacts can be generated by any Codec at any
  bitrate some are perceptually significant
  others are not
• Putting business issues and marketing dynamics
  aside there was no clear performance leader
  even over systems that are very similar like WM
  and MP4

50
Our Opinion

• From An Archival point of view Lossy
  Compression is UNACCEPTABLE any flavor, any
  rate because
• There is no way to reliably predict performance
  over a wide spectum of material unless you know
  in advance and can literally do scene to scene
  compression
• Even for short sequences there was no
  combination that showed outstanding performance
• We believe that LossLESS compression is a viable
  and acceptable option for video preservation.

51
There is a STANDARD!

• JPEG 2000 has Mathematically Lossless inclusion
  in Standard
• Hardware soon available to Encode / Decode
  Mathematically Lossless video in REAL TIME (3rd Q
  2004)
• Cost Effective Hardware
• Hard Disk storage continues to decline in Cost
  and Increase in Density
• Current cost less then 1 per GigabyteAs of
  3/17/04 .83USAs of 5/28/04 .79 US

52
(No Transcript)
53
Mathematically LossLESS offers many advantages

• No Artifacts added due to compression process
• Frames available as discrete units (not true with
  MPEG)
• Additional cost of Lossless compression as
  compared to Lossy becomes small relative to
  overall costs
• Continuing trends of decrease in Hard Drive Cost
  drive cost advantage further as time goes on

54
Mathematically Lossless Compression

• Mathematically Lossless
• 31 Compression
• 72 Gigabytes becomes 24 Gigabytes with NO loss in
  quality
• In 2010 we forecast the RAW storage cost for 1
  hour of content on Hard Drive less then 1.50 US
  2004 Dollars.
• Savings of lossy compression become meaningless
  as compared to overall costs and tradeoff for
  future use.