ERROR CONCEALMENT TECHNIQUES IN H'264AVC, FOR VIDEO TRANSMISSION OVER WIRELESS NETWORKS

1
ERROR CONCEALMENT TECHNIQUES IN H.264/AVC,FOR
VIDEO TRANSMISSION OVERWIRELESS NETWORKS

• Vineeth Shetty Kolkeri
• University of Texas, Arlington

2
Outline

• Introduction
• What is Error Concealment?
• Error Concealment Architecture
• H.264/MPEG-4 AVC Overview
• Error Concealment algorithm
• Error Concealment performance analysis test
  results
• Conclusions
• Future Work
• References

3
Introduction
Figure 1 Typical Situation of 3G/4G cellular
telephony
4
What is Error Concealment?

• The operation adopted to reconstruct lost
  information for video transmission over wireless
  networks.
• Need Compatible with all video streaming
  devices.
• Applications Recovery of lost information which
  is comparable with encoded video sequence with
  minimal complexity.

5
Error Concealment Architecture
6
Overview of H.264 / AVC

• Latest Video coding standard
• Basic design architecture similar to MPEG-x or
  H.26x
• Better compression efficiency
• Upto 50 bitrate reduction from the preceding
  video codec standard
• Subjective quality is better
• Advanced functional element
• Wide variety of applications such as video
  broadcasting, video streaming, video
  conferencing, D-Cinema, HDTV.
• Layered structure - consists of two layers
  Network Abstraction Layer (NAL) and Video Coding
  Layer (VCL) supports 420 chroma sampling
  picture format including QCIF and CIF formats

7
Overview of H.264 / AVC (contd.)

• Uses hybrid block based video compression
  techniques such as
• Transformation - reduction of spatial correlation
• Quantization - bit-rate control
• Motion compensated prediction - reduction of
  temporal correlation
• Entropy coding - reduction in statistical
  correlation
• Includes the following features
• Intra-picture prediction
• 4x4 integer transform
• Multiple reference pictures
• Variable block sizes
• Quarter pel precision for motion compensation
• In-loop de-blocking filter
• Improved entropy coding

8
H.264/AVC Profiles

• Profiles and Levels for particular applications
• Profile a subset of entire bit stream of
  syntax,
• different decoder design based on the
  Profile
• Four profiles Baseline, Main, Extended and High

Applications
Profile
Video Conferencing Videophone
Baseline
Digital Storage Media Television Broadcasting
Main
Streaming Video
Extended
High
Studio editing
9
Specific coding parts for the Profiles
10
Specific coding parts for the Profiles (contd.)

• Common coding parts for the Profiles
• I slice (Intra-coded slice) the coded slice by
  using prediction only from decoded samples within
  the same slice
• P slice (Predictive-coded slice) the coded
  slice by using inter prediction from
  previously-decoded reference pictures, using more
  than one motion vector and reference index to
  predict the sample values of each block
• CAVLC (Context-based Adaptive Variable Length
  Coding) for entropy coding

11
Specific coding parts for the Profiles (contd.)

• Coding parts for Baseline Profile
• Common parts I slice, P slice, CAVLC
• FMO Flexible macroblock order macroblocks may
  not necessarily be in the raster scan order. The
  map assigns macroblocks to a slice group
• ASO Arbitrary slice order the macroblock
  address of the first macroblock of a slice of a
  picture may be smaller than the macroblock
  address of the first macroblock of some other
  preceding slice of the same coded picture
• RS Redundant slice This slice belongs to the
  redundant coded data obtained by same or
  different coding rate, in comparison with
  previous coded data of same slice

12
H.264 Encoder (contd.)
13
H.264 - Transform

• 4x4 multiplier-free integer transform
• Transform coefficients perfectly invertible
• Hierarchical structure - 4 x 4 Integer DCT,
  Hadamard transform
• Hadamard transform applied when (16x16) intra
  prediction mode is used with (4x4) integer DCT
• MB size for chroma depends on 420, 422 and
  444 formats

14
H.264 Transform (contd.)
YUV different formats

• 444 is full bandwidth YUV video, and each
  macroblock consists of 4 Y blocks, and 4 U/V
  blocks. Being full bandwidth, this format
  contains as much as data would if it were in the
  RGB color space.
• 422 contains half as much chrominance
  information as 444 and 420 contains one
  quarter of the chrominance information.

15
H.264 Transform (contd.)

• Codec - A video codec is software that can
  compress a video source (encoding) as well as
  play compressed video (decompress).
• CIF - Common Intermediate Format - a set of
  standard video formats used in videoconferencing,
  defined by their resolution. The original CIF is
  also known as Full CIF (FCIF).
• QCIF - Quarter CIF (resolution 176x144)
• SQCIF - Sub quarter CIF (resolution 128x96)
• 4CIF - 4 x CIF (resolution 704x576)
• 16CIF - 16 x CIF (resolution 1408x1152

16
H.264 - Scaling and Quantization

• Multiplication operation for exact transform
  combined with multiplication of scalar
  quantization
• Scale factor for each element in each sub-block
  varies as a function of quantization parameter
  associated with macro-block that contains sub
  block position of element within sub-block
• Rate-control algorithm in encoder controls value
  of quantization parameter
• Encoder performs post-scaling and quantization

17
Deblocking filter Adaptive

• To reduce the blocking artifacts in the block
  boundary and prevent the propagation of
  accumulated coded noise.
• Filtering is applied to horizontal or vertical
  edges of 4 x 4 blocks in a macroblock, adaptively
  on the several levels (slice, block-edge,
  sample).

18
H.264 Motion Compensation

• Predicted rectangular arrays of pixels 4x4, 4x8,
  8x4, 8x8, 16x8, 8x16, and 16x16.
• Translation from other array positions in
  reference picture specified with quarter pixel
  precision.
• In case of 420 format, the chroma MVs have a
  resolution of 1/8 of a pixel, derived from
  transmitted luma MVs of 1/4 pixel resolution.

19
Prediction of variable block size

• A MB can be partitioned into smaller block sizes
• 4 cases for 16 x 16 MB, 4 cases for 8 x 8 Sub-MB
• Large partition size homogeneous areas, small
  detailed areas

20
Prediction of variable block size (contd.)
Frame divided into multiple macroblocks of 16 x
16, 8 x 8, 4 x 4 variable size to represent
coding profiles
No. of bits in I and P frames
I P I
Graph shows the size of the different I and P
frames obtained after encoding 19 frames of the
Foreman QCIF video sequence. Green line shows the
average values of the bit lost when it is passed
through the lossy algorithm after encoding in a
video sequence
21
H.264 Entropy Coding

• CAVLC (Context-based Adaptive Variable Length
  Coding).
• CABAC (Context-based Adaptive Binary Arithmetic
  Coding).
• CAVLC makes use of run-length encoding.
• CABAC utilizes arithmetic coding codes both MV
  and residual transform coefficients.
• Typically CABAC provides 10-15 reduction in bit
  rate compared to CAVLC, for the same PSNR.
• All syntax elements are encoded by Exp-Golomb
  codes (Universal Variable Length Codes (UVLC)).

22
Error Concealment Algorithm

• 1.Spatial Concealment weighted averaging
• Estimate missing pixels by smoothly extrapolating
  surrounding pixels
• Correctly recovering missing pixels is extremely
  difficult, however correctly estimating the DC
  (average) value is very helpful
• 2.Temporal Concealment copy algorithm
• Copy the pixels at the same spatial location in
  the previous frame
• Effective when there is no motion, potential
  problems when there is motion
• 3.Motion compensated temporal Concealmentmotion
  vector interpolation
• Estimate missing block as motion-compensated
  block from previous frame
• Can use coded motion vector, neighboring motion
  vector, or compute new motion vector

23
Spatial Concealment weighted averaging
Block based weighted averaging
24
Spatial Concealmentweighted averaging (contd.)
Macroblock based weighted averaging
25
Spatial Concealment weighted averaging (contd.)
Recovery of the damaged macroblock in Foreman and
Akiyo video sequence (a) distorted image lying
within a smooth area b) macroblock based
weighted averaging applied on a white smooth
area c) block based weighted averaging applied
on a white smooth area.
26
Spatial Concealment weighted averaging (contd.)
Recovery of the damaged macroblock in Foreman and
Akiyo video sequence (a) distorted image lying
within a smooth area b) macroblock based
weighted averaging applied on a white smooth
area c) block based weighted averaging applied
on a white smooth area.
27
Temporal Concealment Frame Copy
Frames 5, 6 and 7 are the output of H.264
encoded frames after it is transmitted in the
error prone wireless medium
Frame 5 is the decoded frame. Here Frame 6
successfully copied lost information from Frame 5
by copy algorithm Frame 7 is degraded (Because
Frame7 is reconstructed bycollecting the
information from previous reference frames)
28
Temporal Concealment MV Interpolation
Motion vector recovery by a) Using the motion
vectors from the surrounding macroblocks after
frame decoding b) Using the motion vectors from
the surrounding macroblocks during macroblock
decoding
29
Temporal Concealment MV Interpolation (contd.)
Four Prior-decoded pictures
Current Picture as references
30
Motion Vector Interpolation
Frames 5, 6 and 7 of the Original Sequence
Frame 5 of the decoded frame, Successfully
decoded lost Frame 6. Frame 6 was
reconstructed by Motion Copy algorithm. Frame 7
is degraded.
31
Motion Vector Interpolation (contd.)
Recovery of the damaged macroblock in Foreman
video sequence (a) original sequence b) Distorted
Sequence c) Concealed Output using Motion
Estimation.
32
Error Concealment performance analysis test
results (Foreman Sequence)
33
Error Concealment performance analysis test
results (contd.) (Foreman Sequence)
34
Error Concealment performance analysis test
results (contd.) (Foreman Sequence)
35
Error Concealment performance analysis test
results (contd.) (Foreman Sequence)
36
Error Concealment performance analysis test
results (contd.)
Simulation results of different error concealment
algorithms for Foreman QCIF176x144 video sequence.
37
Error Concealment performance analysis test
results (contd.)
Simulation results of different error concealment
algorithms for Foreman QCIF176x144 video sequence.
38
Conclusions

• Implementation of spatial concealment performed
  better in a video with constant background.
• Implementation of temporal concealment performed
  better in a video with linear motion between
  consecutive frames.
• Implementation of temporal concealment performed
  better in a video with dynamic motion between
  consecutive frames.
• At higher bit rates spatial and temporal
  concealments achieved better results.
• Complexity of implementation is negligible and
  does not degrade in the processor performance.

39
Future Research

• Implementing Error Concealment algorithm in
  H.264/SVC video codec as it doesnt support error
  concealment in current implementation.
• Use forward and backward MVs.
• Implementing Error Concealment algorithm in H.264
  extended and High profiles.
• Find a block which has uniform motion within
  small duration.
• Implementing Error Concealment model for real
  time application like video surveillance.

40
References

• T. Stockhammer, M. M. Hannuksela and T. Wiegand,
  H.264/AVC in Wireless Environments, IEEE Trans.
  Circuits and Systems for Video Technology, Vol.
  13, pp. 657- 673, July 2003.
• 2. Soon-kak Kwon, A. Tamhankar and K.R. Rao,
  Overview of H.264 / MPEG-4 Part 10, J. Visual
  Communication and Image Representation, vol. 17,
  pp.186-216, April 2006.
• 3. S. Wenger, H.264/AVC over IP IEEE Trans.
  Circuits and Systems for Video Technology, vol.
  13, pp. 645-656, July 2003.
• 4. M. Wada, Selective Recovery of Video
  Packet Loss using Error Concealment, IEEE
  Journal on Selected Areas in Communication, vol.
  7, pp. 807-814, June 1989.
• 5. I.C.Todoli Performance of Error
  Concealment Methods for Wireless Video, Diploma
  Thesis, Vienna University of Technology, 2007 .
• 6. Video Trace research group at ASU, YUV
  video sequences, http//trace.eas.asu.edu/yuv/ind
  ex.html.
• 7. A.B. Watson, "Toward a perceptual video
  quality metric", SPIE Human Vision, Visual
  Processing, and Digital Display VIII, vol. 3299,
  pp 139-147, 1998.
• 8. F. Xiao, DCT-based video quality
  evaluation, Final Project for EE392J Stanford
  Univ. 2000. http//compression.ru/video/quality_me
  asure/vqm.pdf
• Z. Wang, The SSIM index for image quality
  assessment, http//www.cns.nyu.edu/zwang/files/re
  search/ssim/.

41
References (contd.)

• 10. Z. Wang, et al, Image Quality Assessment
  From Error Visibility to Structural Similarity,
  IEEE Trans. Image Processing, vol. 13,
  pp.600-612, April 2004.
• 11. ISO/IEC JTC1, Joint Draft 8 of SVC
  Amendment, ISO/IEC JTC1/SC29/WG11 and ITU-T SG16
  Q.6, Doc. JVT-U201, Oct. 2006.
• 12. ISO/IEC JTC1, Joint Scalable Video Model
  8.0, ISO/IEC JTC1/SC29/WG11 and ITU-T SG16 Q.6,
  Doc. JVT-U202, Oct. 2006.
• 13. Yi-Hau Chen, et al, Bandwidth-efficient
  encoder framework for H.264/AVC scalable
  extension Ninth IEEE International Symposium on
  Multimedia, pp 401-406, Dec 2007.
• 14. DSP Design Line article on SVC
  http//www.dspdesignline.com/products/206902239.
• 15. eInfochips HD Codecs - H.264 SVC - for
  Digital Media Processors From Texas
  Instruments
• 16. H.264/AVC Reference Software Download
• http//iphome.hhi.de/suehring/tml/download/
• 17. H.264/SVC Reference Software download
  http//ip.hhi.de/imagecom_G1/savce/downloads/SVC-R
  eference-Software.htm
• 18. AVI to YUV converter http//www.sunrayimage.c
  om/

42

• Thanks for your attention!
• Q/A