ARMOR - A System for Adjusting Repair and Media Scaling for Video Streaming

1
ARMOR - A System for Adjusting Repair and Media
Scaling for Video Streaming

• Huahui Wu, Mark Claypool and Robert Kinicki
• Elsevier Journal of Visual Communication and
  Image Representation (JVCIR)
• Volume 19, Number 8, Pages 489-499 December 2008

2
Introduction - Motivation
Video Frames
Repair by Forward Error Correction (FEC)
3
Operations Research Concept

• Adjusting Repair and Media Scaling
• Given Network and Application Environment
• For each valid FEC and scaling combination,
  measure the video quality
• Find the optimal point

4
Outline

• Introduction
• Background ?
• Models
• Algorithms
• User Study
• Implementation
• Conclusions

5
Video Compression Standard

• MPEG
• Popular compression standard
• Intra-compression and inter-compression
• Three types of frames I, P and B
• Group Of Pictures (GOP)
• ARMOR models MPEG dependencies

6
Forward Error Correction (FEC)

• Media-Independent FEC
• Reed-Solomon codes Reed 60
• ARMOR models benefits of FEC for frame
  transmission

7
Media Scaling (1 of 2)

• Sacrifice data to fit the capacity
• Temporal Scaling (TS)
• Pre-Encoding Temporal Scaling
• Post-encoding Temporal Scaling (below ?)

8
Media Scaling (2 of 2)

• Quality Scaling
• MPEG uses quantization in coding to save bits
• Quantization Value (131)
• For example original data 23, 13, 7, 3
• ARMOR models both Temporal Scaling and Quality
  Scaling

Quantization Value After Quantization After DeQuantization
3 7, 4, 2, 1 21, 12, 6, 3
6 3, 2, 1, 0 18, 12, 6, 0
12 1, 1, 0, 0 12, 12, 0, 0
9
Video Quality Measurements

• Subjective Measurement
• User study, expensive, not practical
• Objective Measurements
• Playable Frame Rate (R)
• Good for Temporal Scaling, not for Quality
  Scaling
• Peak Signal Noise Ratio (PSNR)
• Good for Quality Scaling, not for Temporal
  Scaling
• Video Quality Metric (VQM)
• By Institute for Telecommunication science
• Extracts 7 perception-based features
• Only one for frame losses
• Report a distortion value from 0 (no distortion)
  to 1 (many)
• ARMOR uses both R and VQM
• Includes a user study

10
Outline

• Introduction
• Background
• Models?
• Streaming Bitrate Model (cost)
• Video Quality Model (benefit)
• Algorithms
• User Study
• Implementation
• Conclusions

11
Parameters and Variables
Video Frames
Repair by Forward Error Correction (FEC)
12
Streaming Bitrate Model

• Total streaming bitrate (B), including video
  packets and FEC packets
• where G is the constant GOP rate
• NPD and NBD are the numbers of transmitting P and
  B frames depending on Temporal Scaling level lTS

13
Video Quality Model - Overview

• Two distortion factors
• Frame Loss
• Caused by Temporal Scaling and network packet
  loss
• Appears jerky in the video playout
• Measured by Playable Frame Rate
• Quantization Distortion
• Caused by a high quantization value with Quality
  Scaling
• Appears visually as coarse granularity in every
  frame
• Measured by VQM
• Overall Quality
• Distorted Playable Frame Rate

20
14
Playable Frame Rate (R)

• Frame Successful Transmission Probability
• Where Frame Size
• Frame Dependencies
• Total Playable Frame Rate

15
Distorted Playable Frame Rate (RD )

• Quality scaling distortion varies exponentially
  with the quantization level
• Distorted Playable Frame Rate

4
16
ARMOR Algorithm

• For each Repair and Scaling combination
• Estimate video frame sizes (SI, SP, SB)
• Compute streaming bitrate B and make sure its
  under capacity constraint T
• Use frame sizes and FEC amount to get
  successfully frame transmission rate (qI, qP, qB)
• Compute playable frame rate (R)
• Estimate quality scaling distortion (D)
• Compute distorted playable frame rate (RD)
• Exhaustively search all FEC and Scaling
  combinations and find optimal quality

17
Outline

• Introduction
• Background
• Models
• Algorithms
• User Study ?
• Implementation
• Conclusions

18
User Study Goals

• Accuracy of RD
• Correlation with user perceptual quality
• Versus PSNR and VQM?
• Temporal Scaling versus Quality Scaling
• What are the differences?
• Adjusted Repair (FEC) versus No Repair
• Is Adjusted Repair an effective method for
  increasing perceptual quality?

19
Video Clips

• Compare degraded clips to the original
• Original 30 fps, no quality scaling
• Degraded Combinations of 4 independent factors
  (2 options each)
• Video and Network environment
• Video content low motion (News)
  http//www.youtube.com/watch?vuN3yUm0WZwY or
  high motion (Coastguard) http//www.youtube.com/wa
  tch?vJQVclZWH5UM
• Packet loss rate low loss (1) or high loss (4)
• ARMOR Layer
• Repair adjusted repair or no repair
• Scaling Quality Scaling or Temporal Scaling
• 24 16 combinations for evaluation

20
User Study Application
54321

• Two-week volunteer study
• 74 users, most CS undergraduate students (male,
  young)
• Most LCD, high-rez monitor

Double Stimulus Impairment Scale (DSIS) ITU-R
BT.500-11
21
Results Video Quality Metrics (1 of 3)
(Same as original clip)

• User Score vs. PSNR

(Much worse than original clip)
22
Results Video Quality Metrics (2 of 3)

• User Score
• Vs.
• VQM Score
• (1 VQM distortion)

23
Results Video Quality Metrics (3 of 3)

• User Score
• vs.
• Distorted
• Playable
• Frame Rate
• (RD)

24
Results Scaling Methods
ARMOR Prediction (Coastguard)
User Score

• Temporal Scaling versus Quality Scaling

25
Results Repair Methods
User Score
ARMOR Prediction (Coastguard)

• Adjusted Repair versus No Repair

26
Outline

• Introduction
• Background
• Models
• Algorithms
• User Study
• Implementation ?
• Conclusions

27
Implementation Goals

• Provide architecture for ARMOR system
• Validate ARMOR model
• Determine if can make improvements to video
  quality in real-time

28
Architecture
3
3
2
2
1
2
3
4
1
5
6
7
8
29
Experiment Settings
Network (NistNet) Settings Network (NistNet) Settings MPEG Encoder Settings MPEG Encoder Settings
tRTT 50 ms NP 3 frames per GOP
S 1 Kbyte NB 8 frames per GOP
p 0.01 to 0.04 RF 30 frames per sec

• Video clip Paris
• http//www.youtube.com/watch?vXU74KL_72RA
• medium motion and details
• two people sitting, talking, with high-motion
  gestures
• 1200 CIF (352x288) images
• average I / P / B frame sizes 24.24 KB / 5.20 KB
  / 1.18 KB

30
Results
31
Conclusions

• Distorted playable frame rate high correlation
  with user perceptual quality
• Higher than PSNR or VQM
• Adjusting repair improves video streaming quality
  significantly
• Better than fixed repair and no repair
• Quality Scaling is more effective than Temporal
  Scaling
• But when bandwidth low and network loss high,
  Quality Scaling should be used with Temporal
  Scaling
• Proof of concept ? ARMOR can be implemented in
  real-time to effectively improve streaming quality

32
Future Work?
33
Future Work

• Implementation of quality scaling
• Implementation of streaming media protocols
• Bandwidth estimation techniques for initial
  streaming rate
• Alternative repair techniques
• Evaluate with time-varying bandwidth and packet
  loss
• Classification of video motion and scene
  complexity to predict exponential coefficients
• User studies to determine if RD works for
  different scaling combinations

34
ARMOR - A System for Adjusting Repair and Media
Scaling for Video Streaming

• Huahui Wu, Mark Claypool and Robert Kinicki
• Elsevier Journal of Visual Communication and
  Image Representation (JVCIR)
• Volume 19, Number 8, Pages 489-499 December 2008