School of Computing Science Simon Fraser University, Canada

1
School of Computing ScienceSimon Fraser
University, Canada

• Rate-Distortion Optimized Streaming of
  Fine-Grained Scalable Video Sequences
• Mohamed Hefeeda ChengHsin Hsu
• MMCN 2007
• 31 January 2007

2
Motivations

• Multimedia streaming over the Internet is
  becoming very popular
• More multimedia content is continually created
• Users have higher network bandwidth and more
  powerful computers
• ? Users request more multimedia content
• ? And they look for the best quality that their
  resources can support

3
Motivations (contd)

• Users have quite heterogeneous resources
  (bandwidth)
• Dialup, DSL, cable, wireless, , high-speed LANs
• To accommodate heterogeneity ? scalable video
  coding
• Layered coded stream
• Few accumulative layers
• Partial layers are not decodable
• Fine-Grained Scalable (FGS) coded stream
• Stream can be truncated at bit level

4
Motivations (contd)

• Goal Optimize quality for heterogeneous
  receivers
• In general setting
• FGS-coded streams
• Multiple senders with heterogeneous bandwidth and
  store different portions of the stream
• Why multiple senders?
• Required in P2P streaming
• Limited peer capacity and Peer unreliability
• Desired in distributed streaming environment
• Disjoint network path ? Better streaming quality

5
Our Optimization Problem

• Assign to each sender a rate and bit range to
  transmit such that the best quality is achieved
  at the receiver.
• Consider a simple example to illustrate the
  importance of this problem

6
Example Different Streaming Schemes
Non-scalable
Layered
7
Example Different Streaming Schemes
FGS Scalable
1. FGS enables us to get the best quality from
senders 2. However, there too many allocation
options, and we need to carefully choose the
optimal one
8
Problem Formulation

• First single-frame case
• Optimize quality for individual frames
• Then multiple-frame case
• Optimize quality for a block of frames
• More room for optimization
• Details are presented in the extended version of
  the paper

9
Input Parameters

• T fixed frame period
• n number of senders
• bi outgoing bandwidth of sender i
• bI incoming bandwidth of receiver
• si length of (contiguous) bits held by sender i
• Assume w.l.g. s1 lt s2 lt lt sn

10
Outputs

• Allocation A (?i , ri) i 1, 2, , n
• ?i number of bits assigned to i
• ri streaming rate assigned to i
• Specifies
• Sender 1 sends range 0, ?1 -1 at rate r1
• Sender 2 sends range ?1 , ?1?2 -1 at rate r2
• Sender i sends range
  at rate ri

11
Integer Programming Problem

• Minimize distortion
• Subject to
• on-time delivery
• assigned range is available
• assigned rate is feasible
• Aggregate rate not exceeds receivers incoming BW

12
How do we Compute Distortion?

• Using Rate-Distortion (R-D) models
• Map bit rates to perceived quality
• Optimize quality rather than number of bits
• Approaches to construct R-D models
• Empirical Models Many empirical samples ?
  expensive
• Analytic Models Quality is a non-linear function
  of bit rate, e.g., log model Dai 06 and GGF
  model Sun 05
• Semi-analytic Models A few carefully chosen
  samples, then interpolate, e.g., piecewise linear
  R-D model Zhang 03
• Detailed analysis of R-D models in our previous
  work Hsu 06

13
The Linear R-D Model

• Within each bitplane, approximate R-D function by
  a line segment
• Line segments of different bitplanes have
  different slopes

14
Visual Validation of Linear R-D Model
Mother Daughter, frame 110
Foreman, frame 100
15
Rigorous Validation of Linear R-D Model

• Average error is less than 2 in most cases

16
Using the Linear R-D Model

• Let yi be number of bits transmitted from
  bitplane i
• Distortion is
• d base layer only distortion
• gi slope of bitplane i
• z total number of bitplanes

17
Integer Linear Programming (ILP) Problem

• Linear objective function
• Additional constraints
• number of bits transmitted from bit plane h does
  not exceed its size lh
• bits assigned to senders are divided among
  bitplanes

18
Solution of ILP is a Valid FGS Stream

• Lemma 1
• An optimal solution for the integer linear
  program produces a contiguous FGS-encoded bit
  stream with no bit gaps
• Proof sketch
• minimizing
• Since g1 lt g2 lt ltgnlt0 (line segment slopes),
• the ILP will never assign bits to yi1 if yi is
  not full

19
Linear Programming Relaxation

• Solving ILP problem is expensive
• Solution Transform it to Linear Programming (LP)
  problem
• Relax variables to take on real values
• Objective function and constraints remain the same

20
Efficient Rounding Scheme

• Solve LP ?
• Result is real values
• Then, use the following rounding scheme for
  solution of the ILP

21
Correctness/Efficiency of Proposed Rounding

• Lemma 2 (Correctness)
• Rounding of the optimal solution of the relaxed
  problem produces a feasible solution for the
  original problem
• Lemma 3 (Efficiency Size of Rounding Gap)
• The rounding gap is at most nT n, where n is
  the number of senders and T is the frame period
• (Extreme) Example n 30 senders, T 30 fps gt
  gap is 32 bits
• Indeed negligible (frame sizes are in order of
  KBs)

22
FGSAssign Optimal Allocation Algorithm

• Solving LP (using Simplex method for example) may
  still be too much
• Need to run in real-time on PCs (not servers)
• Our solution FGSAssign
• Simple, yet optimal, allocation algorithm
• Greedy Iteratively allocate bits to sender with
  smallest si (stored segment) first

23
Pseudo Code of FGSAssign

• Sort senders based on si, s1 s2 sn
• x0 xn 0 ?1 ?n 0 ragg 0
• for i 1 to n do
• xi min(xi-1 biT, si)
• ri (xi - xi-1)/T
• if (ragg ri lt bI ) then
• ragg ragg ri
• ?i xi - xi-1
• else
• ri bI - ragg
• ?i T ri
• return
• endfor

24
Optimality of FGSAssign

• Theorem 1
• The FGSAssign algorithm produces an optimal
  solution in O(n log n) steps, where n is the
  number of senders.
• Proof see paper
• Experimentally validated as well.

25
Multiple-Frame Optimization

• Solve the allocation problem for blocks of m
  frames each
• Objective minimize total distortion in block
• Why consider multiple-frame optimization?
• More room for optimization
• Solve the problem less often

26
Multiple-Frame Optimization Why?
Distortion (MSE)
Bit rate

• More room for optimization higher quality and
  less quality fluctuation

27
Multiple-Frame Optimization

• Formulation (in extended version of the paper)
• Straightforward extension to single-frame with
  lager number of variables and constraints
• Computationally expensive to solve
• Our Solution mFGSAssign algorithm
• Heuristic (close to optimal results)
• Achieves two goals
• Minimize total distortion in a block of frames
• Reduce quality fluctuations among successive
  frames
• Pseudo code and analysis see extended version of
  the paper

28
Experimental Setup

• Software used
• MPEG-4 Reference Software ver 2.5
• Augmented to extract R-D model parameters
• Algorithms implemented (in Matlab)
• LP solutions using Simplex for the single-frame
  and multiple-frame problems
• FGSAssign algorithm
• mFGSAssign algorithm
• Nonscalable algorithm for baseline comparisons

29
Experimental Setup (contd)

• Streaming scenarios
• Four typical scenarios for Internet and corporate
  environments
• Testing video sequences
• Akiyo, Mother, Foreman, Mobile (CIF)
• Sample results shown for Foreman and Mobile

30
Single Frame Quality (PSNR)
Mobile, Scenario III
Foreman, Scenario I

• Quality Improvement 1--8 dB
• FGSAssign is optimal

31
Multiple Frame Quality (PSNR)
Foreman, Scenario II
Mobile, Scenario III

• Scalable higher improvement than single frame
• mFGSAssign almost optimal (lt 1 gap)

32
Fluctuation Reduction
Foreman, Scenario II
Mobile, Scenario III

• Small quality fluctuations in successive frames

33
Conclusions

• Formulated and solved the bit allocation problem
  to optimize quality for receivers in general
  settings
• Multiple heterogeneous senders
• Considered single and multiple frame cases
• Approach
• Nonlinear problem ? integer linear program
• Using linear R-D model
• Integer linear program ? linear program
• Using simple rounding scheme
• Proposed efficient algorithms
• FGSAssign optimal and efficient
• mFGSAssign close to optimal in terms of average
  distortion, reduces quality fluctuations, runs in
  real time
• Significant quality improvements shown by our
  experiments

34
Thank You!

• Questions??
• All programs/scripts/videos are available
• http//www.cs.sfu.ca/mhefeeda