RateConstrained Conditional Replenishment with Adaptive Change Detection

1
Rate-Constrained Conditional Replenishment with
Adaptive Change Detection
EE368B Project

• Xinqiao Liu
• December 8, 2000

2
Motivation

• Conditional replenishment ---- method of reducing
  temporal redundancy between successive frames
• Efficient in video conferencing with stationary
  cameras and slow motion.
• Study shows that less than 3 of the pixels need
  to be replenished in most head-and-shoulders
  scenes in desktop video
• Computational complexity is significant simpler
  than other video compression methods
• Software-only CODEC is possible
• Appealing for on-sensor compression where pixel
  array and simple image processing are integrated
  on the same chip, i.e, camera system-on-chip

3
Previous Work

• Most of the research concentrate mainly on the
  image quality (Haskell, et al72, Haskell79)
• Recently, a perception-based change detection
  method was proposed (ChiuBerger 96,
  ChiuBerger99)
• Reduces the perceptual redundancy in addition to
  the spatial and temporal redundancy
• Change detection threshold is set based on Webs
  law
• However, the correlation between transmission
  bit-rate and the choice of change detection
  schemes still need to be explored.

4
Outline

• Introduction Problem formulation
• Context-based Arithmetic Encoder
• Change detection --- direct methods
• Subsampling
• Threshold adjusting
• Adaptive change detection
• Noise characteristic
• Adaptive algorithm
• Conclusion

5
Conditional Replenishment Diagram
Goal Given a rate-constrained transmission
channel, find the optimal change detection
algorithm that minimizes the distortion
6
Model and Assumptions

• Assumptions
• Transmitted separately under certain bit-rate
  constrain R1, R2
• Lossless coding for both mask and signal
• Only intra-frame compression is considered

7
Rate-Constrained Change Detection

• Three ways to control the bit rate in the change
  detector
• Subsampling the mask and signal after detection
• Adjusting the detection threshold
• Using adaptive threshold for each pixel based on
  the noise characteristics -----eliminate those
  pixels that have changed due to noise rather than
  the input
• Use unconstrained Lagrangian cost function to
  find the optimum detection parameters for each
  method

8
Problem Formulation (I)
Given previous frame A1, current frame A2, binary
change mask C, the reconstructed frame at decoder
end is
The mean-square distortion is defined as
Assume R1 kR2 since they are proportional to
the number of changed pixels. The total bit-rate
R is
The above assumption allows us to study the
rate-distortion function of conditional
replenishment by only implementing the
compression scheme of the mask.
9
Problem Formulation (II)

• The constrained problem of

Can be converted to the unstrained problem by
introducing the Lagrangian cost function given
Lagrange multiplier l
where s is the adjustable change detection
parameter. The optimal value of s is given by
The desired optimal slop value l is not known a
priori but can be obtained using a fast bisection
search algorithm
10
Outline

• Introduction problem formulation
• Context-based Arithmetic Encoder
• Change detection --- direct methods
• Adaptive change detection
• Conclusion

11
Test Video Sequence

• Captured by a stationary high-speed digital
  camera with a person moving cross the screen

12
Context-based Arithmetic Encoder (CAE)

• Binary bitmap-based shape coding scheme used in
  the MPEG-4 standard
• Three types of 16x16 macroblocks
• "black" block none of the pixel changed (all 0)
• "white" block all pixels changed and to be
  replenished (all 1)
• boundary block encoded with a template of 10
  pixels to define the causal context for
  predicting the binary value of the current pixel
  (S0).

For black and white blocks, only the block type
need to be transmitted For boundary blocks, use
conditional entropy
13
Outline

• Introduction problem formulation
• Context-based Arithmetic Encoder
• Change detection --- direct methods
• Subsampling
• Threshold adjusting
• Adaptive change detection
• Conclusion

14
Change Masks With Subsampling

• Subsample the macroblock by a factor of 2, 4 or 8
  
• Subblocks are encoded using the CAE
• Upsample at the decoder end using pixel
  replication filter combined with a 3x3 median
  filter

15
Rate-distortion of Subsampling
16
Change Masks With Threshold-adjusting

• Control the bit-rate by globally adjusting the
  change detector threshold. As the threshold
  increased, few pixels will be detected

17
Rate-distortion of Threshold-adjusting
18
Outline

• Introduction problem formulation
• Context-based Arithmetic Encoder
• Change detection --- direct methods
• Adaptive change detection
• Noise characteristics
• Adaptive algorithm
• Conclusion

19
Noise Characteristics

• A fundamental problem in designing an optimum
  change detector is how to separate pixels whose
  change is due to noise from pixels whose change
  is due to real input signal change
• For cameras using either CCD or CMOS image
  sensors, the final image is formed by the
  photo-charge Qi,j (or voltage) integrated on each
  photo-detector during the exposure time. Two
  independent additive noise corrupt the output
  signal
• Shot noise Ui,j which is zero mean signal
  dependent gaussian distribution with
• Readout circuit and reset noise Vi,j (including
  quantization noise) with zero mean and variance
  dV2.

20
Adaptive Change Detection

• Thus the total noise variance of pixel (i,j) is
• The noise is signal dependent
• The stronger the luminance level, the noisier the
  pixel will be
• The threshold Ti,j is thus set as
• where m is the sensitivity factor that is set
  globally
• is local average value over a small
  area with size 8x8.
• Note that by changing m, we effectively adjusting
  the detection sensitivity while the threshold is
  still locally adapted

21
Adaptive Threshold
22
Change Masks With Adaptive Threshold
23
Rate-distortion of Adaptive Threshold
24
Performance Comparison

• Subsampling is the most efficient in reducing
  bit-rate
• Adaptive thresholding achieves the best PSNR

25
Conclusion

• Studied three change detection algorithms
• Subsampling
• Threshold-adjusting
• Adaptive threshold based on the noise
  characteristics
• The adaptive change detection algorithm
  efficiently separates pixels whose change is due
  to noise from pixels whose value change is due to
  real input signal change
• Simulation proves that the adaptive change
  detection algorithm achieves the best PSNR among
  all the three algorithms