Advanced Digital Signal Processing

1
DEPARTMENT OF COMPUTER SCIENCE UNIVERSITY OF
JOENSUU JOENSUU, FINLAND

• Advanced Digital Signal Processing
• Lecture 5
• Quantization
• Alexander Kolesnikov

2
Time Sampling Quantization

• Any analog quantity that is to be processed by a
  digital computer or digital system must be
  converted to an integer number proportional to
  its amplitude.
• The conversion process between analog samples
  and discrete-valued samples is called
  quantization.

Q
Quantizer
Input signal
Quantized signal
3
Quantization application areas
Analog-to-digital conversion of signals
(audio,video,etc.)
(http//ptolemy.eecs.berkeley.edu/eecs20/berkeley/
body.html)
Quantization of transform coefficients (JPEG,
JPEG2000)
Binarization and multilevel thresholding of
digital images
4
Quantization Rounding-off
Any real number x can be rounded-off to the
nearest integer value q(x) round(x)
If k?0.5 ? x ltk 0.5 then q(x)k
5 5.5 6
6.5 7
x6.4
q(x)6.0
x5.6
5
Quantizer as line partition
yi ? Reproduction (reconstructed)
levels ai ? Decision levels (thresholds) Si
ai-1, ai) ? Cell ai -ai-1 ? Cell width
a0 -?, a6 ?.
6
Nonuniform quantizer M 8 levels
Rlog2M
Input-output characteristic of nonuniform
quantizer
7
Quantization error
Input signal x
Quantized signal q(x)
Quantization error e(x) x?q(x)
8
Distortion measure MSE, variance
Probability density function (pdf) of x is
p(x) Quantization error e(x) x ? q (x) Mean
(average value) ? of quantization error
Variance ?2 of quantization error as distortion
measure
9
Distortion measure Signal-noise-ratio (SNR)
10
Uniform source MSE
Data x is 0-mean uniformly distributed in
-A/2,A/2. Uniform quantizater with step ?A/M.
11
Uniform source SNR
Data x is 0-mean uniformly distributed in
-A/2,A/2. The number of bits Rlog2M, M is the
number of levels or M2R cells. Uniform
quantizater with step ?A/M
1 bit more in quantizer ? 6 dB more to SNR
6 dB per bit rule
12
Optimal quantization problem
Given a signal x, with probability density
function (or histogram) p(x), find a quantizer
q(x) of x, which minimizes the quantization error
variance ?2
13
Formulation of quantization problem (contd)
Optimization problem Find decision aj and
representation levels yj to minimize variance
?2.
14

• Max-Lloyd scalar quantizer

15
Max-Lloyd solution
16
Max-Lloyd solution
17
Max-Lloyd Conditions for optimal quantizer
Decision levels ai are midpoints
Representation levels yi are centroids
yj1
aj
yj-1
yj
aj-1
18
How to construct optimal quantizer?

• If we have some set of levels, with Max-Lloyd
  equations
• we can check do these levels provide minimum
  of
• error variance.
• But these equations dont tell us how to find
  the
• optimal levels.

?
19
Max-Lloyd Iterative algorithm
0. Guess initial set of decision levels aj 1.
Calculate representation levels (centroids)
yj
?
2. Calculate decision levels aj
3. Repeat 1. and 2. until no further variance ?2
reduction
20
Iterative algorithm Discrete case
0. Guess initial set of decision levels aj 1.
Calculate representation levels
(centroids) yj
?
2. Calculate decision levels aj
3. Repeat 1. and 2. until no further variance ?2
reduction
21
How to construct optimal scalar quantizer?

• Other names of the algorithm k-means, c-means,
  ...
• Iterative algorithm of Lloyd cannot guarante
  global
• minimum of quantizaton error.
• Other heuristic approaches merge, genetic
  algorithm,
• reinforcement learning, etc.
• How to achieve the global minimum
• of the quantization error?

22
High-Rate Quantization
23
High-Rate quantization

• Data X -? lt x lt ?
• p(x) is probability density function
• Lets assume that p(x) is approx. flat over
  cells Cj
• Quantization into M cells M ?? 1.
• Distortion for cell Cj under assumption

• Distortion for uniform quantization D?2/12

24
Centroid density

• Centroid density gC1/?j, one centroid for
  one cell.
• In the limit case M?? and ?j?0, centroid density
  gC(x) is function of x and ?j?1/gC.
• In this case, distortion for one cell is

• The total distortion D

25
Optimal high-rate quantization

• Find centroid density gC(x) such that the total
  distortion
• D is minimal

subject to
26
Lagrangian multiplier method

• Convert the problem into unconstrained
  optimization
• task with Lagrangian cost function D

• Find minimum of Lagrangian cost function

27
Find minimum of D
The centroid density
Use the condition
to calculate coefficient ?.
28
High-rate quantizer
The centroid density
Distortion
29
Optimal scalar quantizer
30
Problem formulation
p2
p1
xN
x1 x2

• Let Xx1, x2, , xN be a finite ordered set of
  real
• numbers (intensity values).
• Let Pp1, p2, , pN be the correspondent set
  of a
• probabilities for the values X (histogram).
• Let r0,r1,r2, ,rM1 be an ordered set of
  integers
• such that defines a partition of the set X into
  M parts
• r0 0 lt r1 lt ... lt rj lt rj1
  lt... lt rM N.

31
Sequence partition problem

• Partition indices r0 0 lt r1 lt... lt rj lt ...
  lt rM N.
• (Iintroduced r0 0 for x0 ??.)
• The total quantization error

• Quantization error for one cell

• Cells centroid yj

32
Optimization task

• For a given data X, probabilities P and number
  of cells M
• find such a partition ro,r1, r2, , rM that
  the total
• quantization error is minimal

where
and
33
Cost function DM(0,N
Let us introduce cost function Dm(0,n that is
minimum quantization error for quantization of
data sub-set Xnx1, x2, , xn with m
cells Then DM(0,N gives us solution of the
problem in question.
34
Dynamic programming approach

• Lets rewrite the cost function

• Finally

35
Reccurent equations
Initialization
Recursion
36
Optimal scalar quantization

• Optimal scalar quantizer as weighted k-link
  shortest
• path in graph.
• DP algorithm 1963 time complexity is O(MN2)

• Wu 1991 reduced time complexity of optimal DP
• algorithm to O(MN) using Monge (monotonicity)
  property
• of quantization error with L2 metrics.
• X,Y,Z 2003 Fast algorithm for multilevel
  thresholding
• O(NM)? O(NM-1)

37
Example M3
Uniform
Input image
Optimal
38
Example M3
Uniform
Optimal
39
Example M12
Centroids density is higher when probability
density is also higher...
40
High-Rate quantization
41
Vector Quantization
42
VQ Definition
Xx1,x2,,xN is set of input vectors in
d-D space cc1,c2,,cM is set of code
vectors in the space P is a partition of the
space into M code cells (cluster)
CC1,C2,,CM
?
?
?
?
?
?
?
VQ is NP-hard!
?
?
?
?
?
43
Example VQ of colors in 3-D space
Input data
Code vectors
N65000
M1000
44
VQ

• More about VQ in courses
• Image Analysis (Spring, 2006)
• Image Compression (Fall, 2004)
• http//cs.joensuu.fi/pages/koles/imagecomp/i
  magecomp.html