# Arithmetic Coding Additional Material Spring 2015 - PowerPoint PPT Presentation

PPT – Arithmetic Coding Additional Material Spring 2015 PowerPoint presentation | free to view - id: 7c1c77-NjBhN The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
Title:

## Arithmetic Coding Additional Material Spring 2015

Description:

### CMPT 365 Multimedia Systems Arithmetic Coding Additional Material Spring 2015 Context-adaptive Arithmetic Coding In many cases, a sample has strong correlation with ... – PowerPoint PPT presentation

Number of Views:50
Avg rating:3.0/5.0
Slides: 72
Provided by: sfu59
Category:
Tags:
Transcript and Presenter's Notes

Title: Arithmetic Coding Additional Material Spring 2015

1
CMPT 365 Multimedia Systems
2015
2
Outline Part I
• Introduction
• Basic Encoding and Decoding
• Scaling and Incremental Coding
• Integer Implementation
• Binary Arithmetic Coding
• Applications
• JBIG, H.264, JPEG 2000

3
Limitations of Huffman Code
• Need a probability distribution
• Hard to adapt to changing statistics
• Minimum codeword length is 1 bit
• Serious penalty for high-probability symbols
• Example Binary source, P(0)0.9
• Entropy -0.9log2(0.9)-0.1log2(0.1) 0.469 bit
• Huffman code 0, 1 ? Avg. code length 1 bit
• Joint coding is not practical for large alphabet.
• Arithmetic coding
• Can resolve all of these problems.
• Code a sequence of symbols without having to
generate codes for all sequences of that length.

4
Introduction
• Recall table look-up decoding of Huffman code
• N alphabet size
• L Max codeword length
• Divide 0, 2L into N intervals
• One interval for one symbol
• Interval size is roughly
• proportional to symbol prob.

1
00
010 011
dcba..
5
Arithmetic Coding
1
0
a b c
• Disjoint and complete partition of the range 0,
1)
• 0, 0.8), 0.8, 0.82), 0.82, 1)
• Each interval corresponds to one symbol
• Interval size is proportional to symbol
probability
• Observation once the tag falls into an interval,
it never gets out of it

6
• Why compression is achieved this way?
• How to implement it efficiently?
• How to decode the sequence?
• Why is it better than Huffman code?

7
Example
Symbol Prob.
1 0.8
2 0.02
3 0.18
• Map to real line range 0, 1)
• Order does not matter
• Decoder need to use the same order
• Disjoint but complete partition
• 1 0, 0.8) 0, 0.7999999
• 2 0.8, 0.82) 0.8, 0.8199999
• 3 0.82, 1) 0.82, 0.9999999
• (Think about the impact to integer implementation)

8
Encoding
• Input sequence 1321

Range 1
Final range 0.7712, 0.773504) Encode 0.7712
Difficulties 1. Shrinking of interval requires
high precision for long sequence. 2. No
output is generated until the entire sequence has
been processed.
9
Cumulative Density Function (CDF)
• For continuous distribution
• Properties
• Non-decreasing
• Piece-wise constant
• Each segment is closed at the lower end.

10
Encoder Pseudo Code
low0.0, high1.0 while (not EOF) n
ReadSymbol() RANGE HIGH - LOW HIGH LOW
RANGE CDF(n) LOW LOW RANGE
CDF(n-1) output LOW
• Keep track of LOW, HIGH, RANGE
• Any two are sufficient, e.g., LOW and RANGE.

Input HIGH LOW RANGE
Initial 1.0 0.0 1.0
1 0.01.00.80.8 0.01.00 0.0 0.8
3 0.0 0.810.8 0.0 0.80.820.656 0.144
2 0.6560.1440.820.77408 0.6560.1440.80.7712 0.00288
1 0.77120.0028800.7712 0.77120.002880.80.773504 0.002304
11
Decoding
Decode 1
Drawback need to recalculate all thresholds each
time.
12
Simplified Decoding
• Normalize RANGE to 0, 1) each time
• No need to recalculate the thresholds.

13
Decoder Pseudo Code
• Low 0 high 1
• x Encoded_number
• While (x ? low)
• n DecodeOneSymbol(x)
• output symbol n
• x (x - CDF(n-1)) / (CDF(n) - CDF(n-1))

14
Outline
• Introduction
• Basic Encoding and Decoding
• Scaling and Incremental Coding
• Integer Implementation
• Binary Arithmetic Coding
• Applications
• JBIG, H.264, JPEG 2000

15
Scaling and Incremental Coding
• Problems of Previous examples
• Need high precision
• No output is generated until the entire sequence
is encoded
• Key Observation
• As the RANGE reduces, many MSBs of LOW and HIGH
become identical
• Example Binary form of 0.7712 and 0.773504
• 0.1100010.., 0.1100011..,
• We can output identical MSBs and re-scale the
rest
• ? Incremental encoding
• This also allows us to achieve infinite precision
with finite-precision integers.

16
E1 and E2 Scaling
• E1 LOW HIGH) in 0, 0.5)
• LOW 0.0xxxxxxx (binary),
• HIGH 0.0xxxxxxx.

0 0.5 1.0
17
Encoding with E1 and E2
Symbol Prob.
1 0.8
2 0.02
3 0.18
Input 1
0
0.8 1.0
Encode any value in the tag, e.g., 0.5
Output 1 All outputs 1100011
18
To verify
• LOW 0.5424    (0.10001010... in binary), HIGH
0.54816 (0.10001100... in binary).
• So we can send out 10001 (0.53125)
• Equivalent to E2?E1?E1?E1?E2
• After left shift by 5 bits
• LOW (0.5424 0.53125) x 32 0.3568
• HIGH (0.54816 0.53125) x 32 0.54112
• Same as the result in the last page.

19
Comparison with Huffman
• Note Complete all possible scaling before
• encoding the next symbol

Symbol Prob.
1 0.8
2 0.02
3 0.18
• Input Symbol 1 does not cause any output
• Input Symbol 3 generates 1 bit
• Input Symbol 2 generates 5 bits
• Symbols with larger probabilities generates less
number of bits.
• Sometimes no bit is generated at all
• ? Advantage over Huffman coding
• Large probabilities are desired in arithmetic
coding
• Can use context-adaptive method to create larger
probability
• and to improve compression ratio.

20
Incremental Decoding
Input 1100011
Decode 1 Need 5 bits (verify) Read 6 bits
Tag 110001, 0.765625
0
0.8 1.0
• Summary Complete all possible scaling before
further decoding
• Adjust LOW, HIGH and Tag together.

21
Summary Part I
• Introduction
• Encoding and Decoding
• Scaling and Incremental Coding
• E1, E2
• Next
• Integer Implementation
• E3 scaling
• Binary Arithmetic Coding
• Applications
• JBIG, H.264, JPEG 2000

22
Outline Part II
• Review
• Integer Implementation
• Integer representation
• E3 Scaling
• Minimum word length
• Binary Arithmetic Coding

23
Encoding Without Scaling
Range 1
• Input sequence 1321

Final range 0.7712, 0.773504) Encode 0.7712
24
E1 and E2 Scaling
• E1 LOW HIGH) in 0, 0.5)
• LOW 0.0xxxxxxx (binary),
• HIGH 0.0xxxxxxx.

0 0.5 1.0
25
Encoding with E1 and E2
Symbol Prob.
1 0.8
2 0.02
3 0.18
Input 1
0
0.8 1.0
Encode any value in the tag, e.g., 0.5
Output 1 All outputs 1100011
26
To verify
• LOW 0.5424    (0.10001010... in binary), HIGH
0.54816 (0.10001100... in binary).
• So we can send out 10001 (0.53125)
• Equivalent to E2?E1?E1?E1?E2
• After left shift by 5 bits
• LOW (0.5424 0.53125) x 32 0.3568
• HIGH (0.54816 0.53125) x 32 0.54112
• Same as the result in the last page.

27
Encoding Pseudo Code with E1, E2
(For floating-point implementation)
Symbol Prob. CDF
1 0.8 0.8
2 0.02 0.82
3 0.18 1
• EncodeSymbol(n)
• //Update variables
• RANGE HIGH - LOW
• HIGH LOW RANGE CDF(n)
• LOW LOW RANGE CDF(n-1)
• //keep scaling before encoding next symbol
• while LOW, HIGH in 0, 0.5) or 0.5, 1)
• send 0 for E1 and 1 for E2
• scale LOW, HIGH

28
Incremental Decoding
Input 1100011
Decode 1 Need 5 bits (verify) Read 6 bits
Tag 110001, 0.765625
0
0.8 1.0
• Summary Complete all possible scaling before
further decoding
• Adjust LOW, HIGH and Tag together.

29
Decoding Pseudo Code with E1, E2
(For floating-point implementation)
• DecodeSymbol(Tag)
• RANGE HIGH - LOW
• n 1
• While ( (tag - LOW) / RANGE gt CDF(n) )
• n
• HIGH LOW RANGE CDF(n)
• LOW LOW RANGE CDF(n-1)
• //keep scaling before decoding next symbol
• while LOW, HIGH in 0, 0.5) or 0.5, 1)
• scale LOW, HIGH by E1 or E2 rule
• Left shift Tag and read one more bit to LSB
• return n

Symbol Prob. CDF
1 0.8 0.8
2 0.02 0.82
3 0.18 1
30
Outline
• Review
• Integer Implementation
• Integer representation
• E3 Scaling
• Complete Algorithm
• Minimum word length
• Binary Arithmetic Coding
• Applications
• JBIG, H.264, JPEG 2000

31
Integer Implementation
• Old formulas

HIGH ? LOW RANGE CDF(n) LOW ? LOW RANGE
CDF(n-1)
• Integer approximation of CDF ( )
• The number of occurrence of each symbol is
usually collected by a counter.

k P(k) nk Cum(k)
0 - - 0
1 0.8 40 40
2 0.02 1 41
3 0.18 9 50
32
Integer Implementation
HIGH ? LOW RANGE CDF(n) LOW ? LOW RANGE
CDF(n-1)
• Why 1 in RANGE and 1 in HIGH?
• HIGH should be less than the LOW of the next
interval
• The best integer value is HIGH (next LOW) 1
• HIGH, HIGH 1) still belongs to the current
interval,
• although we could not represent it explicitly.

33
Example
k P(k) Nk Cum(k)
0 - - 0
1 0.8 40 40
2 0.02 1 41
3 0.18 9 50
• For 8-bit integers, initial LOW 0, HIGH 255 ?
RANGE256

34
E1 Scaling for Integer
• E1 Scaling 0, 0.5) ? 0, 1), E1(x) 2 x.
• LOW 0xxxxxxx, HIGH 0xxxxxxx
• Output the MSB value 0, then shift left by 1 bit
• Important trick Shift in 1 to HIGH and 0 to LOW
• HIGH 0xxxxxxx ? xxxxxxx1 HIGH (HIGH ltlt 1)
1
• LOW 0xxxxxxx ? xxxxxxx0 LOW LOW ltlt 1
• Always assume HIGH ends with infinite number of
1s
• So that it approximates the LOW of the next
interval.
• This also ensures the RANGE is doubled after
scaling
• HIGH LOW 1 ? (2 x HIGH 1 2 x LOW 1)
2(HIGH LOW 1)

35
E2 Scaling for Integer
• E2 Scaling 0.5, 1) ? 0, 1), E2(x) 2 (x -
0.5)
• LOW 1xxxxxxx, HIGH 1xxxxxxx
• Output the MSB, then shift left by 1 bit (mul by
2)
• Same trick Shift in 1 to HIGH and 0 to LOW
• HIGH 1xxxxxxx ? xxxxxxx1 HIGH (HIGH ltlt 1)
1
• LOW 1xxxxxxx ? xxxxxxx0 LOW LOW ltlt 1

36
Integer Encoding
0, 0.8) ? LOW 0, HIGH 203. 0.8,
0.82) ? LOW 204, HIGH 208. Can we represent
an interval in 203, 204) ? (Sequence 1333333)
Input 1
0
203 255
• LOW 167 (10100111) HIGH 203 (11001011)
• After E2 (shift in an 1 to HIGH and 0 to LOW)
• LOW 1(01001110) 78 (8-bit) HIGH 1(10010111)
151 (8-bit)
• In 8.1 format (8 bits for integer, 1 bit for
fractional)
• LOW (10100111.0) 167 HIGH (11001011.1) 203.5
• By shifting in an 1 to HIGH, we can cover the
range 203, 203.5.
• The entire range 203, 204) can be covered by
always shifting in 1 to HIGH.

37
Outline
• Review
• Integer Implementation
• Integer representation
• E3 Scaling
• Complete Algorithm
• Minimum word length
• Binary Arithmetic Coding

38
E3 Scaling 0.25, 0.75)?0, 1)
• If RANGE straddles 1/2, E1 and E2 cannot be
applied,
• but the range can be quite small
• Example LOW0.4999, HIGH0.5001
• Binary LOW0.01111., HIGH0.1000
• We may not have enough bits to represent the
interval.

39
Integer Implementation of E3
• Same trick Shift in 1 to HIGH and 0 to LOW
• HIGH ((HIGH QUARTER) ltlt 1) 1
• LOW (LOW - QUARTER) ltlt 1
• QUARTER 2(M - 2) for m-bit integer. (64 for m
8 bits)

LOW 01xxxxxx ? 0xxxxxx0 HIGH 10xxxxxx ? 1xxxxxx1
• ?Another way to implement E3 (Sayood book pp.
99)
• Left shift old LOW and HIGH, complement new MSB.

40
Signaling of E3
• What should we send when E3 is used?
• Recall we send 1 if E2 is used, send 0 if E1 is
used
• What do they mean?
• A series of E3 followed by an E1 is equivalent to
an E1 followed by a series of E2.
• A series of E3 followed by an E2 is equivalent to
an E2 followed by a series of E1.

41
Example
• Previous example without E3

Input 1
0
0.8 1.0
Input 3
0 0.656 0.8
• With E3
• The range after E2E3 is the same as that after
E1E2

42
Another View of the Equivalence
• Scaling of a range in 0.5, 0.75) with E1E2

E2
E1
• Equivalent scaling of the range in 0.5, 0.75)
with E2E3

E3
E2
43
A Simple Proof of E2E3 E1E2
• Given an original range r
• After applying E2 0.5, 1) ?0, 1), the range
becomes
• r1 (r 0.5) x 2
• After applying E1 0, 0.5) ? 0, 1), the range
becomes
• r2 r1 x 2 ((r 0.5) x 2) x 2
• Given the same range r
• After applying E3 0.25, 0.75) ? 0, 1), the
range becomes
• r3 (r 0.25) x 2
• After applying E2, the range becomes
• r4 (r3 0.5) x 2 ((r 0.25) x 2 0.5) x 2
• (r 0.5) x 2 x 2
• r2

44
Encoding Operation with E3
• Without E3

Input 2
0.312
0.5424 0.54816 0.6
• With E3

Output 0 here!
• Dont send anything when E3 is used, but send a 0
after E2
• The bit stream is identical to that of the old
method
• Subsequent encoding is also same because of the
same final interval

45
Decoding for E3
Input 1100011 Read 6 bits Tag 110001
(0.765625) Decode 1
0
0.8 1.0
Same status as the old method low, high, range,
tag.
46
Summary of Different Scalings
0 0.25 0.5
0.75 1.0
Need E1 scaling
Need E2 scaling
Need E3 scaling
No scaling is required. Ready to encode/decode
the next symbol.
47
Outline
• Review
• Integer Implementation
• Integer representation
• E3 Scaling
• Complete Algorithm
• Minimum word length
• Binary Arithmetic Coding

48
Encoding Pseudo Code with E1, E2, E3
(For integer implementation)
• EncodeSymbol(n)
• //Update variables
• RANGE HIGH - LOW 1
• HIGH HIGH RANGE Cum( n ) / N - 1
• LOW LOW RANGE Cum(n-1) / N
• //Scaling before encoding next symbol
• EncoderScaling() //see next slide

49
Encoding Pseudo Code with E1, E2, E3
• EncoderScaling( )
• while (E1, E2 or E3 is possible)
• if (E3 is possible)
• HIGH ((HIGH - QUARTER) ltlt 1) 1
• LOW (LOW - QUARTER) ltlt 1
• Scale3 //Save number of E3, but send
nothing
• if (E1 or E2 is possible)
• Let b0 for E1 and b1 for E2
• send b
• HIGH (HIGH ltlt 1) 1
• LOW (LOW ltlt 1)
• while (Scale3 gt 0) //send info about E3
now
• send complement of b //E2 (E3)n (E1)n
E2
• Scale3 -- //Send one bit for each E3

50
Decoding Pseudo Code with E1, E2, E3
(For integer implementation)
Intervals 0, 203, 204, 208, 209, 255
• DecodeSymbol(Tag)
• RANGE HIGH - LOW 1
• n 1
• While (Tag gt LOW RANGE Cum(n) / N - 1)
• n
• HIGH LOW RANGE Cum(n) / N - 1
• LOW LOW RANGE Cum(n-1) / N
• //keep scaling before decoding next symbol
• DecoderScaling(Tag) //next slide
• return n

51
Decoding Pseudo Code with E1, E2, E3
• DecoderScaling(Tag)
• while (E1, E2 or E3 is possible)
• if (E1 or E2 is possible)
• LOW LOW ltlt 1
• HIGH (HIGH ltlt 1) 1
• Tag Tag ltlt 1
• if (E3 is possible)
• LOW (LOW - QUARTER) ltlt 1
• HIGH ((HIGH - QUARTER) ltlt 1) 1
• Tag (Tag - QUARTER) ltlt 1

52
Integer Encoding with E1, E2, E3
Input 1
0
203 255
Final output 11000100 10000000
53
Integer Decoding with E1, E2, E3
Input 11000100 10000000
Read 8 bits 11000100 (196) Decode 1
0
203 255
54
Outline
• Review
• Integer Implementation
• Integer representation
• E3 Scaling
• Complete Algorithm
• Minimum word length
• Binary Arithmetic Coding

55
How to decide the word length m?
• Need to guarantee non-zero interval for all
symbols
• in the worst case

k P(k) Nk Cum(k)
0 - - 0
1 0.8 40 40
2 0.02 1 41
3 0.18 9 50
? RANGE cannot be too small at any time.
Intuitive!
56
How to decide the word length m?
• When do we have the smallest RANGE without
triggering a scaling?
• Condition 1/4 (2m) gt N
• Example N 50, ? min m 8 (1/4M64)

57
Outline
• Review
• Integer Implementation
• Binary Arithmetic Coding

58
Binary Arithmetic Coding
• Arithmetic coding is slow in general
• To decode a symbol, we need a seris of decisions
and multiplications
• While (Tag gt LOW RANGE Cum(n) / N - 1)
• n
• The complexity is greatly reduced if we have only
two symbols 0 and 1.

symbol 0 symbol 1
0 x 1
• Only two intervals in 0, 1) 0, x), x, 1)

59
Encoding of Binary Arithmetic Coding
Only need to update LOW or HIGH for each symbol.
60
Decoding of Binary Arithmetic Coding
Tag
0 0.6 1
Only one decision to make
While (Tag gt LOW RANGE Cum(n) / N - 1))
n
if (Tag gt LOW RANGE Cum(Symbol 0) / N - 1)
n 1 else n 0
61
Applications of Binary Arithmetic Coding
• Increasingly popular
• JBIG, JBIG2, JPEG2000, H.264
• Covert non-binary signals into binary first
• H.264 Golomb-Rice Code
• Bit-plane coding
• Various simplifications to avoid multiplication
• H.264 Table look-up for RANGE Cum(n) / N
• JBIG Eliminate multiplication by assuming the
RANGE is close to 1. Scale if RANGE too small.

62
Outline
• Review
• Integer Implementation
• Binary Arithmetic Coding

63
• Observation The partition of 0, 1) can be
different from symbol to symbol
• The bit stream can be decoded perfectly as long
as both encoder and decoder are synchronized (use
the same partition).
• General approach
• Starting from a pre-defined probability
distribution
• Update probability after each symbol
• This is very difficult for Huffman coding
• Has to redesign the codebook when prob changes

64
Example
• Binary sequence 01111
• Initial counters for 0s and 1s
• C(0)C(1)1.
• ? P(0)P(1)0.5

0 0.5 1
65
Decoding
• Input 0.4667.
• Initial counters for 0s and 1s
• C(0)C(1)1 ? P(0)P(1)0.5
• Decode 0

0 0.5 1
66
• In many cases, a sample has strong correlation
with its near neighbors.
• Idea
• Collect conditional probability distribution of a
symbol for given neighboring symbols (context)
• P(x(n) x(n-1), x(n-2), x(n-k))
• Use this conditional probability to encode the
next symbol
• More skewed probability distribution can be
obtained (desired by arithmetic coding)

a b c b c a b c b a b c b a
a b c b c a b
1-D Context template
2-D Context template
67
Example
0 1 1 0 1 0 1
• Binary sequence 0, 1
• Neighborhood (template) size 3
• ?238 possible combinations (contexts) of 3
neighbors
• (x(n-1), x(n-2), x(n-3)).
• Collect frequencies of 0s and 1s under each
context

Context C(0) C(1)
(0, 0, 0) 9 2
(0, 0, 1) 3 6
(0, 1, 0) 10 10
(0, 1, 1)
(1, 0, 0)
(1, 0, 1)
(1, 1, 0)
(1, 1, 1)
Each symbol is coded with the probability
distribution associated with its context.
68
Encoding Pseudo Code
• InitProbModel(ProbModel)
• While (not EOF)
• //Determine the neighboring combination
(context)
• context GetContext()
• //Encode with the corresponding probability
model
• EncodeSymbol(ProbModelcontext, n)
• //update probability counters for this context
• UpdateProb(ProbModelcontext, n)

69
Decoding Pseudo Code
• InitProbModel(ProbModel)
• While (not EOF)
• //Determine the neighboring combination
(context)
• context GetContext()
• //Decode with the corresponding probability
model
• n DecodeSymbol(ProbModelcontext)
• //update probability counters for this context
• UpdateProb(ProbModelcontext, n)

70
Performance of Arithmetic Coding
• For a sequence of length m
• H(X) lt Average Bit Rate lt H(X) 2/m
• Can approaches the entropy very quickly.
• Huffman coding
• H(X) lt Average Bit Rate lt H(X) 1/m
• Impractical need to generate codewords for all
sequences of length m.

71
Summary Part II
• Arithmetic coding
• Partition the range 0, 1) recursively according
to symbol probabilities.
• Incremental Encoding and Decoding
• E1, E2, E3 scaling
• Binary Arithmetic Coding