Arithmetic Coding Additional Material Spring 2015 - PowerPoint PPT Presentation

Loading...

PPT – Arithmetic Coding Additional Material Spring 2015 PowerPoint presentation | free to view - id: 7c1c77-NjBhN



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

Arithmetic Coding Additional Material Spring 2015

Description:

CMPT 365 Multimedia Systems Arithmetic Coding Additional Material Spring 2015 Context-adaptive Arithmetic Coding In many cases, a sample has strong correlation with ... – PowerPoint PPT presentation

Number of Views:50
Avg rating:3.0/5.0
Slides: 72
Provided by: sfu59
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Arithmetic Coding Additional Material Spring 2015


1
CMPT 365 Multimedia Systems
Arithmetic CodingAdditional Material Spring
2015
2
Outline Part I
  • Introduction
  • Basic Encoding and Decoding
  • Scaling and Incremental Coding
  • Integer Implementation
  • Adaptive Arithmetic Coding
  • Binary Arithmetic Coding
  • Applications
  • JBIG, H.264, JPEG 2000

3
Limitations of Huffman Code
  • Need a probability distribution
  • Hard to adapt to changing statistics
  • Minimum codeword length is 1 bit
  • Serious penalty for high-probability symbols
  • Example Binary source, P(0)0.9
  • Entropy -0.9log2(0.9)-0.1log2(0.1) 0.469 bit
  • Huffman code 0, 1 ? Avg. code length 1 bit
  • Joint coding is not practical for large alphabet.
  • Arithmetic coding
  • Can resolve all of these problems.
  • Code a sequence of symbols without having to
    generate codes for all sequences of that length.

4
Introduction
  • Recall table look-up decoding of Huffman code
  • N alphabet size
  • L Max codeword length
  • Divide 0, 2L into N intervals
  • One interval for one symbol
  • Interval size is roughly
  • proportional to symbol prob.

1
00
010 011
dcba..
5
Arithmetic Coding
1
0
a b c
  • Disjoint and complete partition of the range 0,
    1)
  • 0, 0.8), 0.8, 0.82), 0.82, 1)
  • Each interval corresponds to one symbol
  • Interval size is proportional to symbol
    probability
  • Observation once the tag falls into an interval,
    it never gets out of it

6
Some Questions to think about
  • Why compression is achieved this way?
  • How to implement it efficiently?
  • How to decode the sequence?
  • Why is it better than Huffman code?

7
Example
Symbol Prob.
1 0.8
2 0.02
3 0.18
  • Map to real line range 0, 1)
  • Order does not matter
  • Decoder need to use the same order
  • Disjoint but complete partition
  • 1 0, 0.8) 0, 0.7999999
  • 2 0.8, 0.82) 0.8, 0.8199999
  • 3 0.82, 1) 0.82, 0.9999999
  • (Think about the impact to integer implementation)

8
Encoding
  • Input sequence 1321

Range 1
Final range 0.7712, 0.773504) Encode 0.7712
Difficulties 1. Shrinking of interval requires
high precision for long sequence. 2. No
output is generated until the entire sequence has
been processed.
9
Cumulative Density Function (CDF)
  • For continuous distribution
  • Properties
  • Non-decreasing
  • Piece-wise constant
  • Each segment is closed at the lower end.

10
Encoder Pseudo Code
low0.0, high1.0 while (not EOF) n
ReadSymbol() RANGE HIGH - LOW HIGH LOW
RANGE CDF(n) LOW LOW RANGE
CDF(n-1) output LOW
  • Keep track of LOW, HIGH, RANGE
  • Any two are sufficient, e.g., LOW and RANGE.

Input HIGH LOW RANGE
Initial 1.0 0.0 1.0
1 0.01.00.80.8 0.01.00 0.0 0.8
3 0.0 0.810.8 0.0 0.80.820.656 0.144
2 0.6560.1440.820.77408 0.6560.1440.80.7712 0.00288
1 0.77120.0028800.7712 0.77120.002880.80.773504 0.002304
11
Decoding
Decode 1
Drawback need to recalculate all thresholds each
time.
12
Simplified Decoding
  • Normalize RANGE to 0, 1) each time
  • No need to recalculate the thresholds.

13
Decoder Pseudo Code
  • Low 0 high 1
  • x Encoded_number
  • While (x ? low)
  • n DecodeOneSymbol(x)
  • output symbol n
  • x (x - CDF(n-1)) / (CDF(n) - CDF(n-1))

14
Outline
  • Introduction
  • Basic Encoding and Decoding
  • Scaling and Incremental Coding
  • Integer Implementation
  • Adaptive Arithmetic Coding
  • Binary Arithmetic Coding
  • Applications
  • JBIG, H.264, JPEG 2000

15
Scaling and Incremental Coding
  • Problems of Previous examples
  • Need high precision
  • No output is generated until the entire sequence
    is encoded
  • Key Observation
  • As the RANGE reduces, many MSBs of LOW and HIGH
    become identical
  • Example Binary form of 0.7712 and 0.773504
  • 0.1100010.., 0.1100011..,
  • We can output identical MSBs and re-scale the
    rest
  • ? Incremental encoding
  • This also allows us to achieve infinite precision
    with finite-precision integers.

16
E1 and E2 Scaling
  • E1 LOW HIGH) in 0, 0.5)
  • LOW 0.0xxxxxxx (binary),
  • HIGH 0.0xxxxxxx.

0 0.5 1.0
17
Encoding with E1 and E2
Symbol Prob.
1 0.8
2 0.02
3 0.18
Input 1
0
0.8 1.0
Encode any value in the tag, e.g., 0.5
Output 1 All outputs 1100011
18
To verify
  • LOW 0.5424    (0.10001010... in binary), HIGH
    0.54816 (0.10001100... in binary).
  • So we can send out 10001 (0.53125)
  • Equivalent to E2?E1?E1?E1?E2
  • After left shift by 5 bits
  • LOW (0.5424 0.53125) x 32 0.3568
  • HIGH (0.54816 0.53125) x 32 0.54112
  • Same as the result in the last page.

19
Comparison with Huffman
  • Note Complete all possible scaling before
  • encoding the next symbol

Symbol Prob.
1 0.8
2 0.02
3 0.18
  • Input Symbol 1 does not cause any output
  • Input Symbol 3 generates 1 bit
  • Input Symbol 2 generates 5 bits
  • Symbols with larger probabilities generates less
    number of bits.
  • Sometimes no bit is generated at all
  • ? Advantage over Huffman coding
  • Large probabilities are desired in arithmetic
    coding
  • Can use context-adaptive method to create larger
    probability
  • and to improve compression ratio.

20
Incremental Decoding
Input 1100011
Decode 1 Need 5 bits (verify) Read 6 bits
Tag 110001, 0.765625
0
0.8 1.0
  • Summary Complete all possible scaling before
    further decoding
  • Adjust LOW, HIGH and Tag together.

21
Summary Part I
  • Introduction
  • Encoding and Decoding
  • Scaling and Incremental Coding
  • E1, E2
  • Next
  • Integer Implementation
  • E3 scaling
  • Adaptive Arithmetic Coding
  • Binary Arithmetic Coding
  • Applications
  • JBIG, H.264, JPEG 2000

22
Outline Part II
  • Review
  • Integer Implementation
  • Integer representation
  • E3 Scaling
  • Minimum word length
  • Binary Arithmetic Coding
  • Adaptive Arithmetic Coding

23
Encoding Without Scaling
Range 1
  • Input sequence 1321

Final range 0.7712, 0.773504) Encode 0.7712
24
E1 and E2 Scaling
  • E1 LOW HIGH) in 0, 0.5)
  • LOW 0.0xxxxxxx (binary),
  • HIGH 0.0xxxxxxx.

0 0.5 1.0
25
Encoding with E1 and E2
Symbol Prob.
1 0.8
2 0.02
3 0.18
Input 1
0
0.8 1.0
Encode any value in the tag, e.g., 0.5
Output 1 All outputs 1100011
26
To verify
  • LOW 0.5424    (0.10001010... in binary), HIGH
    0.54816 (0.10001100... in binary).
  • So we can send out 10001 (0.53125)
  • Equivalent to E2?E1?E1?E1?E2
  • After left shift by 5 bits
  • LOW (0.5424 0.53125) x 32 0.3568
  • HIGH (0.54816 0.53125) x 32 0.54112
  • Same as the result in the last page.

27
Encoding Pseudo Code with E1, E2
(For floating-point implementation)
Symbol Prob. CDF
1 0.8 0.8
2 0.02 0.82
3 0.18 1
  • EncodeSymbol(n)
  • //Update variables
  • RANGE HIGH - LOW
  • HIGH LOW RANGE CDF(n)
  • LOW LOW RANGE CDF(n-1)
  • //keep scaling before encoding next symbol
  • while LOW, HIGH in 0, 0.5) or 0.5, 1)
  • send 0 for E1 and 1 for E2
  • scale LOW, HIGH

28
Incremental Decoding
Input 1100011
Decode 1 Need 5 bits (verify) Read 6 bits
Tag 110001, 0.765625
0
0.8 1.0
  • Summary Complete all possible scaling before
    further decoding
  • Adjust LOW, HIGH and Tag together.

29
Decoding Pseudo Code with E1, E2
(For floating-point implementation)
  • DecodeSymbol(Tag)
  • RANGE HIGH - LOW
  • n 1
  • While ( (tag - LOW) / RANGE gt CDF(n) )
  • n
  • HIGH LOW RANGE CDF(n)
  • LOW LOW RANGE CDF(n-1)
  • //keep scaling before decoding next symbol
  • while LOW, HIGH in 0, 0.5) or 0.5, 1)
  • scale LOW, HIGH by E1 or E2 rule
  • Left shift Tag and read one more bit to LSB
  • return n

Symbol Prob. CDF
1 0.8 0.8
2 0.02 0.82
3 0.18 1
30
Outline
  • Review
  • Integer Implementation
  • Integer representation
  • E3 Scaling
  • Complete Algorithm
  • Minimum word length
  • Binary Arithmetic Coding
  • Adaptive Arithmetic Coding
  • Applications
  • JBIG, H.264, JPEG 2000

31
Integer Implementation
  • Old formulas

HIGH ? LOW RANGE CDF(n) LOW ? LOW RANGE
CDF(n-1)
  • Integer approximation of CDF ( )
  • The number of occurrence of each symbol is
    usually collected by a counter.
  • Allow adaptive arithmetic coding

k P(k) nk Cum(k)
0 - - 0
1 0.8 40 40
2 0.02 1 41
3 0.18 9 50
32
Integer Implementation
HIGH ? LOW RANGE CDF(n) LOW ? LOW RANGE
CDF(n-1)
  • Why 1 in RANGE and 1 in HIGH?
  • HIGH should be less than the LOW of the next
    interval
  • The best integer value is HIGH (next LOW) 1
  • HIGH, HIGH 1) still belongs to the current
    interval,
  • although we could not represent it explicitly.

33
Example
k P(k) Nk Cum(k)
0 - - 0
1 0.8 40 40
2 0.02 1 41
3 0.18 9 50
  • For 8-bit integers, initial LOW 0, HIGH 255 ?
    RANGE256

34
E1 Scaling for Integer
  • E1 Scaling 0, 0.5) ? 0, 1), E1(x) 2 x.
  • LOW 0xxxxxxx, HIGH 0xxxxxxx
  • Output the MSB value 0, then shift left by 1 bit
  • Important trick Shift in 1 to HIGH and 0 to LOW
  • HIGH 0xxxxxxx ? xxxxxxx1 HIGH (HIGH ltlt 1)
    1
  • LOW 0xxxxxxx ? xxxxxxx0 LOW LOW ltlt 1
  • Always assume HIGH ends with infinite number of
    1s
  • So that it approximates the LOW of the next
    interval.
  • This also ensures the RANGE is doubled after
    scaling
  • HIGH LOW 1 ? (2 x HIGH 1 2 x LOW 1)
    2(HIGH LOW 1)

35
E2 Scaling for Integer
  • E2 Scaling 0.5, 1) ? 0, 1), E2(x) 2 (x -
    0.5)
  • LOW 1xxxxxxx, HIGH 1xxxxxxx
  • Output the MSB, then shift left by 1 bit (mul by
    2)
  • Same trick Shift in 1 to HIGH and 0 to LOW
  • HIGH 1xxxxxxx ? xxxxxxx1 HIGH (HIGH ltlt 1)
    1
  • LOW 1xxxxxxx ? xxxxxxx0 LOW LOW ltlt 1

36
Integer Encoding
0, 0.8) ? LOW 0, HIGH 203. 0.8,
0.82) ? LOW 204, HIGH 208. Can we represent
an interval in 203, 204) ? (Sequence 1333333)
Input 1
0
203 255
  • LOW 167 (10100111) HIGH 203 (11001011)
  • After E2 (shift in an 1 to HIGH and 0 to LOW)
  • LOW 1(01001110) 78 (8-bit) HIGH 1(10010111)
    151 (8-bit)
  • In 8.1 format (8 bits for integer, 1 bit for
    fractional)
  • LOW (10100111.0) 167 HIGH (11001011.1) 203.5
  • By shifting in an 1 to HIGH, we can cover the
    range 203, 203.5.
  • The entire range 203, 204) can be covered by
    always shifting in 1 to HIGH.

37
Outline
  • Review
  • Integer Implementation
  • Integer representation
  • E3 Scaling
  • Complete Algorithm
  • Minimum word length
  • Binary Arithmetic Coding
  • Adaptive Arithmetic Coding

38
E3 Scaling 0.25, 0.75)?0, 1)
  • If RANGE straddles 1/2, E1 and E2 cannot be
    applied,
  • but the range can be quite small
  • Example LOW0.4999, HIGH0.5001
  • Binary LOW0.01111., HIGH0.1000
  • We may not have enough bits to represent the
    interval.

39
Integer Implementation of E3
  • Same trick Shift in 1 to HIGH and 0 to LOW
  • HIGH ((HIGH QUARTER) ltlt 1) 1
  • LOW (LOW - QUARTER) ltlt 1
  • QUARTER 2(M - 2) for m-bit integer. (64 for m
    8 bits)

LOW 01xxxxxx ? 0xxxxxx0 HIGH 10xxxxxx ? 1xxxxxx1
  • ?Another way to implement E3 (Sayood book pp.
    99)
  • Left shift old LOW and HIGH, complement new MSB.

40
Signaling of E3
  • What should we send when E3 is used?
  • Recall we send 1 if E2 is used, send 0 if E1 is
    used
  • What do they mean?
  • A series of E3 followed by an E1 is equivalent to
    an E1 followed by a series of E2.
  • A series of E3 followed by an E2 is equivalent to
    an E2 followed by a series of E1.

41
Example
  • Previous example without E3

Input 1
0
0.8 1.0
Input 3
0 0.656 0.8
  • With E3
  • The range after E2E3 is the same as that after
    E1E2

42
Another View of the Equivalence
  • Scaling of a range in 0.5, 0.75) with E1E2

E2
E1
  • Equivalent scaling of the range in 0.5, 0.75)
    with E2E3

E3
E2
43
A Simple Proof of E2E3 E1E2
  • Given an original range r
  • After applying E2 0.5, 1) ?0, 1), the range
    becomes
  • r1 (r 0.5) x 2
  • After applying E1 0, 0.5) ? 0, 1), the range
    becomes
  • r2 r1 x 2 ((r 0.5) x 2) x 2
  • Given the same range r
  • After applying E3 0.25, 0.75) ? 0, 1), the
    range becomes
  • r3 (r 0.25) x 2
  • After applying E2, the range becomes
  • r4 (r3 0.5) x 2 ((r 0.25) x 2 0.5) x 2
  • (r 0.5) x 2 x 2
  • r2

For formal proof www.copro.org/download/ac_en.pdf
44
Encoding Operation with E3
  • Without E3

Input 2
0.312
0.5424 0.54816 0.6
  • With E3

Output 0 here!
  • Dont send anything when E3 is used, but send a 0
    after E2
  • The bit stream is identical to that of the old
    method
  • Subsequent encoding is also same because of the
    same final interval

45
Decoding for E3
Input 1100011 Read 6 bits Tag 110001
(0.765625) Decode 1
0
0.8 1.0
Same status as the old method low, high, range,
tag.
46
Summary of Different Scalings
0 0.25 0.5
0.75 1.0
Need E1 scaling
Need E2 scaling
Need E3 scaling
No scaling is required. Ready to encode/decode
the next symbol.
47
Outline
  • Review
  • Integer Implementation
  • Integer representation
  • E3 Scaling
  • Complete Algorithm
  • Minimum word length
  • Binary Arithmetic Coding
  • Adaptive Arithmetic Coding

48
Encoding Pseudo Code with E1, E2, E3
(For integer implementation)
  • EncodeSymbol(n)
  • //Update variables
  • RANGE HIGH - LOW 1
  • HIGH HIGH RANGE Cum( n ) / N - 1
  • LOW LOW RANGE Cum(n-1) / N
  • //Scaling before encoding next symbol
  • EncoderScaling() //see next slide

49
Encoding Pseudo Code with E1, E2, E3
  • EncoderScaling( )
  • while (E1, E2 or E3 is possible)
  • if (E3 is possible)
  • HIGH ((HIGH - QUARTER) ltlt 1) 1
  • LOW (LOW - QUARTER) ltlt 1
  • Scale3 //Save number of E3, but send
    nothing
  • if (E1 or E2 is possible)
  • Let b0 for E1 and b1 for E2
  • send b
  • HIGH (HIGH ltlt 1) 1
  • LOW (LOW ltlt 1)
  • while (Scale3 gt 0) //send info about E3
    now
  • send complement of b //E2 (E3)n (E1)n
    E2
  • Scale3 -- //Send one bit for each E3

50
Decoding Pseudo Code with E1, E2, E3
(For integer implementation)
Intervals 0, 203, 204, 208, 209, 255
  • DecodeSymbol(Tag)
  • RANGE HIGH - LOW 1
  • n 1
  • While (Tag gt LOW RANGE Cum(n) / N - 1)
  • n
  • HIGH LOW RANGE Cum(n) / N - 1
  • LOW LOW RANGE Cum(n-1) / N
  • //keep scaling before decoding next symbol
  • DecoderScaling(Tag) //next slide
  • return n

51
Decoding Pseudo Code with E1, E2, E3
  • DecoderScaling(Tag)
  • while (E1, E2 or E3 is possible)
  • if (E1 or E2 is possible)
  • LOW LOW ltlt 1
  • HIGH (HIGH ltlt 1) 1
  • Tag Tag ltlt 1
  • Tag Tag ReadBits(1)
  • if (E3 is possible)
  • LOW (LOW - QUARTER) ltlt 1
  • HIGH ((HIGH - QUARTER) ltlt 1) 1
  • Tag (Tag - QUARTER) ltlt 1
  • Tag Tag ReadBits(1)

52
Integer Encoding with E1, E2, E3
Input 1
0
203 255
Final output 11000100 10000000
53
Integer Decoding with E1, E2, E3
Input 11000100 10000000
Read 8 bits 11000100 (196) Decode 1
0
203 255
54
Outline
  • Review
  • Integer Implementation
  • Integer representation
  • E3 Scaling
  • Complete Algorithm
  • Minimum word length
  • Binary Arithmetic Coding
  • Adaptive Arithmetic Coding

55
How to decide the word length m?
  • Need to guarantee non-zero interval for all
    symbols
  • in the worst case

k P(k) Nk Cum(k)
0 - - 0
1 0.8 40 40
2 0.02 1 41
3 0.18 9 50
? RANGE cannot be too small at any time.
Intuitive!
56
How to decide the word length m?
  • When do we have the smallest RANGE without
    triggering a scaling?
  • Condition 1/4 (2m) gt N
  • Example N 50, ? min m 8 (1/4M64)

57
Outline
  • Review
  • Integer Implementation
  • Binary Arithmetic Coding
  • Adaptive Arithmetic Coding

58
Binary Arithmetic Coding
  • Arithmetic coding is slow in general
  • To decode a symbol, we need a seris of decisions
    and multiplications
  • While (Tag gt LOW RANGE Cum(n) / N - 1)
  • n
  • The complexity is greatly reduced if we have only
    two symbols 0 and 1.

symbol 0 symbol 1
0 x 1
  • Only two intervals in 0, 1) 0, x), x, 1)

59
Encoding of Binary Arithmetic Coding
Only need to update LOW or HIGH for each symbol.
60
Decoding of Binary Arithmetic Coding
Tag
0 0.6 1
Only one decision to make
While (Tag gt LOW RANGE Cum(n) / N - 1))
n
if (Tag gt LOW RANGE Cum(Symbol 0) / N - 1)
n 1 else n 0
61
Applications of Binary Arithmetic Coding
  • Increasingly popular
  • JBIG, JBIG2, JPEG2000, H.264
  • Covert non-binary signals into binary first
  • H.264 Golomb-Rice Code
  • Bit-plane coding
  • Various simplifications to avoid multiplication
  • H.264 Table look-up for RANGE Cum(n) / N
  • JBIG Eliminate multiplication by assuming the
    RANGE is close to 1. Scale if RANGE too small.

62
Outline
  • Review
  • Integer Implementation
  • Binary Arithmetic Coding
  • Adaptive Arithmetic Coding

63
Adaptive Arithmetic Coding
  • Observation The partition of 0, 1) can be
    different from symbol to symbol
  • The bit stream can be decoded perfectly as long
    as both encoder and decoder are synchronized (use
    the same partition).
  • General approach
  • Starting from a pre-defined probability
    distribution
  • Update probability after each symbol
  • This is very difficult for Huffman coding
  • Has to redesign the codebook when prob changes

64
Example
  • Binary sequence 01111
  • Initial counters for 0s and 1s
  • C(0)C(1)1.
  • ? P(0)P(1)0.5

0 0.5 1
65
Decoding
  • Input 0.4667.
  • Initial counters for 0s and 1s
  • C(0)C(1)1 ? P(0)P(1)0.5
  • Decode 0

0 0.5 1
66
Context-adaptive Arithmetic Coding
  • In many cases, a sample has strong correlation
    with its near neighbors.
  • Idea
  • Collect conditional probability distribution of a
    symbol for given neighboring symbols (context)
  • P(x(n) x(n-1), x(n-2), x(n-k))
  • Use this conditional probability to encode the
    next symbol
  • More skewed probability distribution can be
    obtained (desired by arithmetic coding)

a b c b c a b c b a b c b a
a b c b c a b
1-D Context template
2-D Context template
67
Example
0 1 1 0 1 0 1
  • Binary sequence 0, 1
  • Neighborhood (template) size 3
  • ?238 possible combinations (contexts) of 3
    neighbors
  • (x(n-1), x(n-2), x(n-3)).
  • Collect frequencies of 0s and 1s under each
    context

Context C(0) C(1)
(0, 0, 0) 9 2
(0, 0, 1) 3 6
(0, 1, 0) 10 10
(0, 1, 1)
(1, 0, 0)
(1, 0, 1)
(1, 1, 0)
(1, 1, 1)
Each symbol is coded with the probability
distribution associated with its context.
68
Encoding Pseudo Code
  • InitProbModel(ProbModel)
  • While (not EOF)
  • n ReadSymbol( )
  • //Determine the neighboring combination
    (context)
  • context GetContext()
  • //Encode with the corresponding probability
    model
  • EncodeSymbol(ProbModelcontext, n)
  • //update probability counters for this context
  • UpdateProb(ProbModelcontext, n)

69
Decoding Pseudo Code
  • InitProbModel(ProbModel)
  • While (not EOF)
  • //Determine the neighboring combination
    (context)
  • context GetContext()
  • //Decode with the corresponding probability
    model
  • n DecodeSymbol(ProbModelcontext)
  • //update probability counters for this context
  • UpdateProb(ProbModelcontext, n)

70
Performance of Arithmetic Coding
  • For a sequence of length m
  • H(X) lt Average Bit Rate lt H(X) 2/m
  • Can approaches the entropy very quickly.
  • Huffman coding
  • H(X) lt Average Bit Rate lt H(X) 1/m
  • Impractical need to generate codewords for all
    sequences of length m.

71
Summary Part II
  • Arithmetic coding
  • Partition the range 0, 1) recursively according
    to symbol probabilities.
  • Incremental Encoding and Decoding
  • E1, E2, E3 scaling
  • Binary Arithmetic Coding
  • Context Adaptive Arithmetic Coding
  • Next Quantization (lossy compression)
About PowerShow.com