Introduction to Coding Theory

1
Introduction to Coding Theory
2
Outline

• 1 Introduction
• 2 Basic assumptions
• 3 Correcting and detecting error patterns
• 4 Information rate
• 5 The effects of error correction and detection
• 6 Finding the most likely codeword transmitted
• 7 Some basic algebra
• 8 Weight and distance
• 9 Maximum likelihood decoding
• 10 Reliability of MLD
• 11 Error-detecting codes
• 12 Error-correcting codes

3
Introduction to Coding Theory

• 1 Introduction
• Coding theory
• The study of methods for efficient and accurate
  transfer of information
• Detecting and correcting transmission errors
• Information transmission system

n-digit
n-digit
k-digit
k-digit
4
Introduction to Coding Theory

• 2 Basic assumptions
• Definitions
• Digit0 or 1(binary digit)
• Worda sequence of digits
• Example0110101
• Binary codea set of words
• Example1. 00,01,10,11 , 2. 0,01,001
• Block code a code having all its words of the
  same length
• Example 00,01,10,11, 2 is its length
• Codewords words belonging to a given code
• C Size of a code C(codewords in C)

5
Introduction to Coding Theory

• Assumptions about channel

2. Identifying the beginning of 1st word
3. The probability of any digit being affected
in transmission is the same as the other one.
6
Introduction to Coding Theory

• Binary symmetric channel

p
In many books, p denotes crossover
probability. Here crossover probability(error
prob.) is 1-p
7
Introduction to Coding Theory

• 3 Correcting and detecting error patterns

Any received word should be corrected to a
codeword that requires as few changes as possible.
Cannot detect any errors !!!
parity-check digit
8
Introduction to Coding Theory

• 4 Information rate
• Definition information rate of code C
• Examples

9
Introduction to Coding Theory

• 5 The effects of error correction and detection
• 1. No error detection and correction

Let C0,1110000000000, , 11111111111
Transmission rate107 digits/sec
Reliability p1-10-8
Then Pr(a word is transmitted incorrectly)
1-p11 ?11x10-8
11x10-8(wrong words/words)x107/11(words/sec)0.1
wrong words/sec
1 wrong word / 10 sec 6 wrong words / min 360
wrong words / hr 8640 wrong words / day
10
Introduction to Coding Theory

• 2. parity-check digit added(Code length becomes
  12 )
• Any single error can be detected !
• (3, 5, 7, ..errors can be detected too !)
• Pr(at least 2 errors in a word)1-p12-12 x
  p11(1-p)?66x10-16
• So 66x10-16 x 107/12 ? 5.5 x 10-9 wrong
  words/sec

one word error every 2000 days!
The cost we pay is to reduce a little information
rate retransmission(after error detection!)
11
Introduction to Coding Theory
3. 3-repetition code Any single error can be
corrected ! Code length becomes 33 and
information rate becomes 1/3

• Taskdesign codes with
• reasonable information rates
• low encoding and decoding costs
• some error-correcting capabilities

12
Introduction to Coding Theory

• 6 finding the most likely codeword transmitted
• Example

p reliability d digits incorrectly
transmitted n code length
Code length 5
13
Introduction to Coding Theory

• Theorem 1.6.3
• Suppose we have a BSC with ½ lt p lt 1. Let
  and be codewords and a word, each of length
  . Suppose that and disagree in
  positions and and disagree in positions.
  Then

14
Introduction to Coding Theory

• Example

d (number of disagreements with )
01101 3
01001 4
10100 2 ? smallest d
10101 3
15
Introduction to Coding Theory

• 7 Some basic algebra

16
Introduction to Coding Theory

• Kn is a vector space

words of length n
scalar
17
Introduction to Coding Theory

• 8 Weight and distance
• Hamming weight
• the number of times the digit 1 occurs in
• Example
• Hamming distance
• the number of positions in which and
  disagree
• Example

18
Introduction to Coding Theory

• Some facts

19
Introduction to Coding Theory

• 9 Maximum likelihood decoding

wvu

• CMLDComplete Maximum Likelihood Decoding
• If only one word v in C closer to w , decode
  it to v
• If several words closest to w, select arbitrarily
  one of them
• IMLDIncomplete Maximum Likelihood Decoding
• If only one word v in C closer to w, decode it
  to v
• If several words closest to w, ask for
  retransmission

Source string x
codeword
Error pattern
20
Introduction to Coding Theory
The most likely codeword sent is the one with the
error pattern of smallest weight
ExampleConstruct IMLD. M3 , C0000, 1010,
0111
Received w Error Pattern Error Pattern Error Pattern Decode v
Received w 0000 w 1010 w 0111 w Decode v
0000 0000 1010 0111 0000
1000 1000 0010 1111 -
0100 0100 1110 0011 0000
0010 0010 1000 0101 -
0001 0001 1011 0110 0000
1100 1100 0110 1011 -
1010 1010 0000 1101 1010
1001 1001 0011 1110 -
0110 0110 1100 0001 0111
0101 0101 1111 0010 0111
21
Introduction to Coding Theory

• 10 Reliability of MLD
• The probability that if v is sent over a BSC of
  probability p then IMLD correctly concludes that
  v was sent

The higher the probability is, the more correctly
the word can be decoded!
22
Introduction to Coding Theory

• 11 Error-detecting codes

Example
Error Pattern u v 000 v 111
000 000 111
100 100 011
010 010 101
001 001 110
110 110 001
101 101 010
011 011 100
111 111 000
Cant detect
Can detect
23
Introduction to Coding Theory

• the distance of the code C
• the smallest of d(v,w) in C
• Theorem 1.11.14
• A code C of distance d will at least detect all
  non-zero error patterns of weight less than or
  equal to d-1. Moreover, there is at least one
  error pattern of weight d which C will not
  detect.
• t error-detecting code
• It detects all error patterns of weight at most t
  and does not detect at least one error pattern of
  weight t1
• A code with distance d is a d-1 error-detecting
  code.

24
Introduction to Coding Theory

• 12 Error-correcting codes
• Theorem 1.12.9
• A code of distance d will correct all error
  patterns of weight less than or equal to
  . Moreover, there is at least one error
  pattern of weight 1 which C will
  not correct.
• t error-correcting code
• It corrects all error patterns of weight at most
  t and does not correct at least one error pattern
  of weight t1
• A code of distance d is a
  error-correcting code.

25
Introduction to Coding Theory
Received Error Pattern Error Pattern Decode
w 000 w 111 w v
000 000 111 000
100 100 011 000
010 010 101 000
001 001 110 000
110 110 001 111
101 101 010 111
011 011 100 111
111 111 000 111
C corrects error patterns 000,100,010,001