Information theory in the Modern Information Society

1
Information theory in the Modern Information
Society

• A.J. Han Vinck
• University of Duisburg/Essen
• January 2003
• Vinck_at_exp-math.uni-essen.de

2
content

• What is Information theory ?
• Why is it important ?
• Where do we find it ?

3
Shannon (1948) , Information theory, The
Mathematical theory of Communication
Claude Elwood Shannon April 30, 1916 - February
24, 2001
4
What is Information theory about ?
Information knowledge that can be
used Communication exchange of
Information Our goal efficient reliable
secure

5
Express everything in 0 and 1
Discrete ensemble a,b,c,d ? 00, 01, 10, 11 in
general k binary digits specify 2k
messages Analogue signal 1) sample and 2)
represent sample value binary
11 10 01 00
v
Output 00, 10, 01, 01, 11
t
6
Shannons contributions

• Modeling how to go from analogue to digital
• fundamental communication models
• Bounds how far can we go?
• achievability
• impossibility
• Constructions constructive communication
  methods
• with optimum performance
• and many more!!!

1011
P
R
7
efficient general problem statement

• remove redundancy exact, no errors !!
• remove irrelevance distortion !!

Topics how ? how good ? how fast ? how
complex ?
8
efficient text

• represent every symbol with 8 bit
• ? 1 book 8 (500 pages) 1000 symbols 4
  Mbit ? 1 book
• ? compression possible to 1 Mbit (14) 

9
efficient speech

• sampling speed 8000 samples/sec accuracy 8
  bits/sample
• speed 64 kBit/s
• 45 minutes lecture 456064k 180Mbit ? 45
  books
•      compression possible to 4.8 kBit/s (110)

10
efficient CD music

• sampling speed 44.1 k samples/sec accuracy 16
  bits/sample
• storage capacity for one hour stereo 5
  Gbit ? 1250 books
• ? compression possible to 4 bits/sample ( 14
  )
•  

11
efficient digital pictures

• 300 x 400 pixels x 3 colors x 8 bit/sample
•         2.9 Mbit/picture for 25 images/second
  we need 75 Mb/s
• 2 hour pictures need 540 Gbit ? 130.000 books
•        compression needed (1100)

12
efficient summary

• text
• ? 1 book storage 4 Mbit ? 1 book
• speech
• ? 45 minutes lecture 456064k 180Mbit ? 45
  books
• CD music
• storage capacity for one hour stereo 5 Gbit
  ? 1250 books
•  
• digital pictures  
• 2 hour pictures need 540 Gbit ? 130.000
  books     

13
efficient general idea

• represent likely symbols with short length
  binary words
• where likely is derived from
• - prediction of next symbol in source output
• context between the source
  symbols words sounds context in pictures

q
qu
q-ue, q-ua, q-ui, q-uo
14
Morse
15
efficient applications

• Text Zip GIF etc.
• Music MP3
• Pictures JPEG, MPEG
• Contributors in data reduction/compression
• Information theorists A. Lempel and Jacob Ziv
• Huffman a.m.m.

16
efficient example JPEG
17
Secure example 1
Problem Is B the owner of the open lock?
18
Secure classical
Problem Is the key present at B?
19
Secure example 2
20
Reliable
Transmit 0 or 1
Receive 0 or 1
What can we do about it ?
0 0 correct 0 1 in - correct
1 1 correct 1 0 in -
correct
21
Reliable 2 examples
Transmit A 0 0 B 1 1
Receive 0 0 or 1 1 OK
0 1 or 1 0 NOK 1 error detected!
A 0 0 0 B 1 1 1
000, 001, 010, 100 ? A 111, 110, 101, 011 ? B 1
error corrected!
22
Error Sensitivity Illustration
Error sensitivity 0.00010.01
Error sensitivity 0.00050.05
23
Optical Storage

• DVD's seven-fold increase in data capacity over
  the CD has been largely achieved by tightening up
  the tolerances throughout the predecessor system
• The data structure was made more efficient by
  using a better, more efficient error correction
  code system.

24
Errors in networking
25
no- comment
26
a meshed structure
partial

• Fundamental problems to consider
• fast re-routing of information
• how to include redundancy ?
• how much redundancy?
• Cost versus reliability

3 links down
27
The success story Qualcomm CDMA
Founding information theorists Irwin Jacobs and
Andrew Viterbi
28
narrow-band and broad-band noise
time
1 2 3
frequency
SOLUTION FREQUENCY and TIME DIVISION
29
PPM Code Example

• 6 code words 123 231 312 132 213 321
• distance 2

1 2 3
30
Code Division Multiple Access (CDMA)

• 6 users 123 231 312 132 213 321
• 1 2 3 4
  5 6
• 2 transmit at the same time

1 2 3
2/3 2/3 2/3 5/6 5
1/2 2/4 2/4 4/5
2/5 2 2
31
Why IT at this university?

• It is fundamental. The theory is well established
• Based on
• Discrete Mathematics algorithms
• Physics
• Applications
• Communications networking Computer science
• Multi-media medical imaging biology languages
• Information retrieval information control

32
Other application powerline communications
33
Information Theory
In 1948, Bell Labs scientist Claude Shannon
developed Information Theory, and the world of
communications technology has never been the
same.