Information Theory

1
UNIT-I
2
Classification of signals

• Multichannel
• (signals are generated by multiple
  sensors/sources which are measuring the same
  parameter. Example measurement of ground
  acceleration a few kilometers from epicenter of
  an earthquake, ECG etc)
• Multidimentional
• (The value of signal is a function of M
  independent variables.)

3

• Continuous time Vs Discrete time (may not be
  discrete valued)
• Continuous valued Vs discrete valued (quantized
  continuous valued discrete time signal)
• Deterministic Vs Random
• (Deterministic, if signal can be uniquely
  described by an explicit mathematical expression
  e. g. sine, cosine, line, parabola etc. Random
  signal examples noise, unsynchronized digital
  signal etc )

4
Basic sequences and operations

• Unit sample/discrete time impulse/impulse
• dn 1(if n0) 0 (otherwise)
• - Any sequence can be represented as a sum of
  scaled, delayed impulses e.g.
• if pn a-2,a-1, a0, a1,a2, .. then
• pn .. a-2 dn2 a-1dn1 a0 dn a1
  dn-1 ..
• More generally,
• xn ? xkdn-k

5

• Unit Step
• un 1(ngt0) 0 (nlt0)
• un ? dn-k (0ltklt8)
• dn un un-1
• Exponential
• xn Aan
• Sinusoidal
• xn Acos(?0nf) for all n

6
The Concept of frequency

• For Continuous - time sinusoidal signal x(t)
• x(tT) x(t) T is time period 1/F (freq.)
• Increasing F results in an increase in rate of
  oscillations (more periods are included in unit
  time)
• Two sinusoids x1(t) and x2(t) with distinct freq
  F1 and F2 are themselves distinct.

7

• For Discrete time sinusoids
• x(n) Acos(?nf) ? 2pf
• f (cycles per sample) F/Fs Fs is the sampling
  freq
• Periodic, if f is a rational number.
• for periodicity, x(nN) x(n) for all n i.e.
• Acos(2pf(nN)f) Acos(2pfnf)
• This relation is true if and only if there exists
  an integer K such that 2pfN 2pk i.e. fk/N
• If k and N are relatively prime then N is called
  the fundamental period of xn.
• A small change in freq can results in large
  change in period i.e. f1 31/60 implies N160 but
  f230/60 implies N2 2.

8

• Two sinusoids x1n and x2n whose frequencies
  are separated by an integer multiple of 2p are
  identical (more precisely, indistinguishable)
• Acos((? 2p)nf) Acos(2pn (?nf))
  Acos(?nf)
• Sinusoids with plt ? lt p i.e. -1/2ltflt1/2 are
  distinct. If it appears that a sinusoid has f
  outside this range then definitely in the above
  range its identical sinusoid does exists.

9

• The highest rate of oscillation in a discrete
  time sinusoid is attained when ? p (or - p) or
  equivalently f1/2 or -1/2.
• As ? varies from 0, p/8, p/4, p/2, p then f
  ?/2p will increases as 0, 1/16, 1/8, ¼, ½ and N
  8, 16, 8, 4, 2
• When ? gt p e.g. 3p/2 then f ¾ gt ½ which should
  be represented by its identical sinusoid cos(2p-
  ?)n.

10
Signal Types and Properties

• Energy Vs Power signals
• Energy of x(n) E ?-8ltnlt8x(n)2
• If E is finite, x(n) is called energy signal.
• Power of x(n) P lim(N tend to 8) 1/(2N1)
• 
  ?-NltnltNx(n)2
• If E is finite P0 but if E is infinite P may or
  may not be finite. (prove it?)
• If P is finite then x(n) is called power signal.

11

• Periodic Vs Aperiodic signals
• x(nN) x(n) for all n
• If no value of N satisfy it the signal is
  aperiodic.
• A sinusoid x(n) Asin2pfn is periodic if fk/N
• Power of periodic signals
• (1/N)?0ltnltN-1x(n)2

12

• Symmetric (even) Vs Antisymmetric (odd)
• x(n) x(-n) symmetric (even)
• x(n) -x(-n) antisymmetric (odd)
• Any arbitrary signal can be expressed as sum of
  two signal components, even and odd.
• xe(n) ½x(n) x(-n) xo(n) ½x(n) - x(-n)

13
Classification of systems

• Static (memoryless) Vs Dynamic (with memory)
• (static, if o/p depends upon present i/p only
  e.g. y(n) nx(n)bx3(n). Dynamic, if depends upon
  past or present e.g. y(n)x(n-1)3x(n))
• Time-invariant Vs Time Variant systems
• (if i/p, x(n) results in y(n) then x(n-k) must
  results in y(n-k), otherwise system is time
  variant. Ex TI, y(n)x(n)-x(n-1) TV, y(n)
  x(n)cos(wn))

14

• Linear Vs Nonlinear
• (linear, must satisfy superposition principle,
  otherwise nonlinear.)
• Causal Vs Noncausal
• (o/p should depend upon present and past i/p but
  not on future i/p x(n1).., otherwise
  noncausal. If signal is first recorded then
  offline processing is done then only non causal
  systems are possible to implement)
• Stable Vs Unstable
• (BIBO stable if every bounded i/p produces a
  bounded o/p, otherwise unstable.)

15
Convolution sum

• y(n) Tx(n)
• T represents the operation performed by the
  system
• If x(n) is an impulse then y(n) h(n) Td(n)
• y(n) T?(k -8 to 8) x(k) d(n-k)
• T..x(-1)d(n1)x(0) d(n)x(1)d(n-1)..
• x(-1)Td(n1)x(0)Td(n)x(1)Td(n-1)..
• By applying superposition property of linear
  systems
• ? (k -8 to 8) x(k)Td(n-k)
• Apply time-invariance property
• ? (k -8 to 8) x(k)h(n-k) x(n)h(n)

16
Graphical evaluation of convolution

• Fold seq h(n) to get h(-k).
• Multiply x(k) with h(-k) and add the results to
  get y(0).
• Shift h(-k) right to get h(1-k) and repeat step-2
  to get y(1).
• Repeat above step to get y(2), y(3)..
• Shift h(-k) left to get h(-1-k) and repeat step-2
  to get y(-1).
• Repeat above step to get y(-2), y(-3)..

17
Properties of convolution operation

• Commutative law
• y(n) x(n)h(n) h(n)x(n)
• The role of x(n) and h(n) can be interchanged
  (either seq can be folded)
• Distributive law
• y(n) x(n)h1(n)h2(n) x(n)h1(n)x(n)h2(n
  )
• Overall h(n) of two parallel systems equals the
  sum of individual h(n)s of subsystems
• Associative law
• y(n) x(n)h1(n)h2(n) x(n)h1(n)h2(n)
• x(n)h2(n)h1(n) (using commutative law)
• The overall h(n) is convolution of h(n)s of
  cascade subsystems
• Order of intermediate subsystems can be
  interchanged

18
Condition for Causality

• Using convolution sum,
• y(n) ?(k -8 to 8) h(k)x(n-k)
• ?(k 0 to 8) h(k)x(n-k) ?(k -8 to -1)
  h(k)x(n-k)
• h(0)x(n)h(1)x(n-1).. h(-1)x(n1)
  ..
• The first sum involves present and past values of
  x(n).
• The second sum involves future values of inputs
  thus all terms of this sum should be zero for
  causality.
• It is guaranteed if, h(n)0 for nlt0
• It is necessary and sufficient condition for
  causality.

19
Condition for BIBO Stability

• y(n) ?(k -8 to 8) h(k)x(n-k)
• lt ?(k -8 to 8) h(k)x(n-k)
• (absolute value of sum is always less than or
  equal to sum of absolute values)
• x(n) is called bounded when there exists a finite
  constant Mx such that x(n)ltMxlt8
• If i/p is bounded then,
• y(n)lt?(k -8 to 8)h(k)Mx Mx?(k -8 to 8)
  h(k)

20

• Applying same definition, y(n) is bounded if
  y(n)lt8.
• Thus, Mx?(k -8 to 8) h(k)lt8
• As Mx is constant thus sufficient condition for
  BIBO stability is Sh ?(k -8 to 8) h(k)lt8
• Is this necessary condition as well?
• (can be proved by showing that there exists a
  bounded i/p for which the o/p is unbounded if Sh
  is not bounded)

21

• If x(n) h(-n)/h(-n) then it will be a bounded
  seq for any value of n. (even if h(-n) is
  unbounded for some values of n then also x(n)
  will be bounded due to division by its absolute
  value. )
• Substitute in convolution formula, y(0) for above
  sequence will be Sh thus it is unbounded if, Sh
  is unbounded.
• Sh is bounded if h(n) approaches zero as n
  approaches infinity.
• If a limited duration x(n) is applied to a BIBO
  stable system then it will produce an transient
  O/P which will die out to zero at steady state.
  (prove it?)

22
FIR systems

• This classification depends upon system
  characteristics the impulse response.
• If h(n)0 for nlt0 and ngtM then causal FIR system
  (finite number of symbols in h(n))
• For a causal FIR system
• y(n) ?(k0 to M-1) h(k)x(n-k)
  h(0)x(n)h(1)x(n-1) ----- h(M)x(n-M1)

23

• To form the o/p at any instant nn0 the most
  recent M input samples are required.
• System must have a memory size M to remember past
  M Input symbols.
• This type of realization is called non recursive
  realization of systems.
• The FIR system acts as a window of size M thus
  the impulse response of FIR system is also called
  a window

24
IIR systems

• The h(n) has infinite number of symbols e.g.
  h(n) anu(n).
• y(n) ?(k0 to 8) h(k)x(n-k)
  h(0)x(n)h(1)x(n-1) h(2)x(n-2) ----- up to 8
• To form the o/p at any instant of time nn0 the
  system must remember all the previous input
  samples.

25

• To store all previous values of I/P, it will
  require infinite memory space, which is
  practically impossible.
• Is it possible to realize an IIR system?
  (fortunately , Yes.)

26
Recursive systems

• The classification recursive and non recursive
  systems is based on method of implementation.
• An FIR system can be implemented by either method
  but an IIR system can only be implemented by
  recursive method.
• In recursive systems the output at any instant of
  time nn0 depends upon one or more previous
  values of O/Ps. e.g.
• y(n) ay(n-1) bx(n)

27
Iterative method

• y(0) ay(-1)x(0)
• y(1) ay(0)x(1) a2y(-1)ax(0)x(1)
• y(2) a3y(-1)a2x(0)ax(1)x(2)
• On generalizing,
• y(n) an1y(-1) ?(k0 to n) akx(n-k)
• y(n) yzi(n) yzs(n)
• The O/P of the system can also be represented as
  a summation of two terms, one depends upon
  applied input yzs(n) and the other upon initial
  condition yzi(n).

28
Direct method

• Indirect method uses z-transform.
• Obtain characteristic eq by substituting y(n)
  (a)n and x(n)0 in the given equation.
• Find all roots a1, a2, a3, ----- except a00
• The homogeneous sol will be
• yh(n) c1a1n c2a2n, c3a3n -----------
  ?(k1toN)ck(ak)n

29

• Assume particular solution as per given input. 0
  if impulse, ku(n) if u(n), kMn if AMn etc.
• The precaution the assumed particular solution
  should not exists in homogeneous solution.
• Substitute it, along with given input, in given
  equation and solve for k by choosing a value of
  n such that no term should vanish.

30

• y(n) yh(n)yp(n) find values of y(0), y(1),
  y(2), ..
• Compute y(0), y(1), . from given eq and equate
  these values to solve for unknown c1, c2, c3, .
• Substitute the values of c1, c2, c3, . and
  given initial conditions in y(n) yh(n)yp(n) to
  get the total solution.

31
Zero input response

• If input is then particular solution will also be
  zero.
• Now evaluate c1, c2, c3, with y(n) yh(n) and
  given equation.
• Substitute these values in y(n) yh(n) to get
  zero input response.

32
Zero state response

• The total solution is y(n) yh(n)yp(n)
• Substitute all initial conditions to be zero and
  find c1, c2, c3,
• Substitute these values in above equation to get
  zero state response

33
When O/P depends upon current as well as past
values of I/P

• first the o/p should be calculated only for x(n)
  by setting all previous values zero.
• Now using the linearity property other o/ps
  should be evaluated i.e. y(n-1) replace each n
  into n-1 in the y(n) etc.
• The overall o/p will be the sum of these o/ps.
• The final expression should always be in terms of
  unit impulse or unit step function.

34
LCCDE

• y(n) - ?(k1 to N) ak y(n-k) ?(j0 to M) bj
  x(n-j)
• y(n) yh(n)yp(n) and y(n) yzi(n)yzs(n)
• y(n) an1y(-1) ?(k0 to n) akx(n-k) for 1st
  order system
• Just observe that at any instant nn0 the O/P
  depends upon all previous values of I/P. Thus a
  recursive system described by LCCDE is an IIR
  system. However the converse is not true.
• If all ak are zero, in above LCCDE, it will
  describe a non-recursive y(n) ?(j1 to M) bj
  x(n-j) thus FIR system.

35

• y(n) an1y(-1) ?(k0 to n) akx(n-k) for 1st
  order system
• The steady state response yss(n) Lim(n?8) y(n)
• yss(n) 0 a is less than 1 1/(1-a) sum of GP
  assuming unit step input
• yp(n)
• The transient response the part of y(n) which
  has been vanished at steady state
• Other methods for evaluating steady state and
  transient response are also available.

36
Impulse response of LCCDE systems

• For 1st order systems yzs(n) ?(k0 to n)
  akx(n-k)
• Compare it with y(n)?(k -8 to 8) h(k) x(n-k)
• h(n) anu(n) with following conditions
• h(n) must be causal sequence so that lower limit
  k0 and k-8 will produce same result. As if klt0
  then h(-ve) 0.

37

• x(n) should also be causal so that the upper
  limit kn and k8 should produce same result. As
  if kgtn then x(n-k) x(-ve)0
• The recursive system described by LCCDE is a
  causal IIR system with causal input.

38

• If i/p is an impulse then particular solution is
  zero thus yzs(n) will only depend upon
  homogeneous solution of y(n) e.g. for 1st order
  system
• yzs(n) ?(k0 to n) ak?(n-k) ?(k0 to 8)
  akx(n-k) anu(n) h(n)
• If input is an impulse then
• yzs(n) yh(n) ?(k1toN) ck(ak)n h(n)

39
LCCDE system stability

• h(n) ?(k1toN)ck(ak)n
• For stability ?(n0 to 8) h(n) lt 8
• ?(n0 to 8) ?(k1toN)ck(ak)n lt 8
• ?(n0 to 8) ?(k1toN) ck (ak)n lt 8 using
  inequality
• ?(k1toN)ck ?(n0 to 8) (ak)n lt 8 reversing
  order of summation

40

• All cks are finite thus above condition is
  satisfied if
• ?(n0 to 8) (ak)n lt 8 for all k.
• This summation will converge if all aks are less
  than unity (infinite GP sum)
• Necessary and sufficient condition for stability
  of LCCDE systems is,
• all roots of characteristic equation should be
  less than unity

41
Realization of LCCDE systems

• Let we have to realize following system
• y(n) -a1y(n-1)b0x(n)b1x(n-1)
• There are two methods
• Direct form I
• Direct form II (Canonic form)
• Direct form II can be obtained using
  associative property i.e. the order of two sub
  systems of a system can be interchanged

42
Correlation of Discrete time signals

• rxy(l) ?(n -8 to 8) x(n) y(n-l) ?(n -8 to 8)
  x(nl) y(n)
• for l0, 1, 2,
• It is a measure of similarity between two
  signals. (prove it?)
• ryx(l) rxy(-l)
• one is folded version of other hence they provide
  the same information about the similarity of two
  signals.
• Commutative property does not hold (except for
  autocorrelation).
• Application in RADAR, Digital Comm. etc.

43
Correlation properties

• rxx(0) Ex
• rxy(l) x(l) y(-l) (convolution operation)
• rxy(l) lt sqrt (Ex.Ey) (prove it?)
• Coefficient of correlation
• ?xy(l) rxy(l) / sqrt (Ex.Ey)
• thus -1lt Coff. of Corr.lt 1

44
Correlation of periodic sequences

• If period of x(n) and y(n) is N then
• rxy(l) 1/N?(n 0 to N-1) x(n)y(n-l)
• Avg over infinite interval is identical to the
  avg over a single period.
• rxy(l) will also be a periodic sequence of same
  period.
• This property is useful in finding period of
  sequences which have been corrupted by noise.

45
I/P and O/P Correlation Sequences

• y(n) h(n)x(n) (convolution operation)
• ryx(l) y(l)x(-l) h(l)x(l)x(-l) h(l)rxx(l)
• rxy(l) ryx(-l) h(-l)rxx(-l) h(-l)rxx(l)
• ryy(l) y(l)y(-l)
• h(l)x(l)h(-l)x(-l)
  h(l)h(-l)x(l)x(-l)
• rhh(l)rxx(l)
• rhh(l) exists if the system is stable.

46
Key features of Z-Transform

• X(z) Zx(n) ?(n -8 to 8) x(n)z-n zrejw
• The sum will converge (has finite value) with
  some condition and the region of complex z-plane
  where this condition is met is called region of
  convergence (ROC) for the sequence x(n).
• ROC must always be specified along with the X(z).
• Find z-transform and ROC for
• (i) x(n) anu(n) (ii) x(n) -anu(-n-1)

47
ROC

• Consider a sequence with three real poles at za,
  zb and zc (i.e. it has three sub sequences).
• The ROC for the seq is the overlapped ROCs of all
  subsequences.
• Fi.(b), seq is causal (If all sub sequences are
  causal).
• Fig.(c) seq is anticausal (If all sub sequences
  are anti-causal.)
• Fig.(d) Seq is two sided (a is causal and b c
  are anti-causal.)
• Fig.(e) Seq is two sided (a and b are causal but
  c is anti causal)
• Under any other scenario there is no common ROC
  of all three subsequences

48
Causality and stability

• System function is defined as
• H(z) Zh(n) ?(n -8 to 8) h(n)z-n
• ROC of H(z) must be outside of a circle then
  causal and vice versa.
• For stability unit circle must be included in the
  ROC of H(z). (prove it?)

49

• We have derived the condition for stability as
• Sh ?(k -8 to 8) h(k)lt8
• We also know that- H(z) ?(n -8 to 8) h(n)z-n
• H(z) ?(n -8 to 8) h(n)z-n lt ?(n -8 to 8)
  h(n)z-n
• At unit circle i.e. z 1
• H(z1) lt Sh
• For stability Shlt8 i.e. H(z1) lt8
• According to the definition of ROC above
  condition is satisfied if unit circle falls in it
• This is necessary and sufficient condition for
  stability and its converse is also true.
• For a causal linear time invariant system the ROC
  is outside of its highest pole thus condition for
  BIBO stability reduces to the condition that all
  poles must lie inside unit circle.

50
Z transform properties

• Zanu(n) 1/(1-az-1) ROC zgta
• Z-anu(-n-1) 1/(1-az-1) ROC zlta
• Zx(n-k) z-kX(z) e.g. Zanu(n-1)
  z-1/(1-az-1)
• Zanx(n) X(a-1z)
• Zx(-n) X(z-1) e.g. Zu(-n) 1/(1- z)
• Znx(n) -z d/dzX(z) e.g Zn.anu(n)
  z-1/(1-az-1)2
• Zx1(n)x2(n) X1(z)X2(z)
• Zrx1x2(l)Zx1(l)x2(-l) X1(z)X2(z-1)

51
(No Transcript)
52
Inverse Z-Transform

• Power series method
• (long division suited for limited duration
  sequences)
• Partial fraction method
• Cauchys integral theorem

53
Natural and forced response

• Let H(z) B(z)/A(z) has N simple distinct poles
  and X(z) N(z)/Q(z) has L simple distinct poles.
• Using convolution theorem and assuming initial
  conditions zero (two sided z-transform can be
  used)
• Y(z) H(z)X(z) B(z)N(z) / A(z)Q(z)

54

• If X(z) 0 then Y(z) 0 i.e. the above
  expression does not include the zero input
  response
• On partial fraction expansion
• Y(z) ?(k1toN)Ak/(1-pkz-1)
• ?(k1toL) Qk/(1-qkz-1)
• Each coefficient Ak and Qk depends upon both H(z)
  and X(z).

55

• Taking inverse z-transform
• y(n) ?(k1toN)Ak(pk)nu(n)
• ?(k1toL)Qk(qk)nu(n)
• First sum depends upon system poles thus called
  Natural response ynr(n).
• If all pk fall inside the unit circle and n?8 the
  ynr(n) reduces to zero thus it is also called
  transient response ytr(n). (influence of I/P is
  reflected in Ak)
• Second sum depends upon poles of input signal
  thus called forced response yfr(n)

56

• If all qk fall inside unit circle and n?8 then
  the yfr(n) will also reduce to zero. (it is just
  the response to a limited duration i/p)
• If I/P is a sinusoid then the poles will be
  complex conjugate and fall on the unit circle.
  Then O/P will also be a sinusoid of different
  amplitude and phase.
• In this case it is also called the steady state
  response of the system (it persists as long as
  input is ON).

57
Pole zero cancellation

• If a pole/zero of X(z) falls at the same location
  where a zero/pole of H(z) exists then the effect
  of pole will be eliminated in Y(z).
• Sometimes pole-zero cancellation can occur in the
  system function itself (if any coefficient of
  partial fraction expansion is zero). It reduces
  the effective order of the system.
• Proper selection of zeros of X(z) H(z) can
  suppress the effect of corresponding pole of H(z)
  X(z).