Introduction to Markov Chains

1
Introduction to Markov Chains

• Referred Book Gregory F Lawler
• Introduction to Stochastic Processes
• Chapter 1.

2
Quick Recap(1) Random Variable

• A random variable x is a function from sample
  space O to real/complex space Rn.
• x is measurable wrt. s- algebra of O.
• x may be discrete or continuous
• Depends on O
• Random Variable representing position of a
  particle can take any value in real space R3 . (O
  position x x ? R3).
• Random Variable representing population can take
  only positive integral values. (O size of
  population n n ? N)
• Not all elements in O are in its s- algebra.

3
Quick recap(2) Stochastic Process and Filtration

• A stochastic process x(t,?) is a collection of
  random variable wrt. Some parameter (most often
  time).
• A stochastic process X(t,?) is the state of the
  outcome ? of a probabilistic experiment at time
  t.
• At each time t, we filter s-algebra F to get
  s-algebra Ft.
• Ft represents all the events that can take place
  up-to time t total history of the process up-to
  t.
• This sequence Ft is called filtration of F

4
Stochastic Process - Illustration
Y2 X(t2, ?)
Y1 Y2 are 2 different random variables.
Y1 X(t1, ?)
X(t,?1)
X(t,?2)
X(t,?3)
Stochastic Process X(t, ?) is a collection of
these Yis
Time
5
Stochastic Process Another Interpretation

• x(t,?) can also be viewed as a function of t and
  ?.
• The parameter t can be continuous or discrete.
• We can monitor a process continuously or look
  at it after some time interval.
• O can be discrete or continuous
• Depends on the sample space O (position or
  population).
• Hence, Stochastic processes can be divided in 4
  different classes

6
Stochastic Process Classification
t
Continuous
Discrete
?
X(t,?) can take only a countable/finite number of
values at each discrete moment. e.g Discrete
Markov Chain
X(t,?) can take only a countable/finite number of
values at any point. e.g. Continuous time Markov
Chain
Discrete
X(t,?) can take any value in a valid interval at
any point. Realm of Stochastic Differential
Equations and Itos theorem e.g. Brownian Motion
Not Discussed.
Continuous
7
Discrete Stochastic Processes

• In this case time and sample space are discrete.
• Time is indexed and will be designated by integer
  n
• The process x(t,?) is then denoted as xn.
• xn can take only a discrete number of values.
• The probability of xn taking a valid value k is
  determined by its entire history.
• P(xn k) ?.. .? P(xn k,xn-1 kn-1,,x1
  k1) or
• P(xn k) ?.. .? P(xn kxn-1,..,x1) P(xn-1
  kxn-2,..,x1)..P(x1)
• i.e. state of the process at any index n depends
  on the previous values

8
Discrete Time Stochastic Process Illustration
x4 5
x1 5
x9 5
x7 4
x8 3
x2 3
x6 2
x5 2
x3 1
9
Markov property for Discrete Stochastic Processes

• The likelihood of transition to any state at the
  next time index depends only on the current
  state.
• P(xn kxn-1 kn-1,,x1 k1) P(xn kxn-1
  kn-1) or
• P(xn k) ?m P(xn kxn-1 m) or
• P(xn k) ?.. .? P(xn kxn-1) P(xn-1
  kxn-2)..P(x1)
• Any Discrete Stochastic Process Satisfying the
  Markov property is a Discrete Markov Chain

10
Discrete Time Markov Chain(1)

• A discrete time Markov Chain M is a sequence xn
  of random variables.
• Each element of the sequence is a discrete random
  variable.
• The probability of xn being in a particular
  state depends only the value of xn-1.
• A Markov Chain is defined by 2 quantities-
• Initial State probability vector v(0)
• (P(x0 k) vk(0))
• The Transition matrix P(n) at time n.
• P(xn1 kxn i) P(n)ik .
• v(n 1) v(n)P(n).

11
Discrete Time Markov Chain(2) Behavior with Time

• A markov chain is called Time-Homogenous when
  P(n) P is a constant wrt n.
• The likelihood of transitions dose not change
  with time.
• Then v(n 1) v(n)P v(n-1)P2 v(0)P(n1)
• Long Range Behavior
• What happens to the state probabilities when a
  markov chain has continued for a long time?
• A large class of markov chains tend to approach a
  steady state of state probabilities.

12
Discrete Time Markov Chain(3) Behavior with Time

• Steady State Probabilities
• After a long time, generally a markov chain would
  settle to state probabilities that do not change
  with time.
• i.e. v(n) ? p as n ?8.
• p is a left eigenvector of P with eigenvalue 1
• pP p
• We know that such a vector exists as there is
  always a right eigenvector of P with eigenvalue 1
• P.1 1

13
Discrete Time Markov Chain(4) Behavior with Time

• Question now is Which Markov Chains approach
  this steady state, and if it is unique?
• We look at it later after presenting a few more
  concepts

14
Discrete Time Markov Chain(5) Classification of
States

• A discrete markov chain can be considered as a
  random path in a graph (nodes represent the
  states).
• Then, 2 states (nodes) i and k can communicate
  with each (i ? k) other if for some m,n gt 0
• Pmik gt 0 and Pnki gt 0
• There is a chance that the chain will reach state
  i from k in some finite steps and vice-versa
  (existence of a path in probability)
• Communication is an equivalence relation-
• i ? k and k ? h i ? h
• i ? k k ? i

15
Discrete Time Markov Chain(6) Communicating
Classes Illustration
4 is not communicable with any other state
3
5
1
4
7
6
2
3 ? 5
1 ? 2
2 ? 6
6 ? 7
6 ? 5
16
Discrete Time Markov Chain(7) Classification of
States

• Not all states can communicate with each other.
• A set of states that can communicate with each
  other belongs to one communication class. (a
  connected sub-graph in probability)
• If all the states can communicate with each
  other, the markov chain is irreducible (A
  connected graph)
• Or, P has exactly one left eigenvector with
  eigenvalue 1 and all other eigenvalues have
  magnitude less than 1.

17
Discrete Time Markov Chain(8) Classification of
States

• A set of states that can communicate with each
  other belong to a particular communication
  class-
• As time progresses, if the markov chain leaves
  the class with probability 1, then the class is
  transient
• Other Classes are called recurrent classes.
• Markov chain is trapped in the recurrent class
• In such a case, a markov chain is reduced to
  smaller chains

18
Discrete Time Markov Chain(9) Reducible Markov
Chain Transition Matrix
Recurrent Class
R1
Transition From recurrent to transient class
0
R2
R3
0
R4
0
Rr
Transient class
Transition From transient to recurrent class
Q
S
19
Discrete Time Markov Chain(10)

• At infinite time, a recurrent state will be
  visited infinitely often
• At infinite time, a transient state is visited
  only a finite number of times.
• The expected number of visits to a transient
  state j is given by ? Mjk where M (I - Q)-1.
• The probability matrix A of ending up in a
  recurrent state r from any transient state is
  obtained by A M.S (I - Q)-1S

20
Discrete Time Markov Chain(11) Periodicity

• A state i can communicate with itself (existence
  of a cycle)
• Hence, it will return to the same state i in some
  finite steps with a non-zero probability
• The period of a state i is defined as d(i)-
• d(i) gcdn Pnii gt 0
• If d(i) 1, markov chain is aperiodic
• Can be observed from visual inspection

21
Discrete Time Markov Chain(11) Steady State

• If a time-homogeneous markov chain is irreducible
  and aperiodic, it will have a unique steady state
  probability vector p.
• aperiodicity guarantees that only one left
  eigenvector has eigenvalue 1.
• Multiple left eigenvectors with eigenvalue 1
  imply multiple steady state distributions. (each
  one can occur)
• Irreducibility guarantees that no other
  eigenvector has an eigenvalue with magnitude 1.