# EE255CPS226 Discrete Time Markov Chain DTMC - PowerPoint PPT Presentation

PPT – EE255CPS226 Discrete Time Markov Chain DTMC PowerPoint presentation | free to view - id: 2082b6-ZDc1Z

The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
Title:

## EE255CPS226 Discrete Time Markov Chain DTMC

Description:

### pjk(m,n) P(Xn = k | Xm = j ) ... pjk(m,n): probability transition function of a DTMC. Homogeneous DTMC: pjk(m,n) = pjk(m-n) i.e., transition probabilities exhibit ... – PowerPoint PPT presentation

Number of Views:337
Avg rating:3.0/5.0
Slides: 29
Provided by: bharat9
Category:
Tags:
Transcript and Presenter's Notes

Title: EE255CPS226 Discrete Time Markov Chain DTMC

1
EE255/CPS226Discrete Time Markov Chain (DTMC)
• Dept. of Electrical Computer engineering
• Duke University
• Email bbm_at_ee.duke.edu, kst_at_ee.duke.edu

2
Discrete Time Markov Chain
• Markov process dynamic evolution is such that
future state depends only on the present (past is
irrelevant).
• Markov Chain ? Discrete state (or sample) space.
• DTMC time (index) is also discrete i.e. system
is observed only at discrete intervals of time.
• X0, X1, .., Xn, .. observed state (of a
particular ensemble member (of the sample space)
at discrete times, t0, t1,..,tn, ..
• X0, X1, .., Xn , .. describes the states of a
DTMC
• Xn j ? system state at time step n is j. Then
for a DTMC,
• P(Xn in X0 i0, X1 i1, , Xn-1 in-1)
P(Xn in Xn-1 in-1)
• pj(n) ? P(Xn j) (pmf), or,
• pjk(m,n) ? P(Xn k Xm j ),

3
Transition Probability
• pjk(m,n) probability transition function of a
DTMC.
• Homogeneous DTMC pjk(m,n) pjk(m-n) i.e.,
transition probabilities exhibit stationary
property. For such a DTMC,
• 1-step transition prob, pjk pjk(1) P(Xn k
Xn-1 j) ,
• Assuming 0-step transition prob as
• Joint pmf is given by, P(X0 i0, X1 i1,
, Xn in)
• P(X0 i0, X1 i1, , Xn-1 in-1).
P(Xn in X0 i0, X1 i1, , Xn-1 in-1)
• P(X0 i0, X1 i1, , Xn-1 in-1).
P(Xn in Xn-1 in-1) (due to Markov prop)
• P(X0 i0, X1 i1, , Xn-1
in-1).pin-1, in
• pi0(0)pi0, i1 (1) pin-1, in (1) pi0(0)pi0,
i1 pin-1, in

4
Transition Probability Matrix
• The initial prob. is, pi0(0) P(X0 i0 ). In
general,
• p0(0) P(X0 0 ), , pk(0) P(X0 k ) etc,
or,
• p(0) p0(0), p1(0), ,pk(0), .
(initial prob. vector)
• This allows us to define transition prob. matrix
as,
• Sum of ith row elements pi,0(0) pi,1(0)
?
• Any such sq. matrix with non-negative entries
whose row sum 1 is called a stochastic matrix.

5
State Transition Diagram
• pij describes random state value evolution
from i to j
• Node with labels i, j etc. and an arc labeled
pij
• Concept of ri reward (cost or penalty) for each
state I allows evaluation of various interesting
performance measures.
• Example 2-state DTMC for a cascade binary comm.
channel. Signal values 0 or 1 form the state
values.

i
j
pij
6
Total Probability
• Finding total pmf

7
n-Step Transition Probability
• For a DTMC, find
• Events state reaches k (from i) reaches j
(from k) are independent due to the Markov
property (i.e. no history)
• Invoking total probability theorem
• Let P(n) n-step prob. transition matrix (i,j)
entry is pij(n). Making m1, nn-1 in the above
equation,

8
Marginal pmf
• j, in general can assume countable values,
0,1,2, . Defining,
• pj(n) for j0,1,2,..,j, can be written in the
vector form as,
• Or,
• Pn can be easily computed if n is finite.
However, if n is countably infinite, it may not
be possible to compute Pn (and p(n) ).

9
Marginal pmf Example
• For a 2-state DTMC described by its 1-step
transition prob. matrix,
• the n-step transition prob. Matrix is given by,
• Proof follows easily by using induction, that is,
assuming that the above is true for Pn-1. Then,
Pn P. Pn-1

10
Computing Marginal pmf
• Previous example of a cascade digital comm
channels each stage described by a 2-state DTMC,
We want to find p(n) (a0.25 b0.5),
• The 11 element for n2 and n3 are,
• Assuming initial pmf as, p(0) p0(0) p1(0)
1/3 2/3 gives,
• What happens to Pn as n becomes very large (?
infinity)?

11
DTMC State Classification
• From the previous example, as n becomes infinity,
pij(n) becomes independent of n and i !
Specifically,
• Not all Markov chains may exhibit this behavior.
• State classification may be based on the
distinction that asymtotically
• some states may be visited infinitely many
times. Whereas, some other states may be visited
only a small number of times
• Transient state iff there is non-zero
this state.
• Define Xji to be the of visits to state i,
starting from state j, then,
• For a transient state (i), visit count needs to
finite, which requires pji(n) ? 0 as n ?
infinity. Eventually, the system will always
leave state i.

12
DTMC State Classification (contd.)
• State i is a said to be recurrent iff, starting
from state i, the process eventually returns to
the state i with probability 1.
• For a recurrent state, time-to-return is a
relevant measure. Define fij(n) as the cond.
prob. that the first visit to j from i occurs in
exactly n steps.
• If j i, then fii(n) denotes the prob. of
returning to i in exactly n steps.
• Known result
• Let,
• Mean recurrence time for state i is

13
Recurrent state
• Let i be recurrent and pii(n) gt 0, for some n gt
0.
• For state i, define period di as GCD of all such
ve ns that result in pii(n) gt 0
• If di1, ? aperiodic and if digt1, then periodic.
• Absorbing state state i absorbing iff pii1.
• Communicating states i and j are said to be
communicating if there exits directed paths from
i and j and from j and i.
• Closed set of states A commutating set of states
C forms a closed set, if no state outside of C
can be reached from any state in C.

14
Irreducible Markov Chains
• Markov chain states partitioned into two distinct
subsets c1, c2, .., ck-1, ck , such that
• ci, i1,2,..k-1 are closed set of recurrent
nun-null.
• ck transient states.
• If ci contain only one state, then cis form a
set absorbing states
• If k2 and ck empty, then c1 forms an irreducible
Markov chain
• Irreducible Markov chain is one in which every
state can be reached from every other state in a
finite no. of steps, i.e., for all i,j e I, for
some integer n gt 0, pij(n) gt 0. Examples
• Cascade of digital comm. channels is

0
1
15
Irreducible Markov Chains (contd.)
• If one state is recurrent aperiodic, then so are
all the other states. Same result if periodic or
transient.
• For a finite aperiodic irreducible Markov chain,
pij(n) becomes independent of i and n as n goes
to infinity.
• All rows of Pn become identical

16
Irreducible Markov Chains (contd.)
• Law of total probability gives,
• Therefore, 1st eq. can be rewritten as,
• In the matrix form,
• v is a probability vector, therefore,
• Self reading exercise (theorems on pp. 351)
• For an aperiodic, irreducible, finite state DTMC,

17
Irreducible Markov Chain Example
• Typical computer program continuous cycle of
compute I/O
• The resulting DTMC is irreducible with period 1.
Therefore,

18
Sojourn Time
• If Xn i, then Xn1 j should depend only on
the current state i, and not on the time spent in
state i.
• Let Ti be the time spent in state i, before
moving to state j
• DTMC will remain in state i in the next step with
prob. pii and,
• Next step (n1), toss a coin, H? Xn1 i,
T?Xn1 i
• At each step, we perform a Bernoulli trial. Then,

19
Markov Modulated Bernoulli Process
• Generalization of a Bernoulli process the
Bernoulli process parameter is controlled by a
DTMC.
• Simplest case is Binary state (on-off) modulation
• On? Bernoulli param c1 Off ? c2 (or 0)
• Controlling process is an irreducible DTMC, and,
• Reward assignment, r0 c1 and r1c2. This gives,

20
Examples of Irreducible DTMCs
• Example 7.14 non-homogeneous DTMC for s/w
reliability
• Slotted ALOHA wireless multi-access protocol
• 2X more efficient than pure Aloha.
• Automatically adapts to changes in station
population
• Throughput maximum of 36.8 theoretical limit.
• Requires queuing (buffering) for re-transmission
• Synchronization.

21
Slotted ALOHA DTMC
• New and backlogged requests
• Successful channel access iff
• Exactly one new req. and no backlogged req.
• Exactly one backlogged req. and no new req.
• DTMC state of backlogged requests.

backlogged
new
n

x

m-n

x
x
Channel
S
22
Slloted Aloha contd.
• In a particular state n, successful contention
occurs with prob. rn
• rn may be assigned as a reward for state n.

23
Discrete-time Birth-Death Processes
• Special type of DTMC in which P has a
tri-diagonal form,

24
DTMC solution steps
• Solving for v vP, gives the steady state
probabilities.

25
Finite DTMCs with Absorbing States
• Example Program having a set of interacting
modules. Absorbing state failure state (? ps5
unreliability)

26
Finite DTMCs, Absorbing States (contd.)
• M contains useful information.
• Xij rv denoting random number to visits to j
starting from i
• EXij mij (for i, j 1,2,, n-1) . Need
to prove this statement.
• There are three distinct situations that can be
enumerated
• Let rv Y denote the state at step 2
(initial state i)
• EXij y n dij
• EXij y k EXkj dij EXkj dij

dij , occurs with prob. pij Xkj dij ,
occurs with prob. Pik
k1,2,..n (dij term accounts for ij case)
Xij
si
sk
sj
sn
i
27
Finite DTMCs, Absorbing States (contd.)
• Since, P(Yk) pik , k1,2,..n, total
expectation rule gives,
• Over all (i,j) values, we need to work with the
matrix,
• Therefore, fundamental matrix M elements give
the expected of visit to state j (from i)
before absorption.
• If the process starts in state 1, then m1j
gives the average of visits to state j (from
the start state) before absorption.

28
Performance Analysis, Absorbing States
• By assigning rewards values to different state, a
variety of performance measures may be computed.
• Average time to execute a program
• s1 is the start state rjk (fractional)
execution time/visit for sj
• Vj m1j is the average times statement block
sjis executed
• We need to calculate total expected execution
time, I.e. until the process gets absorbed into
stop state (s5 )
• Software reliability jth reward Rj
Reliability of sj .Then,