EE255CPS226 Discrete Time Markov Chain DTMC - PowerPoint PPT Presentation


PPT – EE255CPS226 Discrete Time Markov Chain DTMC PowerPoint presentation | free to view - id: 2082b6-ZDc1Z


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation

EE255CPS226 Discrete Time Markov Chain DTMC


pjk(m,n) P(Xn = k | Xm = j ) ... pjk(m,n): probability transition function of a DTMC. Homogeneous DTMC: pjk(m,n) = pjk(m-n) i.e., transition probabilities exhibit ... – PowerPoint PPT presentation

Number of Views:337
Avg rating:3.0/5.0
Slides: 29
Provided by: bharat9


Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: EE255CPS226 Discrete Time Markov Chain DTMC

EE255/CPS226Discrete Time Markov Chain (DTMC)
  • Dept. of Electrical Computer engineering
  • Duke University
  • Email,

Discrete Time Markov Chain
  • Markov process dynamic evolution is such that
    future state depends only on the present (past is
  • Markov Chain ? Discrete state (or sample) space.
  • DTMC time (index) is also discrete i.e. system
    is observed only at discrete intervals of time.
  • X0, X1, .., Xn, .. observed state (of a
    particular ensemble member (of the sample space)
    at discrete times, t0, t1,..,tn, ..
  • X0, X1, .., Xn , .. describes the states of a
  • Xn j ? system state at time step n is j. Then
    for a DTMC,
  • P(Xn in X0 i0, X1 i1, , Xn-1 in-1)
    P(Xn in Xn-1 in-1)
  • pj(n) ? P(Xn j) (pmf), or,
  • pjk(m,n) ? P(Xn k Xm j ),

Transition Probability
  • pjk(m,n) probability transition function of a
  • Homogeneous DTMC pjk(m,n) pjk(m-n) i.e.,
    transition probabilities exhibit stationary
    property. For such a DTMC,
  • 1-step transition prob, pjk pjk(1) P(Xn k
    Xn-1 j) ,
  • Assuming 0-step transition prob as
  • Joint pmf is given by, P(X0 i0, X1 i1,
    , Xn in)
  • P(X0 i0, X1 i1, , Xn-1 in-1).
    P(Xn in X0 i0, X1 i1, , Xn-1 in-1)
  • P(X0 i0, X1 i1, , Xn-1 in-1).
    P(Xn in Xn-1 in-1) (due to Markov prop)
  • P(X0 i0, X1 i1, , Xn-1
    in-1).pin-1, in
  • pi0(0)pi0, i1 (1) pin-1, in (1) pi0(0)pi0,
    i1 pin-1, in

Transition Probability Matrix
  • The initial prob. is, pi0(0) P(X0 i0 ). In
  • p0(0) P(X0 0 ), , pk(0) P(X0 k ) etc,
  • p(0) p0(0), p1(0), ,pk(0), .
    (initial prob. vector)
  • This allows us to define transition prob. matrix
  • Sum of ith row elements pi,0(0) pi,1(0)
  • Any such sq. matrix with non-negative entries
    whose row sum 1 is called a stochastic matrix.

State Transition Diagram
  • pij describes random state value evolution
    from i to j
  • Node with labels i, j etc. and an arc labeled
  • Concept of ri reward (cost or penalty) for each
    state I allows evaluation of various interesting
    performance measures.
  • Example 2-state DTMC for a cascade binary comm.
    channel. Signal values 0 or 1 form the state

Total Probability
  • Finding total pmf

n-Step Transition Probability
  • For a DTMC, find
  • Events state reaches k (from i) reaches j
    (from k) are independent due to the Markov
    property (i.e. no history)
  • Invoking total probability theorem
  • Let P(n) n-step prob. transition matrix (i,j)
    entry is pij(n). Making m1, nn-1 in the above

Marginal pmf
  • j, in general can assume countable values,
    0,1,2, . Defining,
  • pj(n) for j0,1,2,..,j, can be written in the
    vector form as,
  • Or,
  • Pn can be easily computed if n is finite.
    However, if n is countably infinite, it may not
    be possible to compute Pn (and p(n) ).

Marginal pmf Example
  • For a 2-state DTMC described by its 1-step
    transition prob. matrix,
  • the n-step transition prob. Matrix is given by,
  • Proof follows easily by using induction, that is,
    assuming that the above is true for Pn-1. Then,
    Pn P. Pn-1

Computing Marginal pmf
  • Previous example of a cascade digital comm
    channels each stage described by a 2-state DTMC,
    We want to find p(n) (a0.25 b0.5),
  • The 11 element for n2 and n3 are,
  • Assuming initial pmf as, p(0) p0(0) p1(0)
    1/3 2/3 gives,
  • What happens to Pn as n becomes very large (?

DTMC State Classification
  • From the previous example, as n becomes infinity,
    pij(n) becomes independent of n and i !
  • Not all Markov chains may exhibit this behavior.
  • State classification may be based on the
    distinction that asymtotically
  • some states may be visited infinitely many
    times. Whereas, some other states may be visited
    only a small number of times
  • Transient state iff there is non-zero
    probability that the system will NOT return to
    this state.
  • Define Xji to be the of visits to state i,
    starting from state j, then,
  • For a transient state (i), visit count needs to
    finite, which requires pji(n) ? 0 as n ?
    infinity. Eventually, the system will always
    leave state i.

DTMC State Classification (contd.)
  • State i is a said to be recurrent iff, starting
    from state i, the process eventually returns to
    the state i with probability 1.
  • For a recurrent state, time-to-return is a
    relevant measure. Define fij(n) as the cond.
    prob. that the first visit to j from i occurs in
    exactly n steps.
  • If j i, then fii(n) denotes the prob. of
    returning to i in exactly n steps.
  • Known result
  • Let,
  • Mean recurrence time for state i is

Recurrent state
  • Let i be recurrent and pii(n) gt 0, for some n gt
  • For state i, define period di as GCD of all such
    ve ns that result in pii(n) gt 0
  • If di1, ? aperiodic and if digt1, then periodic.
  • Absorbing state state i absorbing iff pii1.
  • Communicating states i and j are said to be
    communicating if there exits directed paths from
    i and j and from j and i.
  • Closed set of states A commutating set of states
    C forms a closed set, if no state outside of C
    can be reached from any state in C.

Irreducible Markov Chains
  • Markov chain states partitioned into two distinct
    subsets c1, c2, .., ck-1, ck , such that
  • ci, i1,2,..k-1 are closed set of recurrent
  • ck transient states.
  • If ci contain only one state, then cis form a
    set absorbing states
  • If k2 and ck empty, then c1 forms an irreducible
    Markov chain
  • Irreducible Markov chain is one in which every
    state can be reached from every other state in a
    finite no. of steps, i.e., for all i,j e I, for
    some integer n gt 0, pij(n) gt 0. Examples
  • Cascade of digital comm. channels is

Irreducible Markov Chains (contd.)
  • If one state is recurrent aperiodic, then so are
    all the other states. Same result if periodic or
  • For a finite aperiodic irreducible Markov chain,
    pij(n) becomes independent of i and n as n goes
    to infinity.
  • All rows of Pn become identical

Irreducible Markov Chains (contd.)
  • Law of total probability gives,
  • Therefore, 1st eq. can be rewritten as,
  • In the matrix form,
  • v is a probability vector, therefore,
  • Self reading exercise (theorems on pp. 351)
  • For an aperiodic, irreducible, finite state DTMC,

Irreducible Markov Chain Example
  • Typical computer program continuous cycle of
    compute I/O
  • The resulting DTMC is irreducible with period 1.

Sojourn Time
  • If Xn i, then Xn1 j should depend only on
    the current state i, and not on the time spent in
    state i.
  • Let Ti be the time spent in state i, before
    moving to state j
  • DTMC will remain in state i in the next step with
    prob. pii and,
  • Next step (n1), toss a coin, H? Xn1 i,
    T?Xn1 i
  • At each step, we perform a Bernoulli trial. Then,

Markov Modulated Bernoulli Process
  • Generalization of a Bernoulli process the
    Bernoulli process parameter is controlled by a
  • Simplest case is Binary state (on-off) modulation
  • On? Bernoulli param c1 Off ? c2 (or 0)
  • Controlling process is an irreducible DTMC, and,
  • Reward assignment, r0 c1 and r1c2. This gives,

Examples of Irreducible DTMCs
  • Example 7.14 non-homogeneous DTMC for s/w
  • Slotted ALOHA wireless multi-access protocol
  • Advantages
  • 2X more efficient than pure Aloha.
  • Automatically adapts to changes in station
  • Disadvantages
  • Throughput maximum of 36.8 theoretical limit.
  • Requires queuing (buffering) for re-transmission
  • Synchronization.

  • New and backlogged requests
  • Successful channel access iff
  • Exactly one new req. and no backlogged req.
  • Exactly one backlogged req. and no new req.
  • DTMC state of backlogged requests.




Slloted Aloha contd.
  • In a particular state n, successful contention
    occurs with prob. rn
  • rn may be assigned as a reward for state n.

Discrete-time Birth-Death Processes
  • Special type of DTMC in which P has a
    tri-diagonal form,

DTMC solution steps
  • Solving for v vP, gives the steady state

Finite DTMCs with Absorbing States
  • Example Program having a set of interacting
    modules. Absorbing state failure state (? ps5

Finite DTMCs, Absorbing States (contd.)
  • M contains useful information.
  • Xij rv denoting random number to visits to j
    starting from i
  • EXij mij (for i, j 1,2,, n-1) . Need
    to prove this statement.
  • There are three distinct situations that can be
  • Let rv Y denote the state at step 2
    (initial state i)
  • EXij y n dij
  • EXij y k EXkj dij EXkj dij

dij , occurs with prob. pij Xkj dij ,
occurs with prob. Pik
k1,2,..n (dij term accounts for ij case)
Finite DTMCs, Absorbing States (contd.)
  • Since, P(Yk) pik , k1,2,..n, total
    expectation rule gives,
  • Over all (i,j) values, we need to work with the
  • Therefore, fundamental matrix M elements give
    the expected of visit to state j (from i)
    before absorption.
  • If the process starts in state 1, then m1j
    gives the average of visits to state j (from
    the start state) before absorption.

Performance Analysis, Absorbing States
  • By assigning rewards values to different state, a
    variety of performance measures may be computed.
  • Average time to execute a program
  • s1 is the start state rjk (fractional)
    execution time/visit for sj
  • Vj m1j is the average times statement block
    sjis executed
  • We need to calculate total expected execution
    time, I.e. until the process gets absorbed into
    stop state (s5 )
  • Software reliability jth reward Rj
    Reliability of sj .Then,