Loading...

PPT – Discrete Time Markov Chains PowerPoint presentation | free to view - id: c8e9b-YjgwO

The Adobe Flash plugin is needed to view this content

Discrete Time Markov Chains

Many real-world systems contain uncertainty and

evolve over time

Stochastic processes (and Markov chains) are

probability models for such systems.

A discrete-time stochastic process is a

sequence of random variables X0, X1, X2, . . .

typically denoted by Xn .

where index T or n takes countable discrete values

An Example Problem -- Gamblers Ruin

At time zero I have X0 2, and each day I make

a 1 bet. I win with probability p and lose with

probability 1 p. Ill quit if I ever obtain 4

or if I lose all my money.

Xt amount of money I have after the bet on day

t.

if Xt 4 then Xt1 Xt2 4

if Xt 0 then Xt1 Xt2 0.

The possible values of Xt is S 0, 1, 2, 3, 4

Components of Stochastic Processes

The state space of a stochastic process is the

set of all values that the Xts can take. (we

will be concerned with stochastic processes

with a finite of states )

Time t 0, 1, 2, . . . State m-dimensional

vector, s (s1, s2, . . . , sm ) or s (s1,

s2, . . . , sm ) or (s0, s1, . . . , sm-1)

Sequence Xt, Xt takes one of m values, so Xt ? s.

Markov Chain Definition

A stochastic process Xt is called a Markov

Chain if Pr Xt1 j X0 k0, . . . , Xt-1

kt-1, Xt i Pr Xt1 j Xt i ?

transition probabilities for every i, j, k0,

. . . , kt-1 and for every t.

The future behavior of the system depends only on

the current state i and not on any of the

previous states.

Stationary Transition Probabilities

Stationary Markov Chains

Pr Xt1 j Xt i Pr X1 j X0 i

for all t (They dont change over time) We will

only consider stationary Markov chains.

The one-step transition matrix for a Markov chain

with states S 0, 1, 2 is

where pij Pr X1 j X0 i

Property of Transition Matrix

If the state space S 0, 1, . . . , m1 then

we have ?j pij 1 ? i and pij ? 0 ? i,

j

(we must (each transition

go somewhere) has prob ?

0)

Stationary Property assumes that these values

does not change with time

Transition Matrix of the Gamblers problem

At time zero I have X0 2, and each day I make

a 1 bet. I win with probability p and lose with

probability 1 p. Ill quit if I ever obtain 4

or if I lose all my money.

Xt amount of money I have after the bet on day

t.

Transition Matrix of Gamblers Ruin Problem

State Transition Diagram

State Transition Diagram Node for each

state, Arc from node i to node j if pij gt 0

The state-transition diagram of Gamblers problem

Notice nodes 0 and 4 are trapping node

Printer Repair Problem

- Two printers are used for Russ Center.
- When both working in morning, 30 chance that one

will fail by evening and a 10 chance that both

will fail. - If only one printer is serving at the beginning

of the day, there is a 20 chance that it will

fail by the close of business. - If neither is working in the morning, the office

sends all work to a printing service. - If failed during the day, a printer can be

repaired overnight and be returned earlier the

next day

States for Computer Repair Example

Events and Probabilities for Computer Repair

Example

State-Transition Matrix and Network

State-Transition Matrix

The major properties of a Markov chain can be

described by the m ? m matrix

P (pij). For printer repair example ?

State-Transition Network Node for each

state, Arc from node i to node j if pij gt 0.

For printer repair example ?

Market Share/Brand Switching Problem

Market Share Problem

You are given the original market of three

companies. The following table gives the number

of consumers that switches from brand i to brand

j in two consecutive weeks

How to model the problem as a stochastic process ?

Empirical Transition Probabilities for Brand

Switching, pij

Transition Matrix

Assumption Revisited

- Markov Property

Pr Xt1 j X0 k0, . . . , Xt-1 kt-1, Xt

i Pr Xt1 j Xt i ?

transition probabilities for every i, j, k0,

. . . , kt-1 and for every t.

- Stationary Property

Pr Xt1 j Xt i Pr X1 j X0 i

for all t (They dont change over time) We will

only consider stationary Markov chains.

Transform a Process to a Markov Chain

Sometimes a non-Markovian stochastic process can

be transformed into a Markov chain by expanding

the state space.

Example Suppose that the chance of rain

tomorrow depends on the weather conditions for

the previous two days (yesterday and today).

Specifically, P rain tomorrow?rain last 2 days

(RR) .7 P rain tomorrow?rain today but not

yesterday (NR) .5 P rain tomorrow?rain

yesterday but not today (RN) .4 P rain

tomorrow?no rain in last 2 days (NN) .2

Does the Markovian Property Hold ??

The Weather Prediction Problem

How to model this problem as a Markovian Process

??

The state space

0 (RR) 1 (NR) 2(RN) 3(NN)

The transition matrix

0 (RR) 1 (NR) 2(RN) 3(NN)

0 ( RR) .7 0 .3 0 P 1 (NR) .5 0 .5 0 2

(RN) 0 .4 0 .6 3 (NN) 0 .2 0 .8

This is a Discrete Time Markov Process

Repair Operation Takes Two Days

One repairman, two days to fix computer. ? new

state definition required s (s1, s2) s1

number of days the first machine has been in the

shop s2 number of days the second machine has

been in the shop

For s1, assign 0 if 1st machine has not failed

1 if it is in the first day of repair 2 if it

is in the second day of repair For s2, assign 0

or 1

State Definitions for 2-Day Repair Times

State-Transition Matrix for 2-Day Repair Times

0 1 2 3 4

Choosing Balls from an Urn

An urn contains two unpainted balls at present.

We choose a ball at random and flip a coin.

If the chosen ball is unpainted and the coin

comes up heads, we paint the chosen unpainted

ball red

If the chosen ball is unpainted and the coin

comes up tails, we paint the chosen unpainted

ball blue.

If the ball has already been painted, then

(whether heads or tails has been tossed), we

change the color of the ball (from red to blue or

from blue to red)

Model this problem as a Discrete Time Markov

Chain (represent it using state diagram

transition matrix)

Multi-step (t-step) Transitions

Example IRS auditing problem

Assume that whether a tax payer is audited by IRS

or not in the n 1 is dependent only on whether

he was audit in the previous year or not.

- If he is not audited in year n, he will not be

audited with prob 0.6, and will be audited with

prob 0.4

- If he is audited in year n, he will be audited

with prob 0.5, and will not be audited with prob

0.5

How to model this problem as a stochastic process

?

An IRS Auditing Problem

State Space Two states s0 0 (no audit), s1

1 (audit)

Transition matrix

Transition Matrix P is the prob. of transition in

one step

How do we calculate the probabilities for

transitions involving more than one step?

Notice p01 0.4, is conditional probability of

audit next year given no audit this year.

p01 p (x1 1 x0 0)

OR

2-Step Transition Probabilities

p (x2 0x0 0)

p (x1 1x0 0) ? p (x2 0x1 1)

p (x1 0x0 0) ? p (x2 0x1 0)

n-Step Transition Probabilities

This idea generalizes to an arbitrary number of

steps P(3) P(2) P P2 P P3 or more

generally P(n) P(m) P(n-m)

The ij'th entry of this reduces to Pij(n) ?

Pik(m) Pkj(n-m) 1 ? m ? n?1

m

k0

Chapman - Kolmogorov Equations

The probability of going from i to k in m steps

then going from k to j in the remaining n?m

steps, summed over all possible intermediate

states k

Transition Probabilities for n Steps

Property 1Let Xn n 0, 1, . . . be a Markov

chain with state space S and state-transition

matrix P. Then for i and j ? S, and n 1, 2, .

. . PrXn j X0 i pij, where the

right-hand side represents the ijth element of

the matrix P(n) and

P(n) P ? P ? ? P

(n)

n

Conditional vs. Unconditional Probabilities

Perform the following calculations q(t)

q(0)P(t) or q(t) q(t1)P where q(0) is

initial unconditional probability. The

components of q(t) are called the transient

probabilities.

Brand Switching Example

The initial unconditional qi(0) can be obtained

by dividing total customers using brand i by

total sample size q(0) (125/500, 230/500,

145/500) (0.25, 0.46, 0.29)

To predict market shares for, say, 2 weeks into

the future, we simply apply equation with t 2

q(2) q(0)P(2)

(0.327, 0.406, 0.267) expected market share

from brands 1, 2, 3

Steady-State Solutions n Steps

What happens with t get large?

the IRS example.

Steady-State Solutions n Steps

the IRS example -- Continued.

Steady State Transition Probability

Observations as n gets large, the values in row

of the matrix becomes identical OR they

asymptotically approach a steady state value

What does it mean?

The probability of being in any future state

becomes independent of the initial state as time

process

? j limn?? Pr (Xnj X0i limn?? pij (n)

for all i and j

These asymptotical values are called

Steady-State Probabilities

Compute Steady-State Probabilities

Let p (p1, p2, . . . , pm) is the

m-dimensional row vector of steady-state

(unconditional) probabilities for the state space

S 1, ,m.

Brand switching example

p1 p2 p2 1, p1 ? 0, p2 ? 0, p3 ? 0

Solve linear system p pP,

?pj 1, pj ? 0, j 1, ,m

Steady-State Equations for Brand Switching Example

p1 0.90p1 0.02p2 0.20p3 p2 0.07p1

0.82p2 0.12p3 p3 0.03p1 0.16p2 0.68p3 p1

p2 p3 1 p1 ? 0, p2 ? 0, p3 ? 0

Total of 4 equations in 3 unknowns.

- Discard 3rd equation and solve the remaining

system - We get
- p1 0.474, p2 0.321, p3 0.205
- ? Recall q1(0) 0.25, q2(0) 0.46, q3(0)

0.29

Comments on Steady-State Results

1. Steady-state predictions are never achieved in

actuality due to a combination of (i) errors in

estimating P, (ii) changes in P over time, and

(iii) changes in the nature of dependence

relationships among the states. Nevertheless,

the use of steady-state values is an important

diagnostic tool for the decision maker.

2. Steady-state probabilities might not exist

unless the Markov chain is ergodic.

A stead State Does not Always Exist -- Gamblers

Ruin Example

For the Gamblers Problem, assume p 0.75, t

30

0 1 2 3 4 0 1 0 0 0 0 1 .325

0 0 0 .675 2 .1 0 0 0 .9 3

.025 0 0 0 .975 4 0 0 0 0 1

P(30)

What does matrix mean?

A Steady State Probability Does Not Exist in This

Case

Existence of Steady-State Probabilities

A Markov chain is ergodic if it is aperiodic and

allows the achievement of any future state from

any initial state after one or more transitions.

If these conditions hold, then

Classification of States in Markov Chain

Example

A state j is accessible from state i if pij(t) gt

0 for some t gt 0. In example, state 2 is

accessible from state 1 state 3 is

accessible from state 5 but state 3 is not

accessible from state 2.

States i and j communicate if i is accessible

from j and j is accessible from i. States 1 2

communicate also states

3, 4 5 communicate. States 2 4 do not

communicate

States 1 2 form one communicating class.

States 3, 4 5 form a 2nd communicating class.

If all states in a Markov chain communicate

(i.e., all states are members of the same

communicating class) then the chain is

irreducible.

The current example is not an irreducible Markov

chain.

How about the Gamblers Ruin Problem

Not an irreducible Markov Chain. It has 3

classes 0, 1, 2, 3 and 4.

Let fii probability that the process will

return to state i (eventually)

given that it starts in state i. If fii 1

then state i is called recurrent. If fii lt 1

then state i is called transient.

If pii 1 then state i is called an absorbing

state. Above example has no

absorbing states States 0 4 are absorbing in

Gamblers Ruin problem.

The period of a state i is the smallest n gt 1

such that all paths leading back to i have a

length that is a multiple of n i.e., pii(t)

0 unless t n, 2n, 3n, . . . If a process can

be in state i at time t or time t 1 having

started at state i then state i is aperiodic.

Each of the states in the current example are

aperiodic

Example of Periodicity - Gamblers Ruin

States 1, 2 and 3 each have period 2.

0 1 2 3 4 0 1 0 0 0 0 1 1-p 0 p 0 0 2 0

1-p 0 p 0 3 0 0 1-p 0 p 4 0 0 0 0 1

If all states in a Markov chain are recurrent,

aperiodic, the chain is irreducible then it

is ergodic.

Classification of States (continued)

An absorbing state is one that locks in the

system once it enters.

This diagram might represent the wealth of a

gambler who begins with 2 and makes a series of

wagers for 1 each. Let ai be the event of

winning in state i and di the event of losing in

state i. There are two absorbing states 0 and 4.

Illustration of Concepts

Example 1

Every pair of states communicates, forming a

single recurrent class however, the states are

not periodic. Thus the stochastic process is

aperiodic and irreducible.

Illustration of Concepts

Example 2

States 0 and 1 communicate and for a recurrent

class. States 3 and 4 form separate transient

classes. State 2 is an absorbing state and forms

a recurrent class.

Illustration of Concepts

Example 3

Every state communicates with every other state,

so we have irreducible stochastic process.

Periodic?

Yes, so Markov chain is irreducible and periodic.

Existence of Steady-State Probabilities

A Markov chain is ergodic if it is aperiodic and

allows the achievement of any future state from

any initial state after one or more transitions.

If these conditions hold, then

Conclusion chain is ergodic.

Game of Craps

The Game of Craps in Las Vegas plays as follows

The player rolls a pair of dice and sums the

numbers showing.

A total of 7 or 11 on the first rolls wins for

the player

Where a total of 2, 3, 12 loses

Any other number is called the point.

The player rolls the dice again.

If he/she rolls the point number, she wins

If he/she rolls number 7, she loses

Any other number requires another roll

The game continues until he/she wins or loses

Game of Craps as a Markov Chain

All the possible states

Start

Lose

Win

P4

P5

P6

P8

P9

P10

Continue

Game of Craps Network

Game of Craps

Probability of win Pr 7 or 11 0.167

0.056 0.223 Probability of loss Pr 2, 3, 12

0.028 0.56 0.028 0.112

Transient Probabilities for Craps

0.026

Recall, This is not an ergodic Markov Chain

Where you start counts

Absorbing State Probabilities for Craps

Interpretation of Steady-State Conditions

- Just because an ergodic system has steady-state

probabilities does not mean that the system

settles down into any one state.

2. ?j is simply the likelihood of finding the

system in state j after a large number of steps.

3. The limiting probability pj that the process

is in state j after a large number of steps is

also equals the long-run proportion of time that

the process will be in state j.

4. When the Markov chain is finite, irreducible

and periodic, we still have the result that the

pj, j Î S, uniquely solves the steady-state

equations, but now pj must be interpreted as the

long-run proportion of time that the chain is in

state j.

Insurance Company Example

An insurance company charges customers annual

premiums based on their accident history in

the following fashion

- No accident in last 2 years 250

annual premium - ? Accidents in each of last 2 years 800 annual

premium - ? Accident in only 1 of last 2 years 400 annual

premium

- Historical statistics
- If a customer had an accident last year then they

have a 10 chance of having one this year - If they had no accident last year then they have

a 3 chance of having one this year.

Find the steady-state probability and the

long-run average annual premium paid by the

customer.

Solution approach Construct a Markov chain with

four states (N, N), (N, Y), (Y, N), (Y,Y) where

these indicate (accident last year, accident

this year).

State-Transition Network for Insurance Company

.03

.90

.90

.03

.97

.10

.97

.10

This is an ergodic Markov chain All states

communicate (irreducible) Each state is

recurrent (you will return, eventually) Each

state is aperiodic.

Solving the steady state equations

?(N,N) 0.97 ?(N,N) 0.97 ?(Y,N) ?(N,Y)

0.03 ?(N,N) 0.03 ?(Y,N) ?(Y,N) 0.9

?(N,Y) 0.9 ?(Y,Y) ?(N,N) ?(N,Y)?(Y,N)

?(Y,Y) 1

Solution ?(N,N) 0.939, ?(N,Y) 0.029, ?(Y,N)

0.029, ?(Y,Y) 0.003

the long-run average annual premium

is 0.939250 0.029400 0.029400 0.003800

260.5

First Passage Times

Let ?ij expected number of steps to

transition from state i to state j If

the probability that we will eventually visit

state j given that we start in i is less

than one then we will have ?ij ?.

For example, in the Gamblers Ruin problem,

?20 ? because there is a positive probability

that we will be absorbed in state 4 given

that we start in state 2 (and hence visit

state 0).

Computations for All States Recurrent

If the probability of eventually visiting state

j given that we start in i is 1 then the

expected number of steps until we first visit

j is given by

We go from i to r in the first step with

probability pir and it takes mrj steps from r to

j.

It will always take at least one step.

For j fixed, we have linear system in m equations

and m unknowns mij , i 0,1, . . . , m1.

First-Passage Analysis for Insurance Company

Suppose that we start in state (N,N) and want to

find the expected number of years until we have

accidents in two consecutive years (Y,Y). This

transition will occur with probability 1,

eventually.

For convenience number the states 0 1

2 3 (N,N) (N,Y) (Y,N) (Y,Y) Then,

?03 1 p00 ?03 p01 ?13 p02?23 ?13

1 p10 ?03 p11 ?13 p12?23 ?23 1

p20 ?03 p21 ?13 p22?23

(N, N) (N, Y) (Y, N) (Y, Y)

(N, N) .97 .03 0 0 (N, Y) 0 0 .90 .10 (Y,

N) .97 .03 0 0 (Y, Y) 0 0 .90 .10

0 1 2 3

Using P

?03 1 0.97?03 0.03?13 ?13 1

0.9?23 ?23 1 0.97?03 0.03?13

Solution ?03 343.3, ?13 310, ?23

343.3

So, on average it takes 343.3 years to transition

from (N,N) to (Y,Y).

Expected First Passage Times

The average time it takes to reach other states

Recurrence time first passage time from a state

back to itself

It is the inverse of the

steady state probability

uii 1/?i

Absorbing States

An absorbing state is a state j with pjj 1.

Given that we start in state i, we can

calculate the probability of being absorbed in

state j. We essentially performed this

calculation for the Gamblers Ruin problem by

finding

P(t) (pij(t)) for large t.

But we can use a more efficient analysis like

that used for calculating first passage times.

Let 0, 1, . . . , k be transient states and k

1, . . . , m 1 be absorbing states.

Let qij probability of being absorbed in state

j given that we start in

transient state i. Then for each j we have the

following relationship

qij pij ? pirqrj , i 0, 1, . . . , k

k r0

Go directly to j Go to r and then to j

For fixed j (absorbing state) we have k1 linear

equations in k1 unknowns, qrj , i 0, 1, . . .

, k.

Absorbing States Gamblers Ruin

Suppose that we start with 2 and want to

calculate the probability of going broke, i.e.,

of being absorbed in state 0. We know p00 1

and p40 0, thus q20 p20 p21 q10 p22

q20 p23 q30 ( p24 q40) q10 p10

p11 q10 p12 q20 p13 q30 0 q30

p30 p31 q10 p32 q20 p33 q30

0 where P

0 1 2 3 4 0 1 0 0 0 0 1 1-p 0 p 0 0

2 0 1-p 0 p 0 3 0 0 1-p 0 p 4 0 0 0 0 1

Solution to Gamblers Ruin Example

Now we have three equations with three

unknowns. Using p 0.75 (probability of winning

a single bet) we have q20 0 0.25 q10

0.75 q30 q10 0.25 0.75 q20 q30 0

0.25 q20 Solving yields q10 0.325, q20

0.1, q30 0.025 (This is consistent with the

values found earlier.)

Applications of Markov Chain (Reading)

Linear Programming Min cx

Subject to Ax b, x ?0 where A is

a m ?n matrix, n variables, m constraints. n?m

Simplex algorithm Search along the boundary for

improving extreme points (vertex)

,exponential , of vertexes,

There might be as many as

It seems that simplex algorithm, on average, need

exponential number of iterations?

NO, see a Markov Chain Model for simplex

Algorithm (Handout)