Agenda for This Week

1
Agenda for This Week
Monday April 11 Case 2 due Markov Processes
Wednesday, April 13 Markov Processes (HWs) Final Project Topic Due
Friday, April 15 Case 3 Review Dynamic Programming
Monday, April 18 Dynamic Programming
2
Chapter 17

• Markov Processes Part 3

3
Review

• A Markov Process describes a situation where a
  system is in one state at a time
• Switching between states is probabilistic
• The state of the system is dependent ONLY on the
  previous state of the system
• Steady State probabilities The long term
  probability of being in a particular state no
  matter which state you begin in

4
Absorbing States

• Markov chains can also be used to analyze the
  properties of a system in which some states are
  absorbing, that is, where once the system
  reaches that state, it never leaves that state.
• In an absorbing state, the probability that the
  process remains in that state once it enters the
  state is 1.
• States that are not absorbing are called
  transient states.

5
Absorbing State Examples

• Account aging in Accounts Receivable departments
• Movement of biological populations to extinction
• Depletion of non-renewable resources (such as
  oil, gas, etc.)

6
Absorbing States

• Provided that all states communicate with each
  other, the system will eventually end up in one
  of the absorbing states.
• In other words, as long as there is a pathway for
  the transient states to get to an absorbing
  state, it will eventually end up in an absorbing
  state.

7
Absorbing State - Questions

1. How long will it take before the system hits an
   absorbing state?
2. How much time will the system spend in each
   transient state before it is eventually absorbed?
3. If there are multiple absorbing states, what is
   the probability that the system will end up in
   each of those absorbing states?

8
Absorbing States

• If a Markov chain has both absorbing and
  nonabsorbing states, the states may be rearranged
  so that the transition matrix can be written as
  the following composition of four submatrices I,
  0, R, and Q

I O
R Q
9
Absorbing State
I An identity matrix indicating one always remains in an absorbing state once it is reached
O A zero matrix representing 0 probability of transitioning from the absorbing states to the nonabsorbing states
R The transition probabilities from the nonabsorbing states to the absorbing states
Q The transition probabilities between the nonabsorbing states
10
Fundamental Matrix

• The computation of absorbing state probabilities
  requires the determination and use of what is
  called a fundamental matrix
• The fundamental matrix, N, is the inverse of the
  difference between the identity matrix and the Q
  matrix.

Note I and Q must be the same size, ex 2x2,
3x3
N (I-Q) -1
11
NR Matrix

• The NR matrix is the product of the fundamental
  (N) matrix and the (R) matrix.
• It gives the probabilities of eventually moving
  from each nonabsorbing state to each absorbing
  state.
• Multiplying any vector of initial nonabsorbing
  state probabilities by NR gives the vector of
  probabilities for the process eventually reaching
  each of the absorbing states.

12
Calculating Inverse Matrices

• Use the following equations to calculate the
  inverse (I-Q) -1

INVERSE
1 2
1 a11 a12
2 a21 a22
1 2
1 a22/d -a12/d
2 -a21/d a11/d
d (a11) (a22) (a21) (a12)
13
Absorbing State Example 12

• Xmas tree farm has 5000 trees. 1500 trees
  classified as protected trees, 3500 available for
  cutting. Even if available, may not be sold.
  Most trees not cut live to next year but some
  diseased trees lost each year.

State 1 Cut and Sold State 2 Lost to
disease State 3 Too small for cutting State 4
Available but not cut and sold
1.0 0.0 0.0 0.0
0.0 1.0 0.0 0.0
0.1 0.2 0.5 0.2
0.4 0.1 0.0 0.5
P
14
Transition Matrix
I R O Q
Cut Sold Lost Too Small Avail but not cut
Cut Sold 1 0 0 0
Lost 0 1 0 0
Too small 0.1 0.2 0.5 0.2
Avail but not cut 0.4 0.1 0.0 0.5
15
12 Continued

• N (I-Q) -1
• I-Q -
  
• (1-Q) -1
  N
• D (.5)(.5) (0)(-.2) .25

.5 -.2
0 .5
.5 .2
0 .5
1 0
0 1
.5/.25 -.2/.25
0 .5/25
2 .8
0 2
16
12 Continued

• NR
• If we have 5000 trees (1500 protected and 3500
  available), we can multiply this by the NR matrix
  to how many will be eventually sold and lost.
• 
  3580 1420

2 .8
0 2
.1 .2
.4 .1
.52 .48
.8 .2
.52 .48
.8 .2
1500 3500
17
For Next Class

• Do HWs 1-4
• Try 13
• Look for Case 3 on Class Website