# Lecture 5: State-Based Methods cont. - PowerPoint PPT Presentation

PPT – Lecture 5: State-Based Methods cont. PowerPoint presentation | free to download - id: 719efc-NDBhY

The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
Title:

## Lecture 5: State-Based Methods cont.

Description:

### ... recall that P If one thinks of accessibility in terms of the graphical representation, ... behavior can be computed by solving the linear equation n ... – PowerPoint PPT presentation

Number of Views:4
Avg rating:3.0/5.0
Slides: 13
Provided by: k328
Category:
Tags:
Transcript and Presenter's Notes

Title: Lecture 5: State-Based Methods cont.

1
Lecture 5 State-Based Methods cont.
• CS 7040
• Trustworthy System Design, Implementation, and
Analysis

Spring 2015, Dr. Rozier
Adapted from slides by WHS at UIUC
2
DTMCs - Refresh
3
Classifications
It is much easier to solve for the steady-state
behavior of some DTMCs than others. To
determine if a DTMC is easy to solve, we need
to introduce some definitions.
Definition A state j is said to be accessible
from state i if there exists an n 0 such (n)
that P
gt 0. We write i ? j.
ij
PX (n) j X (0) i
(n)
Note recall that P
ij
If one thinks of accessibility in terms of the
graphical representation, a state j is accessible
from state i if there exists a path of non-zero
edges (arcs) from node i to node j.
4
State Classification in DTMCs
Definition A DTMC is said to be irreducible if
every state is accessible from every other
state. Formally, a DTMC is irreducible if i ?
j for all i,j ? S. A DTMC is said to be
reducible if it is not irreducible. It turns out
that irreducible DTMCs are simpler to solve.
One need only solve one linear equation p
pP. We will see why this is so, but first there
is one more issue we must confront.
5
Periodicity
Consider the following DTMC
1
1
2
1 p(0) (1,0)
Does lim p(n)exist? No!
n?8
However,
does exist it is called the time-averaged
denoted by p. Definition A state i is said to
itself after n transitions only when n is some
multiple of d gt1, d the smallest integer for
which this is true. If d 1, then i is said to
be aperiodic.
Formally,
( j )
?d gt 1, j ? kd ? Pii 0
All states in the same strongly connected
component have the same periodicity. An
irreducible DTMC is said to be periodic if all
its states are periodic A steady-state solution
for an irreducible DTMC exists if all the states
solution for an irreducible DTMC always exists.
6
Transient and Recurrent States
7
Mean Recurrence Time
8
Connected States and Type
9
Examples on Board
Which of these are (a) irreducible? periodic?
10
The steady-state behavior can be computed by
solving the linear equation n
p pP, with the constraint that ?pi
For irreducible DTMCs, it can be
1.
i1
shown that this solution is unique. If the DTMC
is periodic, then this solution yields p.
One can understand the equation p pP in two
different ways. ? In steady-state, the
probability distribution p(n 1) p(n)P, and by
definition p(n 1) p(n) in steady-state. ?
Flow equations. Flow equations require some
visualization. Imagine a DTMC graph, where the
nodes are assigned the occupancy probability, or
the probability that the DTMC has the value of
the node.
11
Flow Equations Probability must be conserved,
i.e.,
. . .
. . .
i
?pi
1.
Let piPij be the probability mass that moves
from state j to state i in one time- step. Since
probability must be conserved, the probability
mass entering a state must equal the probability
mass leaving a state. Prob. mass in Prob. mass
out
n n ?p j Pji ?pi Pij
j1 j1
n p i ? Pij j1 p i Written in matrix form, p
pP.
12
For next time
• Homework 2 is posted, and due February 5th.