Applications of Stochastic Techniques to Partially Observed Discrete Event Systems

1
Applications of Stochastic Techniques to
Partially Observed Discrete Event Systems

• David Thorsley
• Department of
• Electrical Engineering and Computer Science
• University of Michigan
• April 28, 2006

2
Presentation Overview

• Introduction
• General Approach
• Discrete Event System Models
• Earlier Research
• Diagnosability of Stochastic DES
• Active Acquisition of Information
• Intrusion Detection in Centralized Systems
• Intrusion Detection in Decentralized Systems
• Current Research Issues

3
General Approach

• Investigate the similarities between stochastic
  systems and discrete-event systems
• Adding probabilistic information to DES models
  can result in more realistic results
• Information is better understood in the context
  of stochastic systems
• Optimization in stochastic systems is a more
  advanced field than optimization in DES
• The particular problems we consider relate to
  fault diagnosis and security

4
Definition of a DES

• The state space is discrete
• This state space may be finite or countable
• The state transition mechanism is event-driven
• At certain time instants, events announce that
  they are occurring
• Time may or may not be explicitly modeled

5
Logical Automata

• An automaton (or finite state machine) is a
  labeled directed graph with the following
  components

• Set of states X
• Set of events S
• Transition function d
• Initial state x0
• Set of marked states Xm

6
Stochastic Automata

• In a stochastic automaton, the transition
  function ? is replaced with a probabilistic
  function p

• The sum of the probabilities of all events out
  of a given state is 1
• The transition function ? can be derived from p

7
Information States in DES

• In stochastic systems, an information state p is
  a quantity that possess two mathematical
  properties
• (Causality) ?t can be determined from ?0, yk,
  uk-1
• (Recursion) ?t1 can be determined from ?t, yk1,
  uk
• An information state must also be sufficient for
  some particular purpose
• Examples of information states in DES include
• Diagnoser states
• Observer states

8
Presentation Overview

• Introduction
• General Approach
• Discrete Event System Models
• Earlier Research
• Diagnosability of Stochastic DES
• Active Acquisition of Information
• Intrusion Detection in Centralized Systems
• Intrusion Detection in Decentralized Systems
• Current Research Issues

9
Diagnosability Problem Formulation

• Given
• a DES G generating a language L(G)
• a set of observable events So ? S
• a set of failure events Sf ? Suo
• Under what conditions can we know that any
  instance of a failure event will be detected
  within a bounded amount of time?

10
Logical Diagnosability
f
true behavior s
observed behavior P(s)
possible true behaviors PL-1P(s)
11
Diagnosability of Stochastic DES

• Extension of aforementioned diagnosability
  problem to stochastic automata
• Proposed definitions of diagnosability
• Derived conditions for testing whether a system
  is diagnosable
• Constructed a stochastic diagnoser
• Applied results from Markov Chain theory to DES
• Results applied to heating, ventilation, and air
  conditioning systems
• Results published in IEEE Transactions on
  Automatic Control, April 2005

12
New Definition A-Diagnosability

• A language is A-diagnosable with respect to a
  projection P and a set of transition
  probabilities p if
• The diagnosability condition function D is given
  by

13
New Definition AA-Diagnosability

• A language L(G) is AA-diagnosable with respect to
  a projection P and a set of transition
  probabilities p if
• The diagnosability condition function Da is
  defined for each a lt 1 as

14
Comparison of A- and AA-Diagnosability
a
1
1-e
A priori probability of diagnosing the failure
Diagnosability
AA-Diagnosability
A-Diagnosability
1
A posteriori probability of failure occurrence
15
Stochastic Diagnoser

• A stochastic diagnoser is a sextuple
• Qd is the set of diagnoser logical elements
• So is the observable event set of G
• ?d is the partial transition function of the
  diagnoser
• q0 (0,N) ? Qd is the initial logical element
• ? is a set of probability transition matrices
• ?0 1 is the initial probability vector

16
State of the Stochastic Diagnoser

• The state of the stochastic diagnoser is a pair
  (q,?) where
• q is the current logical element of the diagnoser
• ? is a probability vector determined by
  multiplying the matrices that correspond with the
  observed transitions
• The stochastic diagnoser is an infinite-state
  machine
• The stochastic diagnoser state satisfies the
  necessary properties for an information state

17
Diagnosability of Stochastic DES

• Algorithms to construct the stochastic diagnoser
  and test for A- and AA-diagnosability implemented
  in UM-DES software library

18
Conditions for A- and AA-Diagnosability

• Conditions necessary and sufficient for
  A-diagnosability can be stated using the
  stochastic diagnoser
• As can conditions sufficient for AA-diagnosability

19
Presentation Overview

• Introduction
• Discrete Event System Models
• General Approach
• Earlier Research
• Diagnosability of Stochastic DES
• Active Acquisition of Information
• Intrusion Detection in Centralized Systems
• Intrusion Detection in Decentralized Systems
• Current Research Issues

20
Active Acquisition of Information

• Acquiring all possible information about large
  systems may be impractical
• Observable events may or may not be observed each
  time they occur
• A cost is incurred each time a sensor is
  activated to observe an event
• How to schedule observations so as to minimize
  the cost necessary for diagnosing failures?

21
System Model

• For this problem we consider a logical
  automaton G (S,X,?,x0)
• No probabilities!
• For this talk we make two simplifying
  assumptions
• The automaton G is acyclic and the maximum length
  of a string in L(G) is T
• Events are synchronized to ticks of the clock
• The thesis covers the problem for the case of
  general automata models

22
Active Acquisition System Structure
(monolithic) decision maker
DES
observer
scheduler
s
p
g(p)

• Conceptually, the decision maker consists of an
  observer and a scheduler
• The observer reads an event from a DES and sends
  its information to the scheduler
• The scheduler calculates an observation action
  based on that information

23
Decision Maker Operation
Problem Design an optimal decision maker
off-line i.e. Find an optimal g
24
How is information characterized?

• The structure of the diagnoser is not defined
  until its observation policy g is defined
• Finding an optimal observation policy requires
  describing the behavior of the decision maker
  along all possible strings
• How can we find a space such that, regardless of
  the specific observation policy, the information
  state is always an element of that space?
• Our approach involves constructing a filtration
  of maximal s-fields

25
Generation of Information Spaces

• We generate a sequence of maximal s-fields F0,
  F1, Ft
• At time t the information state ?(t) is an
  element of Ft
• The elements of the partition that generates Ft
  are sets of strings in LT that have identical
  projections for the first t events
• The partition that generates Ft is the finest
  information available to the diagnoser at time t
  it corresponds to the information available if
  all observations are available atall times

26
Example of Information States
27
Dynamic Programming Equations

• An optimal observation policy can be determined
  by solving the following dynamic program
• A final cost is assigned to all elements in FT
• Costs are calculated using backward induction
• The optimal observation cost is given by V0(LT)

28
Results from Cyclic Systems

• The assumptions made thus far are merely
  simplifying, not necessary
• The problem can be developed for general cyclic
  automata
• Logical or stochastic
• Conditions for solution existence rely on logical
  and stochastic diagnosability results
• Dynamic programming approach can still be used

29
Active Acquisition Results

• Limited lookahead algorithms devised to improve
  computational efficiency
• Method shown to be applicable for both diagnosis
  and supervisory control
• Results published in
• 42nd Allerton Conference on Control,
  Communication, and Computing, 2004
• 8th International Workshop on Discrete Event
  Systems, 2006

30
Presentation Overview

• Introduction
• Earlier Research
• Intrusion Detection in Centralized Systems
• Conditions for Intrusion Detection
• Damage Assessment
• Optimizing Control Specifications
• Intrusion Detection in Decentralized Systems
• Current Research Issues

31
Motivation

• Supervisory control theory assumes that all parts
  of the system are interacting as expected
• In practice, communications channels may be
  subject to interference
• How do we design our control system to take this
  interference into account?

32
System Structure
33
System Model

• Given an automaton G(X,S,d,x0,Xm)
• G is partially observable
• G is partially controllable
• Some of the controllable events can be overridden
  by the intruder set ?f ? ?c and Sc,f Sc \ Sf
• Partition Xm into good states Xm and bad
  states Xm-
• Define a specification K Lm that the
  supervisor can achieve in the absence of
  intrusion
• Assumption K is acyclic

34
Three Problems

• Can all good strings be executed without allowing
  an intruder to execute any bad strings?
• If an intruder can execute bad strings, how can
  we assess the damage caused by the intruder?
• How can we optimize our specification to minimize
  damage in the presence of an intruder?

35
Conditions for a Disarmable Language

• Given a specification K that is controllable and
  observable, we can construct a supervisor SP that
  achieves K
• Consider the set of strings that are disabled by
  K under intrusion
• A language Lm- is disarmable if

36
Disarming Supervisor Existence

• There exists a supervisor SP that achieves K and
  can prevent the execution of any string in Lm-
  under intrusion if and only if
• K is controllable and observable
• Lm- is disarmable

37
Example of Disarmability
38
Damage Assessment Language Measure

• Suppose the set of undesirable strings is not
  disarmable
• How can we assess the damage that an intruder can
  cause to the system?
• We address this question using the signed
  language measure technique
• Wang Ray, 2001

39
Assigning Costs to a Language

• Assign a terminal cost to each state
• For each state and transition, assign a value
  between 0 and 1

40
Assigning Costs to a Language

• Extend the transition measure to strings
  recursively
• Calculate the language measure m

41
Damage Assessment

• Given a set of failed controllers Sf, we can
  define the set of bad strings reachable under
  intrusion as
• The damage associated with Sf is simply

42
Example Computation
43
Optimal Specifications

• To find an optimal specification under intrusion,
  we consider both the rewards for reaching good
  states and the penalties for reaching bad
  states
• Find a specification K such that for all K ?
  COin(K)

44
Information Necessary for Optimization

• In the intrusion detection problem, observations
  are fixed
• Not like in the active acquisition of information
  problem
• Control actions need to be determined
• What information states do we need for this
  situation?

45
Information States for Optimization

• The supervisor acquires information through both
  its observations and control actions
• Strings with the same projection can be
  distinguished if the supervisor disables one but
  not the other
• Two strings s1 and s2 are equivalent under
  control if they have
• the same projection
• the same set of unobservable controllable events
  between each pair of successive observable events

46
Control Projection

• A function analogous to the standard projection
  operation for observation
• Strings with the same control projection are
  equivalent under control
• Observable events are preserved
• Strings of unobservable events are replaced by
  symbols indicating the set of controllable events
  included within them

47
Control Projection Definition

• The function CP is defined for events as
• CP is extended to strings recursively

48
Control Projection Examples

• s1 and s2 cannot be distinguished under any
  control policy, but s3 can be distinguished from
  both s1 and s2

49
Defining Information and Spaces

• We can use CP to generate a sequence of
  information states
• First define a sequence Xn
• For each Xn, define the set of associated
  information states as

50
Dynamic Programming Approach

• We initialize the dynamic programming by setting
  the cost of all information states outside the
  largest possible specification K
• If the supervisor sees behavior that cant be in
  the specification, it disables all possible
  events
• In practice, we would only calculate the costs of
  these states when required

51
Dynamic Programming Approach

• For each information state, we search over the
  set of admissible actions
• The DP equation is given by
• The current cost is the measure of the current
  information state and its unobservable reach
• The cost-to-go is the cost of all reachable
  continuations under the action g(?)

52
Example of an Optimal Specification
53
Presentation Overview

• Introduction
• Earlier Research
• Intrusion Detection in Centralized Systems
• Intrusion Detection in Decentralized Systems
• Conditions for Decentralized Intrusion Detection
• Optimization through Fictitious Play
• Current Research Issues

54
Decentralized Optimization
55
Decentralized Control Architecture

• Control specifications can be achieved if the
  specification is co-observable with respect to
  the observation capabilities of the given
  supervisors
• If an event needs to be disabled in order to
  achieve the specification, at least one
  supervisor will know for certain to disable it
• Each supervisor can be expressed with the
  equation

56
Conditions for a Co-Disarmable Language

• Define the language
• Lm- is co-disarmable if
• Then there exists a supervisor that achieves K
  while preventing the executing of Lm- under
  intrusion if and only if
• K is controllable and co-observable
• Lm- is co-disarmable

57
Decentralized Optimization Fictitious Play

• Two statisticians play a game
• Each player has a fixed finite action space
• Each player assumes that all other players are
  employing a mixed strategy that it computes from
  the observed history of the game
• At each stage of the game, each player chooses
  the best action with respect to the mixed
  strategies that it believes the other players are
  using

58
Fictitious Play for Supervisor Optimization

• In all players in the game have an identical
  payoff function, the belief path of a fictitious
  play process will converge
• Monderer Shapley, 1996
• All supervisors receive the same reward for
  optimizing the performance of the controlled
  system
• Thus we can use fictitious play to find a locally
  optimal solution
• With one caveat

59
Fictitious Play for Supervisor Optimization

• Convergence in belief path is guaranteed
• Belief paths correspond to mixed strategies
• Convergence to a pure strategy equilibrium is not
  guaranteed
• A pure strategy equilibrium is necessary because
  we want our supervisors to be deterministic
• A pure strategy equilibrium can be assured using
  the repeated restart method
• Lambert, 2003

60
Repeated Restart Algorithm

• For t1k
• For each supervisor j1n
• Find the best response yj(t1) for supervisor j
  based on the belief path fy (t)
• End for loop j
• Set
• End for loop t
• Find t such that
  for t k
• If y(t) is an equilibrium, STOP and output
  y(t) otherwise, set fy(1) y(t) and go to
  step 1

61
Dynamic Program for Best Response

• We find the best response at each stage of the
  repeated restart algorithm by using dynamic
  programming methods similar to those for the
  centralized case

62
Intuition Behind Repeated Restart

• For any game with common interests, a pure
  strategy equilibrium exists
• Each time the algorithm restarts
• The initial belief path is the previous
  iterations best response
• Since this belief is not an equilibrium, the
  first step of the restarted algorithm will
  improve the joint payoff
• Since we have a finite number of supervisors and
  a finite number of possible actions for each
  supervisor, the algorithm is guaranteed to
  terminate

63
Presentation Overview

• Introduction
• Earlier Research
• Intrusion Detection in Centralized Systems
• Intrusion Detection in Decentralized Systems
• Current Research Issues
• Modeling Different Intrusions
• Improved Computational Efficiency

64
Modeling Different Types of Intrusion
65
Improved Computation Efficiency

• General problems with partial observation,
  especially decentralized problems, are difficult
  computationally
• The problem of state explosion is compounded by
  the problem of information state explosion
• Restricting attention to specific classes of DES
  could result in more efficient algorithms
• Developing efficient models and heuristics for
  real-world systems is a challenge

66
Long-Term Interests

• Developing a standard model for solving
  optimization problems in discrete event systems
• Applying DES theory to communication and
  industrial applications
• Investigating similarities between DES theory and
  theoretical security

67
Conclusion

• Discrete event system models can be used to
  address questions of control and diagnosis in
  large systems
• My research addresses techniques for properly
  handling information and performing optimization
  in DES
• Many avenues still to explore