Models, Algorithms, and Tools for the Analysis of Systems

1
Models, Algorithms, and Tools for theAnalysis
of Systems
Rupak Majumdar Department of Computer Science
2
4 billion development effort
50 system integration and validation cost
3
240 Hp
100 processors
4
The Curse of Concurrency
300,000 latches
5
11
11
10 stars
100,000
10 states
6
Complexity Management

• Model Building capture the relevant aspects of
  the system formally (in mathematics) and
  automatically
• 2. Model Checking algorithms (i.e. software
  tools) for model analysis, rather than for model
  execution (simulation)

Model
Mathematics
Calculate
Predict
Abstract
Processors Aircraft Software
System
Test
Advantage of model over system analysis cost
Advantage of model analysis over execution
coverage
7
Formal Verification

• Programming Langs
• Semantics
• Concurrency
• Flow analysis

• Software Engg
• Modularity
• Abstraction

Formal Verification

• Theory
• Automata theory
• Proof theory
• Logic and theorem proving
• Algorithms

• Applications
• EDA hardware and processor verification
• Software verification
• Embedded systems (automotive, avionics)

8
Research Directions

• Model Design
• Language Design for Event-driven Programming
  ICSEsubmitted
• Quantitative Theory of Systems TCS05
• Stochastic Models and Games CSL04,UAI05,InfoProcL
  etters
• Model Checking
• Stack Size Analysis InfoComp
• Real-time Software EMSOFT05,HSCCsubmitted
• Model Extraction
• Counterexample guided refinement
• Lazy Abstraction for sofware verification
  POPL04,PLDI04,ICSE04,PLDI05,FSE05a,FSE05b
• Interface Inference

9
Research Directions

• Model Design
• Language Design for Event-driven Programming
• Nash equilibria in open games
• Discounted temporal logic
• Pareto curves in MDPs
• Model Checking
• Stack Size Analysis
• Timed context-free language inclusion problem
• Tree automata with memory
• Model Extraction
• Lazy Abstraction
• Sequential techniques for events
• Decision Procedure for Strings

Jeff Fischer
Krish Chatterjee
Mike Emmi
RuGang Xu, Manav Mital
Collaborations _at_UCLA Eddie Kohler, Todd
Millstein, Jens Palsberg
Other collaborators Tom Henzinger (EPFL), Ranjit
Jhala (UCSD), Shaz Qadeer (MSR), Luca de Alfaro
(UCSC), Marco Faella (Napoli)
10
Funding

• Jeff Fischer NSF ITR Event-driven quality
  project
• Ru-Gang Xu NSF SGER OS Verification
• Mike Emmi ITR Supplement for Collaboration with
  Europe
• Manav Mital Start-up package

11
Research Directions

• Model Design
• Language Design for Event-driven Programming
  ICSEsubmitted
• Quantitative Theory of Systems TCS05
• Stochastic Models and Games CSL04,UAI05,InfoProcL
  etters
• Nash equilibria in open games
• Model Checking
• Stack Size Analysis InfoComp
• Real-time software EMSOFT05,HSCCsubmitted
• Model Extraction
• Lazy Abstraction POPL04,PLDI04,ICSE04,PLDI05,FSE0
  5a,FSE05b
• Permissive Interface Inference

12
A Problem with Program Analysis
Library
Client

• Whole Program Analysis not always possible
• Availability Client code missing
• Scalability Whole system too large
• One main issue in scaling BLAST to large systems!

13
BLAST
Yes
BLAST
Safe
Search
C Program
POPL 02
Refine
No
Safety Property
POPL 04
Trace/ Test Case

• http//www.cs.ucla.edu/rupak/blast
• Performs Abstract Reachability Analysis based on
• Lazy predicate abstraction POPL02,CAV03
• Interpolation based refinement POPL04
• Scales to 100K lines of C code

Applied to Device driver code, Security-critical
code, Race-checking in nesC OS code,
event-driven code
14
Modular Program Analysis
Library
Client

• Modular Program Analysis
• Find interface for Library
• Use interface to verify client

15
Modular Program Analysis
Library
Interface
Availability Interface independent of
Client Scalability Interface small, abstraction
of Library
16
What is an Interface ?
Library States
API
Interface

• Interface Constraints on legal uses of API
• API Calls after which library is in a legal state

17
Example
Static e0 Static aNULL
Library States
Interface
API
rel
acq() if(aNULL) a m_new() else
e1 return
n0
read
read() if(a!NULL) a m_rd(a) else
e1 return
acq
rel
n2
acq
n1
rel() aNULL return
read
Safe Interface µ Legal Call Sequences
18
Safety Not Enough!
Interface
API
Static e0, aNULL, x0
acq() if(aNULL) am_new() else e1
return
acqx() if(aNULL) am_new() x1
else e1
read() if(a!NULL) am_rd(a) else
e1 return
write() if(x!0) m_wr(a) else e1
return
rel() aNULL return
relx() aNULL x0

• Disallows calls to write
• Useless for Modular Program Analysis

19
Permissive Interfaces
Static e0, aNULL, x0
Interface
API
rel/x
acq() if(aNULL) am_new() else e1
return
acqx() if(aNULL) am_new() x1
else e1
n0
acqx
acq
read() if(a!NULL) am_rd(a) else
e1 return
write() if(x!0) m_wr(a) else e1
return
n1
n2
relx
n3
rel() aNULL return
relx() aNULL x0
read
write read
Permissive Legal Call Sequences µ
Interface Modular Analysis Safe Permissive
Interfaces
20
Related Work

• Whaley-Lam ISSTA 02
• Use data-flow analysis, Error condition via
  exceptions
• Bar call to b if a modifies a variable guarding
  exn branch
• Not permissive
• Alur et. al. POPL 05
• Use machine learning to find set of legal
  sequences after
• Manually supplied finite abstraction
• Not permissive
• Strom-Yemini 86, Fahndrich-Deline ECOOP 04
• Typestate interpretation. No inference
  algorithm
• Chandra et. al. OOPSLA 05
• Dataflow based. Manual abstraction. Not
  permissive

21
Plan

• 1. Characterizing Safe, Permissive Interfaces
• 2. Computing Safe, Permissive Interfaces

22
Typestate Interpretations
Interface is a Typestate System - Abstraction of
librarys internal state Typestate
Interpretation - Overapprox possible internal
states
rel
n0
read
a0
acq
rel
n2
acq
e?0
n1
a?0
read
23
Safe Interpretations
Interface is a Typestate System - Abstraction of
librarys internal state Typestate
Interpretation - Overapprox possible internal
states
rel
n0
read
a0
acq
rel
n2
(P1) Initial states in r0
n0
r0
acq
e?0
n1
a?0
(P2) Every edge Post(r,f) µ r
read
(P3) Every legal typestate r µ Err

n
r
24
Safe Interpretations
Theorem Safe Interpretation implies Safe
Interface
rel
n0
read
a0
acq
rel
n2
(P1) Initial states in r0
n0
r0
acq
e?0
n1
a?0
(P2) Every edge Post(r,f) µ r
read
(P3) Every legal typestate r µ Err

n
r
25
Permissive Interpretations
Interface is a Typestate System - Abstraction of
librarys internal state Typestate
Interpretation - Overapprox possible internal
states
rel
n0
read
a0
acq
rel
n2
(P1) Initial states in r0
n0
r0
acq
e?0
n1
a?0
(P2) Every edge Post(r,f) µ r
read
(P4) Every illegal typestate r µ Err

n
r
26
Permissive Interpretations
Theorem Permissive Interpretation implies
Permissive Interface
rel
n0
read
a0
acq
rel
n2
(P1) Initial states in r0
n0
r0
acq
e?0
n1
a?0
(P2) Every edge Post(r,f) µ r
read
(P4) Every illegal typestate r µ Err

n
r
27
Computing Interfaces

• Problem A Interface Checking
• Given Library, candidate interface I,
  abstraction ?,
• Check if I is safe, permissive.
• Problem B Interface Reconstruction
• Given Library, abstraction ?,
• Reconstruct a safe, permissive interface I.
• Problem C Interface Inference
• Given Library,
• Infer a safe, permissive interface I.

28
A. Interface Checking Safe
Static e0 Static aNULL
rel
acq() if(aNULL) am_new() else e1
return
n0
read
rel
acq
n2
read() if(a!NULL) am_rd(a) else
e1 return
acq
n1
read
rel() aNULL return
Interface
Library
Problem A Interface Checking Given Library,
candidate interface I, abstraction ?, Check if
I is safe, permissive.
29
A. Interface Checking Safe
Static e0 Static aNULL
rel

acq() if(aNULL) am_new() else e1
return
Library States
n0
read
rel
acq
n2
read() if(a!NULL) am_rd(a) else
e1 return
acq
n1
read
rel() aNULL return
Interface Client
Library
Idea Analyze Interface Client Library Verify
assertion Client in legal location )
Library in legal state
n
30
B. Interface Checking Permissive
Static e0 Static aNULL
rel

acq() if(aNULL) am_new() else e1
return
Library States
n0
read
rel
acq
n2
read() if(a!NULL) am_rd(a) else
e1 return
acq
n1
read
rel() aNULL return
Interface Client
Library
Idea Analyze Interface Client Library Verify
assertion Client in illegal location
) Library in illegal state
n
31
A. Interface Checking
Safe, Permissive checkable by Assertion
Verification!
Problem A Interface Checking Given Library,
candidate interface I, abstraction ?, Check if
I is safe, permissive.
32
Computing Interfaces

• Problem A Interface Checking
• Given Library, candidate interface I,
  abstraction ?,
• Check if I is safe, permissive.
• Problem B Interface Reconstruction
• Given Library, abstraction ?,
• Reconstruct a safe, permissive interface I.
• Problem C Interface Inference
• Given Library,
• Infer a safe, permissive interface I.

Solution Assertion verification, Abstract
Reach. Graph
33
B. Interface Reconstruction
Static e0 Static aNULL
acq() if(aNULL) am_new() else e1
return

• a0,e0

read() if(a!NULL) am_rd(a) else
e1 return
Abstraction
rel() aNULL return
Library
Problem B Interface Reconstruction Given
Library, abstraction ?, Reconstruct a safe,
permissive interface I.
34
B. Interface Reconstruction
Static e0 Static aNULL
acq() if(aNULL) am_new() else e1
return
rel

• ? a0,e0

read() if(a!NULL) am_rd(a) else
e1 return
acq
read
Abstraction
Maximal Client
rel() aNULL return
Library
Idea I Abs Reach Graph of Max Client Library
(using ?) ARG Vertices w/ legal library state )
legal typestates ARG Vertices w/ illegal library
state ) illegal typestates
35
ARG of MaxLibrary
rel
read
n0
a0,e0
rel
Predicate Labels Typestate Interpretation
acq
n2
e0
acq
n1
a0, e0
read
Safe, Permissive by construction
Interface
36
Computing Interfaces

• Problem A Interface Checking
• Given Library, candidate interface I,
  abstraction ?,
• Check if I is safe, permissive.
• Problem B Interface Reconstruction
• Given Library, abstraction ?,
• Reconstruct a safe, permissive interface I.
• Problem C Interface Inference
• Given Library,
• Infer a safe, permissive interface I.

Solution Assertion verification, Abstract
Reach. Graph
Solution Interface ARG (w.r.t. ?) of Max
Client Library
37
Computing Interfaces

• Problem A Interface Checking
• Given Library, candidate interface I,
  abstraction ?,
• Check if I is safe, permissive.
• Problem B Interface Reconstruction
• Given Library, abstraction ?,
• Reconstruct a safe, permissive interface I.
• Problem C Interface Inference
• Given Library,
• Infer a safe, permissive interface I.

Solution Assertion verification, Abstract
Reach. Graph
Solution Interface ARG (w.r.t. ?) of Max
Client Library
38
C. Interface Inference
Require sufficiently precise abstraction ? -
Then B (reconstruction) suffices
Imprecise abstraction ) imprecise Abstract Reach
Graph

• Q How to deal w/ imprecise vertices ?
• Idea Any call sequence into vertex is either
  legal or illegal
• Legal sequence ) Infeasible path to Err
• Illegal sequence ) Infeasible path to Err
• Refine abstraction using call sequence into
  imprecise vertex
• Repeat until ARG precise, i.e. Interface found

39
Computing Interfaces

• Problem A Interface Checking
• Given Library, candidate interface I,
  abstraction ?,
• Check if I is safe, permissive.
• Problem B Interface Reconstruction
• Given Library, abstraction ?,
• Reconstruct a safe, permissive interface I.
• Problem C Interface Inference
• Given Library,
• Infer a safe, permissive interface I.

Solution Assertion verification, Abstract
Reach. Graph
Solution Interface ARG (w.r.t. ?) of Max
Client Library
Solution Refine abstraction using imprecise ARG
vertices
40
Two Requirements, Two Abstractions

• Safe, Permissive Orthogonal
• ? ?safe ?permissive
• ?safe calls allowed µ legal calls
• ?permissive calls disallowed µ illegal calls
• Build largest safe Interface I ,using ?safe
• Build ARG, imprecise vertices illegal
• Check I is permissive, using ?permissive
• Fails possibly legal, prohibited sequence to
  imprecise
• If sequence illegal then Refine ?permissive
• legal then Refine ?safe

41
To sum up

• Modular PA requires Safe,Permissive Interfaces
• Safe I µ legal sequences
• Perm legal sequences µ I
• Interface Typestate Graph
• Safe, Permissive via Typestate Interpretation
• Compute Interface via Abs. Reach Graph
• Implementation
• Safe, Permissive Interfaces for Java classes
• Automatic synthesis of Typestate Systems

42
Research Directions

• Model Design
• Language Design for Event-driven Programming
  ICSEsubmitted
• Quantitative Theory of Systems TCS05
• Stochastic Models and Games CSL04,UAI05,InfoProcL
  etters
• Nash equilibria in open games
• Model Checking
• Stack Size Analysis InfoComp
• Real-time software EMSOFT05,HSCCsubmitted
• Model Extraction
• Lazy Abstraction POPL04,PLDI04,ICSE04,PLDI05,FSE0
  5a,FSE05b
• Permissive Interface Inference

43
Non Zero Sum Games

• Model (2 players)
• Finite set of states S
• Finite set of actions ?
• Transition function ?(s,a1,a2) Prob.
  Distribution over S
• Objective For player i, to reach Siµ S

1,1 1,2
1,1 1,2 2,2
2,1 2,2
a
c
b
2,1
44
One Shot Games

• Games in strategic form
• Bimatrix games
• A matrix of payoffs for each player
• If player 1 plays a, and player 2 plays b, then
• player 1 gets P1a,b
• Player 2 gets P2a,b

45
Examples

• Prisoners Dilemma

Chicken
46
Nash Equilibrium

• A pair of strategies (?1, ?2) is an ?-Nash
  equilibrium if
• For all ?1, ?2
• Value2(?1, ?2) Value2(?1, ?2) ?
• Value1(?1, ?2) Value1(?1, ?2) ?
• Neither player has advantage of more than ? in
  deviating from the equilibrium strategy
• A 0-Nash equilibrium is called a Nash equilibrium

47
Nashs Theorem

• Theorem Every bimatrix game has a Nash
  equilibrium in randomized strategies.
• This also shows Nash equilibria exist in finitely
  repeated games
• Unfortunately, reachability games are infinite

48
Main Theorem

• Theorem ChatterjeeMJurdzinski04 An n-player
  nonzero sum reachability game has an ? Nash
  equilibrium in memoryless strategies for all ?gt0.
  An ?-equilibrium value profile can be
  approximated in NP.
• Solves an open question raised by Fink64

49
Idea of proof

• Define ?-discounted games, show memoryless Nash
  equilibria exist in such games. Similar results
  in Fink64,Sobel71
• Consider a Nash equilibrium in the ?-discounted
  reachability game. This equilibrium can be
  approximated by strategies of a simple form
  (k-uniform). This uses a result by
  LiptonMarkakisMehta03 for bimatrix games
• This strategy profile is an ?-Nash equilibrium in
  the original game for suitable ?.
• This is because if I fix the strategy of player
  2, in the resulting MDP, the value is close
  to the discounted value. This uses a result by
  Condon90
• Similarly for player 1

50
Perfect Information Games

• Theorem ChatterjeeMJurdzinki04
• n-Player perfect information probabilistic games
  with Borel payoffs have ?-Nash equilibria in
  deterministic strategies.
• n-player perfect information deterministic games
  with Borel payoffs have Nash equilibria in
  deterministic strategies.

51
Las Vegas Game
Work
Go to Vegas
Play again
1/2
Jackpot
Sorry you lose
52
Las Vegas Game

• For every ?gt0, Las Vegas game has a (1-?)-optimal
  winning strategy
• For ? 1/2n, work for n days before heading to
  Vegas
• But no optimal winning strategy
• The winning condition is not ?-regular
• Number of times you are allowed to play is the
  number of days you have worked

53
Conclusion

• There are many exciting practical as well as
  theoretical questions in formal verification
• There are exciting new tools for analyzing
  hardware and software systems
• The tools are grounded in deep theory
• Talk to me if you are interested!

54
Thank you

• http//www.cs.ucla.edu/rupak

55

• Backup Slides

56
BLAST
Abstract Non-uniformly
Is model unsafe ?
Check On the fly
Integrated
Interpolation based predicate discovery
Refine locally
Why infeasible ?
Infeasible
57
History Non Zero Sum Games

• Every finite n-player game has an equilibrium
  Nash50
• Complexity of finding a Nash equilibrium is open
  Pap94,Pap01
• Discounted stochastic n player games have a Nash
  equilibrium Fick64,MertensParthasarathy86
• 2-player nonzero sum stochastic games with
  limiting average payoff Vieille00
• Closed sets SuddherthSecchi02
• Open Sets (Reachability) ChatterjeeJurdzinskiM03
  
• (This talk)

58
No Nash Equilibrium Example
01 10
00
11
Player 1 has a randomized strategy to win with
probability 1-e for all e But no optimal strategy
59
Discounted Reachability Games

• A ?-discounted reachability game is played as
  follows.
• At each stage, the game stops with probability ?,
  and continues with probability 1- ?.
• Theorem A ?-discounted reachability game has a
  Nash equilibrium in memoryless strategies.
• The proof is an application of Kakutanis
  fixpoint theorem
• This is related to Nash equilibria in discounted
  reward games Fink64,Sobel71

60
Approximating Strategies

• Let J be a bimatrix game with n players
• Each player has m actions
• A strategy is k-uniform if it is a uniform
  distribution over a multiset of size k
• Let ? be a Nash equilibrium profile.
• LiptonMarkakisMehta03 For every ?gt0, for every
• k gt (3n2 ln (n2m))/?2 there exists a
  k-uniform strategy profile ? s.t. for every
  action a,
• if ?(a)0, then ?(a)0.
• if ?(a)gt0 then ?(a)- ?(a) lt ?

61
Approximating Equilibria inDiscounted Games

• For an n-player discounted reachability game G?,
  for every ?gt0, there exists a memoryless strategy
  profile ? such that
• ? is an ?-Nash equilibrium profile of G? and
• for every player i, the minimum transition
  probability in the MDP Gi is at least f(?,n,G).

62
Markov Decision Processes

• A Markov decision process (MDP) is a one player
  game.
• Reachability, discounted reachability is defined
  on MDPs by restriction from games.
• When we fix the strategies of all but one player
  i, we have an MDP Gi.

63
Approximating MDPs

• Let G be a MDP reachability game
• Condon90 For all ?gt0 there exists discount
  factor ? such that for all states s2 S of the
  ?-discounted game G? we have
• v(s) v?(s) lt ?

64
Complexity

• Can approximate an ?-Nash equilibrium to within ?
  for constant ?, ? in NP
• Guess the memoryless (k-uniform) strategy
  profiles
• Solve the MDPs after fixing all but one players
  strategies
• Payoffs can be irrational, so we can only hope to
  approximate

65
Trick with Deterministic Strategies

• For an n-player game where player i has objective
  ?i
• Consider the zero sum game of player i with
  objective ?i against all other players with
  objective ?i
• Suppose this zero sum game has a deterministic
  winning strategy ?i for i and ?i for all the
  others
• Nash equilibrium
• Every player i plays ?i from above.
• As soon as someone deviates, all the other
  players punish by switching to ?i
• Deterministic strategies are necessary to observe
  deviations
• Folk result? ThuijsmanRaghavan97.

66
Turn Based Games

• A careful study of Martins determinacy proof
  shows that we can construct ?-optimal
  deterministic strategies for turn based
  probabilistic games
• And optimal pure strategies for deterministic
  turn based games