Finite State Verification: An Emerging Technology for Validating Software Systems - PowerPoint PPT Presentation

About This Presentation
Title:

Finite State Verification: An Emerging Technology for Validating Software Systems

Description:

An Emerging Technology for Validating ... Testing, Theorem-proving based verification, Finite state verification(FSV) ... that does not impact the proof ... – PowerPoint PPT presentation

Number of Views:61
Avg rating:3.0/5.0
Slides: 66
Provided by: ccGa
Category:

less

Transcript and Presenter's Notes

Title: Finite State Verification: An Emerging Technology for Validating Software Systems


1
Finite State Verification An Emerging
Technology for Validating Software Systems
Lori A. ClarkeUniversity of Massachusetts Clarke_at_
cs.umass.eduhttp//laser.cs.umass.edu/
Laboratory for Advanced Software Engineering
Research
UMASS
2
Outline of Presentation
  • Lay of the Land
  • Testing, Theorem-proving based verification,
    Finite state verification(FSV)
  • Overview of FSV
  • Look at 3 Different Approaches to FSV
  • Model Checking
  • Flow Equations
  • Data Flow Analysis
  • Major Challenges to be Addressed

3
Sorry State of Affairs
  • Testing consumes about half the cost of s/w
    development
  • Maintenance consumes about 80 of the full life
    cycle costs--much of that devoted to testing
  • Most companies use ad hoc QA practices
  • Unhappy with the results Unhappy with the cost
  • Failed projects
  • Delayed product releases

4
Testing
  • can
  • Uncover failures
  • Show specifications are (not) met for specific
    test cases
  • Be an indication of overall reliability
  • cannot
  • Prove that a program will/will not behave in a
    particular way

5
Must do better!
  • Increasing number of high assurance applications
  • Medical applications
  • Flight control software
  • Electronic commerce
  • Increasing number of complex systems
  • Systems of systems
  • Distributed systems

6
Distributed Systems
  • Better performance, better flexibility,
    but there is a cost
  • distributed systems are more difficult to
    test than sequential systems
  • number of execution paths can grow exponentially
    with the number of processes
  • Testing can not even demonstrate that a system
    works on the selected/executed test data

Laboratory for Advanced Software Engineering
Research
UMASS
7
Complexity of Distributed Systems
T1
T2
6
1
7
8
3
4
9
8
Uncertainty of Testing
T1
T2
6
1
7
X 2
8
3
X1
4
9
X?
9
Formal Verification An Alternative to Testing
  • Theorem Proving Based Verification
  • Use mathematical reasoning
  • Prove properties about all possible executions
  • Difficult and error prone
  • Finite State Verification
  • Reason about a finite model of the system
  • Prove properties about all possible executions,
    but not as powerful as theorem proving
  • Almost a totally automated process

10
Spectrum of Difficulty
Ad-hoc Testing
Systematic Testing
Theorem Proving
Finite State Verification
  • Arbitrary testcases
  • Reqts based test planning
  • Requirements captured as properties
  • Properties guaranteed on all possible executions

11
Finite State Verification (FSV)
  • Holds the promise of providing a cost effective
    way of verifying important properties about a
    system
  • Not all faults are created equal
  • Invest effort into most important properties
  • Several promising prototypes
  • Reachability Based
  • SPIN or Symbolic Model Checking (SMV)
  • Flow Equations
  • Integer Necessary Conditions (INCA)
  • Data Flow Analysis
  • FLAVERS

12
High-Level Architecture of FSV Systems
Property
Property Translator
Property Representation
System
System Model
System Translator
Property Verified
ReasoningEngine
Counter Examples for Model
Laboratory for Advanced Software Engineering
Research
UMASS
13
Conservative Analysis
  • If property verified, property holds for all
    possible executions of the system
  • If property not verified
  • An error OR
  • A spurious result
  • System model abstracts information to be
    tractable
  • Conservative abstractions over-approximate
    behavior
  • If inconsistency relies upon over-approximations,
    then a spurious result
  • e.g. counter example corresponds to an infeasible
    path

14
System Model
  • Depends on property being verified
  • Eliminate information that does not impact the
    proof
  • Abstraction techniques allows states in the
    model to be reduced/collapsed

15
Some Properties of Properties
  • State-based versus event-based
  • Once temperature is greater than 100 degrees,
    lock is true
  • Elevator door closes before elevator moves
  • Single locations versus (sub)paths
  • Deadlock or race conditions
  • Sequences of states or events
  • Safety versus Liveness

16
A quick look at three approaches to FSV
  • Model Checking
  • Flow Equations
  • Data Flow Analysis

Big Disclaimer!
17
Model Checking some history
  • Originally proposed for hardware
  • Early 80s E. Clarke and Emerson Quielle and
    Sifakis
  • Late 80s Improved algorithms and property
    notations (E. Clarke, Emerson, Sistla)
  • 90s Symbolic Model Checking (SMV) and other
    optimizations (Burch, E. Clarke, Dill, Long, and
    McMillan)
  • Current Hybrid approaches

18
Model Checking
  • Properties usually expressed in a temporal logic
  • System represented as a (possibly abstracted)
    reachability graph
  • State based
  • Reasoning engine propagates valid subformulas
    through the graph

19
High-Level Architecture of Model Checking
Temporal Logic Property
Property Translator
Property Representation
State-based Reachability Graph
System
System Translator
Property Verified
Subformula propagation
Counter Examples for Model
Laboratory for Advanced Software Engineering
Research
UMASS
20
Representing Properties
  • CTL operators
  • G - globally
  • F - future
  • X- next
  • U - until
  • At a state in the model
  • AG p means that for all paths from this state, p
    is true and will remain true
  • EF p means that for some path from this state, p
    will eventually be true

21
Propagating Propositions
p
AF p
AF p
22
Example mutual exclusion protocol
reachability graph
n1,n2,turn0
n1,t2,turn2
t1,n2,turn1
n1,c2,turn2
t1,t2,turn2
c1,n2,turn1
t1,t2,turn1
c1,t2,turn1
t1,c2,turn2
McMillan
23
Example Property
  • AG(t1gtAF c1)
  • If process1 tries (t1) to get the lock then
    eventually it gets into its critical region (c1)
  • Note, would like to prove this for all processes
    but FSV approaches usually must instantiate
    property (and system)

24
Example propagation
AG(t1gtAF c1)
n1,n2,turn0
AF c1
t1gt
n1,t2,turn2
t1,n2,turn1
t1gt
AF c1
t1gt
n1,c2,turn2
t1,t2,turn2
c1,n2,turn1
t1,t2,turn1
AF c1
c1,t2,turn1
t1,c2,turn2
AF c1
AF c1
t1gt
25
Formula Propagation
  • Propagate until no change
  • propagate from smaller to larger subformulas
  • smart algorithm linear in the size of model
    and size of the formula
  • Many optimization techniques
  • Symbolic model checking
  • Use efficient algorithms that propagate
    subformula for sets of values

26
Symbolic Model Checking
  • With abstraction, nodes may represent sets of
    values
  • BDD
  • Worst case bound exponential in size of the model
  • For some examples, able to deal with 10120
    states

abc
a
a
0
1
0
1
b
b
b
0
1
0
0
1
1
1
c
0
1
1
1
27
Some observations Model Checking
  • Worst case bound linear in size of the model
  • Model exponential
  • Experimentally often very effective
  • Not clear if model checking or symbolic model
    checking is superior
  • Depends on the problem

28
Flow Equations some history
  • Originally proposed for designs
  • Early 80s Initial development (Avrunin,
    Dillon, and Wileden)
  • 90s Optimized and extended to real-time
    (Avrunin, Buy, Corbett, Dillon, and Wileden)
  • Current INCA prototype (Avrunin, Corbett, and
    Siegel)

29
Flow Equations
  • Model system as finite state automata
  • Use extended network flow inequalities to capture
    legal flow through a concurrent system
  • Represent negation of the property as a set of
    inequalities

30
Solving the Set of Inequalities
  • Determine if combined system of inequalities is
    consistent
  • Use integer linear programming
  • If consistent, there is a set of flows through
    automata that violate the property
  • Provides guidance for trace through the model
    (but may not be executable)

31
High-Level Architecture of INCA
Property
System
Property Translator
System Translator
Set ofInequalities
FSAs
Set of Inequalities
Property Verified(no solution)
FSA Translator
Integer Linear Programming System
Counter Examples for Model (solution)
Laboratory for Advanced Software Engineering
Research
UMASS
32
Example Process Flow Equations
a
b
a
b
33
Example Inter-process Flow Equations
a
b
a
b
x1 x5 x9 x6
34
Solving for a property
x1 x0 x2x1 x2 x3x0 1 x3 1x5 x7
x4 x6 x5 x6x4 1 x7 1x9 x8 x10x9
x10 x11x8 1 x111x1 x5x9 x6?j 0
xj
Property For all paths, event a occurs more than
event brepresent complement (x1 gt x9)
(x1 x9)
Solution exists e.g., x2, x10 0, all other xi
1 gt property does not hold
35
Seeing the counter example
Property For all paths, event a occurs more than
event b
a
b
a
b
x2, x10 0, all other xi 1
36
Some Limitations
  • Integer Linear Programming has an exponential
    worst case bound
  • Inter-process order information is not preserved
  • only checks whether event counts are consistent
  • Like most static techniques, may produce spurious
    results

37
Some Benefits
  • Does not enumerate the state space!
  • Integer linear Programming is often very
    efficient
  • Empirical evidence linear inequality systems
    usually grow linearly and take sub-exponential
    times to solve
  • In practice, INCA is usually an effective
    technique

38
Data Flow Based Verification some history
  • Mid-70s Originally proposed for def-ref
    anomalies in FORTRAN (Osterweil and Fosdick)
  • Early 80s Extended to general properties
    (Olender and Osterweil) concurrency (Taylor and
    Osterweil)
  • 90s Deadlock detection (Masticola and Ryder)
    Efficient representation of concurrency
    incremental precision improvement (Dwyer and L.
    Clarke)
  • Recent Optimizations, Java (Avrunin, L. Clarke,
    Cobleigh, Naumovich, and Osterweil)

39
Data Flow Analysis FLAVERS
  • Represents property as a finite state automaton
  • System model is collection of annotated control
    flow graphs
  • Inter-process communication and interleavings are
    represented with additional edges
  • does not enumerate all reachable states
  • over-approximates relevant executable behaviors
  • Reasoning engine based on data flow analysis

40
High-Level Architecture of FSV Systems
Property
Property Translator
FSA
Collection of annotated CFGs
System
System Translator
Property Verified
State Propagation
Counter Examples from Model
Laboratory for Advanced Software Engineering
Research
UMASS
41
Modeling the System
1,6
T1
T2
6
1
2,6
1,7
3,6
2,7
1,8
8
3
3,7
2,8
4
3,8
9
  • State explosion

4
5,9
42
Modeling the System
T1
T2
6
  • Automatically creates the program model from
    source code
  • Instead of the state space, explicitly represents
    interleaved execution via edges
  • Smaller model
  • Loss of precision

1
8
3
4
9
43
Representing Properties
Example
44
State Propagation
  • States of the property are propagated through the
    model
  • The property is proved if only accepting
    (non-accepting) states are contained in the final
    node of the model

45
Example
public static void main (String args)
if (elevatorStopped) ...
openDoors() recordState() if
(elevatorStopped) ... closeDoors()
moveToNextFloor()
if
open
if
close
move
46
Example
0
if
1
open
0,1
if
0
close
0,2
move
47
Incrementally Improving Precision
Property
Property Translator
FSA
System model
System Translator
State Propagation
System
Property Verified
Counter Examples for Model
Laboratory for Advanced Software Engineering
Research
UMASS
48
Example with Constraints
Property
Constraint
(0,0)
0
0
(0,1)
Sfalse
Strue
open
close
(1,1)
Sfalse
Strue
(1,1)
open
2
1
1
(1,viol)
move
Strue
Sfalse
viol
2
Strue
Sfalse
49
Example with Constraints
Property
Constraint
(0,0)
0
(0,1)
(0,2)
Sfalse
Strue
(1,1)
Strue
Sfalse
(1,1), (0,2)
2
1
(1,1), (0,viol)
(1,viol), (0,2)
Strue
Sfalse
viol
(0,1)
Strue
(0,1), (0,2)
Sfalse
50
Some Observations Data Flow Analysis
  • Overall complexity is O(N2S)
  • N is the nodes in the model
  • S is the number of states property x constraints
  • Experimentally performance subexponential
  • Usually requires several iterations to determine
    needed constraints
  • Constraints
  • Many automatically generated on request
  • Can be used to model other information

51
Experimental Comparisons
  • All these approaches are
  • very effective on some problems
  • disappointing on some problems
  • Hard to predict how they will perform
  • Experimental results
  • George S. Avrunin, James C. Corbett, Matthew B.
    Dwyer, Corina S. Pasareanu, and Stephen F.
    Siegel, Comparing Finite-State Verification
    Techniques for Concurrent Software

Very Big Disclaimer!
52
(No Transcript)
53
url laser.cs.umass.edu
54
Can we move beyond academic prototypes to
practitioners tools?
  • Yes, but there is more work to be done
  • Optimization, optimization, optimization
  • Process support
  • Better support for specifying properties
  • Better support for generating, selecting,
    visualizing counter example traces
  • Better approaches for dealing with dynamism
  • Full support for real languages
  • Full lifecycle support
  • Integration with testing

55
Specifying Properties
  • It is very hard to specify properties precisely
  • E.g., open and close file repeatedly
  • Must file always be opened?Or, IF it is opened,
    then it must be closed?
  • Can file be opened repeatedly before it is
    closed?
  • Need notations that are easy to use
  • Specification patterns
  • Need tools to help understand properties
  • need to test the properties

56
Counter Example Traces
  • Want short but useful counter examples
  • How to select the next counter example?
  • How to incorporate user guidance?
  • How to go from traces in the model to traces in
    the program?

57
Dynamism
  • FSV is a static analysis approach that deals with
    static models
  • Must create a specific instance of the model
  • E.g., N philosophers gt 5 philosphers
  • Can not handle
  • dynamic objects
  • dynamic process creation
  • Need hybrid techniques that integrate theorem
    proving with FSV

58
Support for Real Languages
  • Many language features have not been addressed
  • Aliasing
  • Exception handling
  • Event based notification

59
Lifecycle-based Verification
  • High-level architectural design
  • Extremely important for distributed systems
  • Detect problems early
  • Need to support heterogeneous interaction models
  • Low-level design
  • Additional detail leads to additional properties
  • Need to maintain consistency with the HLA

60
Lifecycle-based Verification (continued)
  • Coding
  • Partial systems
  • Incremental, compositional development/verificatio
    n
  • Debugging
  • Hypothesize fault in terms of a property
  • FSV provides a counter example trace or
    invalidates hypothesis

61
Lifecycle-based verification (continued)
  • Testing
  • Generalize test cases to their corresponding
    property
  • Test planning via requirements based property
    specification
  • Regression testing
  • re-verify properties that should not have changed
  • Need efficient re-verification techniques

Laboratory for Advanced Software Engineering
Research
UMASS
62
Integrating Testing and Verification
  • Testing and verification complement one another
  • verification makes assumptions that should be
    monitored dynamically
  • testing finds problems that should then be
    examined globally
  • Need to develop integrated techniques

63
Synergy between Testing and Verification
Testing
Counter examples
Assertions
Test plans/cases
Properties
Assumptions/constraints
Verification
64
Conclusions
  • Testing alone can not provide the assurance that
    is needed for many applications
  • especially distributed systems
  • FSV a promising technology
  • Applicable to a wide range of properties
  • Applicable throughout the lifecycle
  • Initial empirical results promising

65
Conclusion
  • Finite State Verification is a
    major paradigm shift
  • More difficult than testing, but
    not that much more difficult
  • Cultural resistance to doing anything different
  • Is the pain worth the gain?
  • Grand challenge Can we lower the obstacles
    to adoption?

Laboratory for Advanced Software Engineering
Research
UMASS
Write a Comment
User Comments (0)
About PowerShow.com