Dimensions of Formal Verification and Validation - PowerPoint PPT Presentation

Loading...

PPT – Dimensions of Formal Verification and Validation PowerPoint presentation | free to download - id: 1224a4-MDc1O



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

Dimensions of Formal Verification and Validation

Description:

(2) Requires many human testers (slow and expensive) (3) Validation is missing (effectively tester does both V&V) Cognitive requirement ... – PowerPoint PPT presentation

Number of Views:86
Avg rating:3.0/5.0
Slides: 40
Provided by: dorondr
Learn more at: http://www.nasa.gov
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Dimensions of Formal Verification and Validation


1
Dimensions of Formal Verification and Validation
  • Doron Drusinsky
  • Bret Michael
  • Mantak Shing
  • Naval Postgraduate School

2
Contents
  • Tradeoffs MC vs. TP vs. EMC

3
The Role of Specification Have we built the
right product?
E.g., if pump pressure is turned Low then High
and then Low again all within 10 seconds then
pump should not be High for at least 20
additional seconds
Customer cognitive requirements
Spec. Formal representation
class PumpCtl int x void pumpOn() …

4
The Role of Verification Have we built the
product right?
E.g., if pump pressure is turned Low then High
and then Low again all within 10 seconds then
pump should not be High for at least 20
additional seconds
Verification The bridge between specification
and implementation
class PumpCtl int x void pumpOn() …

5
Verification vs. Validation Emphasis
Most academic work is verification centered We
care about modeling, programming, and validation
just as much.
6
Background Primary Verification Techniques
True Model-Checking automatic verification
Cognitive/NL requirement
Formal spec
Formalization and validation
(Finite State) Model of system
?
Test suite many inputs sequences
Manual modeling (e.g., in Promela) or via
abstraction tool
system
7
Background Primary Verification Techniques
True Model-Checking
Formal spec
(Finite State) Model of system
?
Manual translation (e.g., Promela) or via
abstraction tool
system
8
Background Primary Verification Techniques
True Model-Checking
Formal spec
A significant limitation in our opinion
(Finite State) Model of system
Focus of academic interest
?
Manual translation (e.g., Promela) or via
abstraction tool
system
9
Background Primary Verification Techniques
Theorem Proving
Cognitive/NL requirement
Formal spec
Formalization and validation
(Infinite State) system
?
Test suite many inputs sequences
system
10
Background Primary Verification Techniques
Theorem Proving
Limitations (1)
limited validation
(even weaker…, hard-to-use
spec.-langs)
(2) Requires Ph.D. driver
Formal spec
(Infinite State) system
?
system
11
Background Primary Verification Techniques
Manual Testing
Cognitive requirement
Express as NL
Limitations (1) Slow, poor verification
coverage, expensive, hard to repeat
(2) Requires many human testers (slow and
expensive) (3) Validation
is missing (effectively tester does both VV)
12
Background Primary Verification Techniques
Execution based Model-Checking (EMC) Run-time
Execution Monitoring (REM/RV) Automatic Test
Generation (ATG)
Cognitive/NL requirement
Formalization and validation
ATG
Monitor (REM)
13
Background Primary Verification Techniques
Execution based Model-Checking (EMC)
PRO Better validation -- easy to use, expressive
languages, with simulation.
Limitations (1) No
absolute coverage
(but more can be specified…)
14
The Coverage Cube (more is better)
15
The Coverage Cube (more is better)
16
The Cost Cube (more is worse)
17
The Cost Cube (more is worse)
18
Example1 of A Validation Issue Weak
Specification Coverage
if pump pressure is repeatedly turned Low then
High N or more times (Ngt1) within 10 seconds then
pump should not be Low for at least 20 additional
seconds
19
Example 1 (cont.)
Customer cognitive requirement
Statechart-assertion for RV and EMC
20
Example 1 (cont.)
  • Outside the scope of MC/TP. They do not support
  • Real-time constraints (10 sec, 20 sec…)
  • Counting (N times…)
  • In general, they support at most ?-regular
    properties.

21
Example2 of Poor Specification Coverage
Customer cognitive requirement
Statechart-assertion for RV and EMC
NL (time-series) Whenever the track count (cnt)
Average Arrival Rate (ART) exceeds 80 of the
MAX_COUNT_PER_MIN cnt ART must be reduced back to
50 of the MAX_COUNT_PER_MIN within 2 minute and
cnt ART must remain below 60 of the
MAX_COUNT_PER_MIN for at least 10 minutes.
2min
10min
If ARTgt80
Then ARTgt50
And ARTgt60
22
More about Specification Languages LTL or
Buchi-Automata vs. Statechart-Assertions
  • LTL Buchi-Automata have lower specification
    coverage and are more expensive to use, partial
    list of reasons
  • Theoretical weak descriptive power (?-regular at
    best).
  • Hard to use the National Team can attest w.r.t.
    LTL
  • Lack of support for most basic constraints
    (real-time).
  • Infinite sequence semantics.
  • They are propositional (e.g., Always P ?
    Eventually Q ), while real systems are both
    conditional (propositional) and event-driven (see
    UML standard).

23
Example of Poor Program Coverage
Can we verify the property in the context of the
REAL code?
Program InfusionPump.java
24
Validation using JUnit or MSC
  • Validation. The StateRover uses JUnit-based
    simulation for validation.

JUnit-based scenario assertion.P() assertion.Q()
assertion.Q() assertion.P() assertion.Q() ass
ertTrue( assertions.isSuccess())
3 Qs after 1st P. Is that OK? Depends on
cognitive expectation
No more than N (e.g., 2) Q events can follow a P
event
25
Validation What can go Wrong?
1. Assertion is incorrect (usually where blame
is assigned). 2. Natural lang. is ambiguous. 3.
NL was written for main scenario, doesnt work as
well for other scenarios. 4. Validation scenario
is not what we think it is…
JUnit-based scenario assertion.P() assertion.Q()
assertion.Q() assertion.P() assertion.Q() ass
ertTrue( assertions.isSuccess())
3 Qs after 1st P. Is that OK? Depends on
cognitive expectation
No more than N (e.g., 2) Q events can follow a P
event
26
Thank you
27
Blunt User Questions
  • Q1. A property says light must be on for at
    least 5 seconds after door opens. My program
    already implements that, why write a
    spec.-property for that?
  • A.
  • Indeed, if everything we implemented was always
    correct the world would be a nice place…
  • When the implementation changes, who is the
    lobbyist for this requirement?
  • ? We need a separate representative for each
    requirement.

28
Blunt User Questions
  • Q2. Why not write a specification in Java (or in
    the language of the model).
  • A. We write specs as statechart-assertions. The
    motivation for not writing in Java is the same
    motivation that applies to using a code generator
    in general.

No more than N newCar events can follow a
newTruck event
29
Blunt User Questions
  • Q3. Whats the difference between a model and a
    program.
  • Abstraction.
  • Once the model has sufficient detail
    to be used as source code then its a program.
    Thats how StateRover statechart models/programs
    are used.

30
Blunt User Questions
  • Q4. Who says the spec. is correct?
  • Validation. The StateRover uses JUnit-based
    simulation for validation.

No more than N (e.g., 2) Q events can follow a P
event
JUnit-based scenario assertion.P() assertion.Q()
assertion.Q() assertion.P() assertion.Q() ass
ertTrue( assertions.isSuccess())
3 Qs after 1st P. Is that OK? Depends on
cognitive expectation
31
Comments of IVV Director Dr. Caffal
  • 1.  Natural language requirements are typically
    vague, inconsistent, and incomplete.
  • 2.  Natural language requirements frequently have
    counter-examples to the expressed logic.  The
    counter-examples are not easily observed by
    reading the requirement.
  • 3.  Unlike other disciplines, software developers
    oftentimes do not employ tools to describe
    behavior and elicit requirements.
  • 4.  Behavior specification comes in three
    flavors  what we want the system to do, what we
    do not want the system to do, and what we want
    the system to do under adverse conditions.
  • 5.  Nearly impossible to detect missing
    requirements.
  • 6.  Natural language requirements typically
    express constraints and limitations - rarely
    express behavior. The Team had to be hard pressed
    to come-up with behavioral requirements
  • 7.  Without specifying behavior, developers
    implicitly allow programmers to define
    behaviors.  As such, system behaviors emerge
    without design and structure.  Thus, emergent
    behaviors of systems are frequently an unhappy
    surprise to developers.

32
Behavioral specifications about What we want
the system to do
  • Whenever stop
  • command is
  • received then
  • vehicle should
  • reach complete
  • stop within 30
  • seconds

33
Behavioral specifications about what we do not
want the system to do (negative behavior)
  • Pump should never operate until at least two
    seconds after valve-shut.

This is where the end user says Ive already
implemented this behavior the positive way, why
do I need a negative behavior assertion?
34
Behavioral specifications about what the system
will do under adverse conditions (recovery)
35
Doing More for Validation
Under development tool that point out missing
assertion simulation/validation scenarios
36
Thank You
37
backup
38
Behavioral specifications about what we do not
want the system to do (negative behavior)
  • As of cruiseSet speed should not change by more
    than 2 unless incline is more than 5 for more
    than 10 seconds.

Speed instability
stable
V
Ambiguous a. Incline after speed instability b.
Incline during speed instability
39
Behavioral specifications about what we do not
want the system to do (negative behavior)
  • Negative statement
  • As of cruiseSet speed should not change by more
    than 2 unless incline is more than 5 for more
    than 10 seconds.

Positive statement As of cruiseSet speed should
be 98 stable unless incline is more than 5 for
more than 10 seconds.
The key about negative behavior is not the way
its phrased. Its the fact that a system is built
to do the positive, so it is assumed the negative
is
About PowerShow.com